Commit Graph

833 Commits

Author SHA1 Message Date
Eugen Rochko
93897134ca
Permit dots in usernames with conditions (#6844)
* Permit dots in usernames with conditions

- Dot cannot be the start or end of username
- a.lice and al.ice are considered the same during sign-up

* Fix regex mixin flags
2018-03-21 10:26:53 +01:00
Jenkins
df70571bfe Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-19 20:17:18 +00:00
Alexander
0306e3e9be bugfixes and gem update (#6831)
* update to new version of devise_pam_authenticatable2

* fix behaviour if suffix is nil, fix environment loading, fix user email creation

* code cleanup/fix linter warning
2018-03-19 20:08:56 +01:00
Jenkins
9341b982b6 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-17 17:17:21 +00:00
Daniel Hunsaker
4a0a19fe54 Handle Mastodon::HostValidationError when pulling remoteable assets (#6782)
This will prevent, for example, `rake mastodon:redownload_avatars` from crashing when an instance is no longer responding to connection attempts, instead silently continuing as expected.
2018-03-17 13:27:50 +01:00
David Yip
f8e934f955
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
	README.md
2018-03-12 17:52:19 -05:00
Eugen Rochko
1aaec701bb
Fix #6715: Make catalan words with the L geminate letter work in hashtags (#6741) 2018-03-11 14:55:49 +01:00
Jenkins
447d7e6127 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-09 00:17:17 +00:00
Renato "Lond" Cerqueira
9aba44ea79 Rescue when there's no extension in the remotable (#6358)
* Rescue when there's no extension in the remotable
Sometimes the remotable is pointing to a directory with no file
extension. Maybe it should not be expecting to identify based on
extensions to begin with, but since it's the case, it should be ready
for it.

* Fix codeclimate issue

* Check if filename is nil instead of rescueing exception
Suggestion made in the PR

* Avoid concatenation issue if filename is nil
If filename is nil, extname was undefined

* Invert condition
Address PR comments
2018-03-08 02:25:10 +01:00
Eugen Rochko
cfa9b6e13a
Remove text requirement when media attached from statuses (#6672) 2018-03-07 08:28:52 +01:00
Effy Elden
dd9d00d293 Add additional first_name and last_name SAML attribute statement options, and modify Omniauthable concern to use full_name or first_name + last_name if not available (#6669) 2018-03-07 06:19:10 +01:00
Eugen Rochko
78d772af86
Fix #3807: Increase avatars to 400x400 max (#6651)
But do not upscale when they are smaller
2018-03-06 06:29:01 +01:00
Akihiko Odaki
c110fa62ac Provide default OTP_SECRET value for development environment (#6617) 2018-03-04 20:28:24 +01:00
Eugen Rochko
9110db41c5
Federate pinned statuses over ActivityPub (#6610)
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix #6117

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
2018-03-04 09:19:11 +01:00
David Yip
1b8fcd4df5
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
 	README.md
 	app/controllers/follower_accounts_controller.rb
 	app/controllers/following_accounts_controller.rb
 	app/serializers/rest/instance_serializer.rb
 	app/views/stream_entries/_simple_status.html.haml
 	config/locales/simple_form.ja.yml
2018-03-02 21:46:44 -06:00
Eugen Rochko
47bdb9b33b
Fix #942: Seamless LDAP login (#6556) 2018-02-28 19:04:53 +01:00
Eugen Rochko
41a01bec23
Federated reports (#6570)
* Fix #2176: Federated reports

* UI for federated reports

* Add spec for ActivityPub Flag handler

* Add spec for ReportService
2018-02-28 06:54:55 +01:00
Ian McCowan
c33931b613 Fix prev/next links on public profile page (#6497)
* Fix prev/next links on public profile page

* Don't make pagination urls if no available statuses

* Fix empty check method

* Put left chevron before prev page link

* Add scope for pagination "starting at" a given id

* Status pagination try 2:

s/prev/older and s/next/newer
"older" on left, "newer" on right
Use new scope for "newer" link
Extract magic 20 page size to constant
Remove max_id from feed pagination as it's not respected

* Reinstate max_id for accounts atom stream

* normalize
2018-02-26 03:31:28 +01:00
Eugen Rochko
5cc716688a
Ensure the app does not even start if OTP_SECRET is not set (#6557)
* Ensure the app does not even start if OTP_SECRET is not set

* Remove PAPERCLIP_SECRET (it's not used by anything, actually)

Imports are for internal consumption and the url option isn't even
used correctly, so we can remove the hash stuff from them
2018-02-26 01:31:44 +01:00
imncls
bb6988a7ac
Merge branch 'master' of https://github.com/tootsuite/mastodon
# Conflicts:
#	app/controllers/settings/exports_controller.rb
#	app/models/media_attachment.rb
#	app/models/status.rb
#	app/views/about/show.html.haml
#	docker_entrypoint.sh
#	spec/views/about/show.html.haml_spec.rb
2018-02-23 23:28:31 +09:00
Ghislain Loaec
e668180044 New variable OAUTH_REDIRECT_AT_SIGN_IN + Ref #6538 (not only SAML strategies) (#6540) 2018-02-23 01:16:17 +01:00
Ghislain Loaec
3084fe4959 New env variable: SAML_SECURITY_ASSUME_EMAIL_IS_VERIFIED + fixes #6533 (#6538) 2018-02-22 23:31:25 +01:00
Eugen Rochko
90f12f2e5a
Focal points (#6520)
* Add focus param to media API, center thumbnails on focus point

* Add UI for setting a focal point

* Improve focal point icon on upload item

* Use focal point in upload preview

* Add focalPoint property to ActivityPub

* Don't show focal point button for non-image attachments
2018-02-22 00:35:46 +01:00
Eugen Rochko
d3a62d2637
Fix #6525: Make sure file is opened in LazyThumbnail processor (#6529) 2018-02-22 00:28:19 +01:00
Eugen Rochko
61ed133fea
Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
2018-02-21 23:21:32 +01:00
Eugen Rochko
a7171af0a3
Fix avatar and header issues by using custom geometry detector (#6515)
* Fix avatar and header issues by using custom geometry detector

Revert a part of #6508. The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.

One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.

Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.

* Only invoke convert if dimension or format needs to be changed
2018-02-21 03:40:12 +01:00
Jenkins
1445ba1703 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-02-19 15:17:14 +00:00
Eugen Rochko
66105929e0
Don't resize avatars/headers if their dimensions are already right (#6508)
Also don't apply "-quality 80" option which is probably the reason
for slight color differences between original and remote image
(because it would apply it twice, once on original instance, and
again on the receiving instance)
2018-02-19 16:06:12 +01:00
Eugen Rochko
cba2897108
Cache relationships in API (#6482)
* Cache relationships in API

* Fetch relationships for search results in UI

* Only save one account's maps in each cache item
2018-02-18 03:14:46 +01:00
Eugen Rochko
a71af98401
Push discovered status through streaming API within a time window (#6484)
Time window of 6 hours
2018-02-17 14:28:48 +01:00
David Yip
3d033a4687
Merge remote-tracking branch 'tootsuite/master' into merge-upstream
Conflicts:
	Gemfile
	config/locales/simple_form.pl.yml
2018-02-17 00:02:37 -06:00
Eugen Rochko
9dbae6e8a1
Save video metadata and improve video OpenGraph tags (#6481)
* Save metadata from video attachments, put correct dimensions into OG tags

* Add twitter:player for videos

* Fix code style and test
2018-02-16 07:22:20 +01:00
Jenkins
efc9ffcb80 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-02-11 04:10:16 +00:00
David Yip
b6159bcb0a
Merge remote-tracking branch 'origin/merge-upstream' 2018-02-10 22:08:45 -06:00
David Yip
f1f67c46c5
Use Html2Text to generate plaintext for keyword mutes. #236.
This allows us to match URLs inside link hrefs.
2018-02-10 10:36:16 -06:00
David Yip
53c86b29f0
Glitch::FilterHelper -> Glitch::KeywordMuteHelper. #234.
The class helps out with keyword mutes, not just some general concept of
"filtering".
2018-02-10 10:36:15 -06:00
David Yip
d263e3bc2d
Fill out some examples for Glitch::FilterHelper. #234.
Also add HTML entity decoding to Glitch::FilterHelper, which is needed
to e.g. match "<" to the tag-stripped version of "<p><3</p>" or
"<p>&lt;3</p>".
2018-02-10 02:40:27 -06:00
David Yip
29b5b46c87
Strip HTML from keyword mute input. #234. 2018-02-10 02:40:27 -06:00
Eugen Rochko
3ebc0ad4d3
Full-text search for authorized statuses (#6423)
* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix #5880
Fix #4293
Fix #1152

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues
2018-02-09 23:04:47 +01:00
David Yip
020b40efdb
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
	.travis.yml
	app/lib/user_settings_decorator.rb
	app/models/user.rb
	app/serializers/initial_state_serializer.rb
	app/views/stream_entries/_detailed_status.html.haml
	app/views/stream_entries/_simple_status.html.haml
	config/locales/simple_form.en.yml
2018-02-09 09:25:53 -06:00
Eugen Rochko
76f3d5d16b
Add preference to always display sensitive media (#6448) 2018-02-09 00:26:57 +01:00
Jenkins
5ba2c300d8 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-02-08 23:17:14 +00:00
abcang
298c81c00f Clear account cache of notification target_status (#6442) 2018-02-08 15:33:23 +01:00
Akihiko Odaki
67f7ffa792 Change user_id column non-nullable (#6435) 2018-02-07 16:35:44 +01:00
David Yip
a6fb1c58ee
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
	.env.production.sample
	app/controllers/auth/confirmations_controller.rb
	db/schema.rb
2018-02-04 16:36:19 -06:00
Eugen Rochko
38e0133e1b
Make PAM gem optional, allow configuration over environment (#6415) 2018-02-04 15:05:53 +01:00
Renato "Lond" Cerqueira
4e4f1b0dcb Add option to show only local toots in timeline preview (#6292)
* Add option to show only local toots in timeline preview
Right know, toots from all the known fediverse are shown in the main
page of an instance. That however doesn't reflect the instance itself.
With this option the admin may choose to display only local toots so
that users checking the instance get a better idea of internal
conversations.

* Fix issues pointed by codeclimate and eslint

* Add default message for community timeline

* Update pl.yml
2018-02-04 06:00:10 +01:00
Eugen Rochko
26f21fd5a0
CAS + SAML authentication feature (#6425)
* Cas authentication feature

* Config

* Remove class_eval + Omniauth initializer

* Codeclimate review

* Codeclimate review 2

* Codeclimate review 3

* Remove uid/email reconciliation

* SAML authentication

* Clean up code

* Improve login form

* Fix code style issues

* Add locales
2018-02-04 05:42:13 +01:00
David Yip
4c1fd9a19c
Merge remote-tracking branch 'tootsuite/master' into merge-upstream
Conflicts:
      app/javascript/styles/mastodon/components.scss
2018-02-02 08:39:52 -06:00
Alexander
04fef7b888 pam authentication (#5303)
* add pam support, without extra column

* bugfixes for pam login

* document options

* fix code style

* fix codestyle

* fix tests

* don't call remember_me without password

* fix codestyle

* improve checks for pam usage (should fix tests)

* fix remember_me part 1

* add remember_token column because :rememberable requires either a password or this column.

* migrate db for remember_token

* move pam_authentication to the right place, fix logic bug in edit.html.haml

* fix tests

* fix pam authentication, improve username lookup, add comment

* valid? is sometimes not honored, return nil instead trying to authenticate with pam

* update devise_pam_authenticatable2 and adjust code. Fixes sideeffects observed in tests

* update devise_pam_authenticatable gem, fixes for codeconventions, fix finding user

* codeconvention fixes

* code convention fixes

* fix idention

* update dependency, explicit conflict check

* fix disabled password updates if in pam mode

* fix check password if password is present, fix templates

* block registration if account is maintained by pam

* Revert "block registration if account is maintained by pam"

This reverts commit 8e7a083d650240b6fac414926744b4b90b435f20.

* fix identation error introduced by rebase

* block usernames maintained by pam

* document pam settings better

* fix code style
2018-02-02 10:18:55 +01:00
Jenkins
0e10667fbe Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-24 19:17:11 +00:00
Eugen Rochko
1cc44cba81
Fix #6331 (#6341)
UserTrackingConcern is circumvented by SessionsController#create
because it calls warden, which calls the User#update_tracked_fields!
method directly. Move returning user logic to that method.
2018-01-23 20:52:30 +01:00
Akihiko Odaki
613e7c7521 Rename ResolveRemoteAccountService to ResolveAccountService (#6327)
The service used to be named ResolveRemoteAccountService resolves local
accounts as well.
2018-01-22 14:25:09 +01:00
David Yip
2ca965c704
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
	app/javascript/styles/mastodon/components.scss
	app/javascript/styles/mastodon/modal.scss
2018-01-19 15:22:10 -06:00
Yamagishi Kazutoshi
238de58e65 Change belongs_to_required_by_default to true (#5888) 2018-01-19 20:56:47 +01:00
Eugen Rochko
d799921c75
Replace tutorial modal with welcome e-mail (#6273)
* Remove onboarding modal

* Welcome e-mail

* Send welcome e-mail after confirmation

* Remove obsolete translations
2018-01-18 19:17:25 +01:00
Jenkins
0c7dc6c781 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-15 06:17:15 +00:00
Eugen Rochko
ed867eca9d
Move e-mail digest task to sidekiq, reduce workload, improve hint (#6252) 2018-01-15 04:34:28 +01:00
Jenkins
f839ac694c Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-08 10:17:15 +00:00
Eugen Rochko
dbda87c31f
Revert #5772 (#6221) 2018-01-08 10:57:52 +01:00
Jenkins
c69a23ae46 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-04 23:17:11 +00:00
Patrick Figel
5ec25ff3e1 Fix email confirmation link not updating email (#6187)
A change introduced in #6125 prevents
`Devise::Models::Confirmable#confirm` from being called for existing
users, which in turn leads to `email` not being set to
`unconfirmed_email`, breaking email updates. This also adds a test
that would've caught this issue.
2018-01-05 00:15:35 +01:00
Jenkins
fc884d015a Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-03 20:17:13 +00:00
Akihiko Odaki
161c72d66d Allow to dereference Follow object for ActivityPub (#5772)
* Allow to dereference Follow object for ActivityPub

* Accept IRI as object representation for Accept activity
2018-01-03 18:08:57 +01:00
Jenkins
933840bebf Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-01-03 04:17:11 +00:00
Patrick Figel
04ecf44c2f Add confirmation step for email changes (#6071)
* Add confirmation step for email changes

This adds a confirmation step for email changes of existing users.
Like the initial account confirmation, a confirmation link is sent
to the new address.

Additionally, a notification is sent to the existing address when
the change is initiated. This message includes instruction to reset
the password immediately or to contact the instance admin if the
change was not initiated by the account owner.

Fixes #3871

* Add review fixes
2018-01-02 16:55:00 +01:00
David Yip
4cca1d1e7e
Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
	app/controllers/auth/confirmations_controller.rb
2017-12-30 17:20:07 -06:00
Eugen Rochko
38fc1b498d
Add more instance stats APIs (#6125)
* Add GET /api/v1/instance/peers API to reveal known domains

* Add GET /api/v1/instance/activity API

* Make new APIs disableable, exclude private statuses from activity stats

* Fix code style issue

* Fix week timestamps
2017-12-29 19:52:04 +01:00
David Yip
82b2e224a2
Merge branch 'gs-master' into prevent-local-only-federation
Conflicts:
	db/schema.rb
2017-12-15 12:20:56 -06:00
David Yip
e35a350119
Examples for Status#set_locality and .as_tag_timeline.
This commit also:

- exposes the local-only emoji so that it can be used in examples
- allows local_only to be set explicitly, i.e. for timeline filtering
  specs
2017-12-14 02:27:42 -06:00
Jenkins
82236a3703 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-13 18:17:16 +00:00
Eugen Rochko
ad75ec8b5b
Add filters to admin UI for custom emojis (#6003) 2017-12-13 13:28:31 +01:00
Eugen Rochko
20a6584d2d
Clean up admin UI for accounts (#6004)
* Add staff filter to admin UI for accounts, remove obsolete columns

* Only display OStatus section in admin UI for accounts when OStatus data
2017-12-13 12:15:10 +01:00
David Yip
a057ed5cfe
Merge remote-tracking branch 'tootsuite/master' into merge-upstream 2017-12-12 02:54:13 -06:00
Akihiko Odaki
14c4a33cd9 Change account_id non-nullable in lists (#5979) 2017-12-12 04:11:17 +01:00
Eugen Rochko
481fac7c84
Exclude moved accounts from search results (#5984) 2017-12-12 02:14:33 +01:00
Erin
c5a4eda694 move outbox filtering to Status#permitted_for (as per @ekiru) 2017-12-11 15:28:04 -06:00
Erin
288f1293ef set local_only in a before_create callback instead of status service 2017-12-10 21:39:27 -06:00
Erin
0c46058a43 remove vestigial Status#local_only? definition 2017-12-10 21:25:36 -06:00
Erin
24f36ca912 Status#not_local_only scope should match nils too 2017-12-10 20:35:42 -06:00
Erin
6bd18e43ba filter local-only statuses from public pages 2017-12-10 17:23:01 -06:00
Erin
08519cd4f4 status: stub local_only?, add scope, add marked_local_only? 2017-12-10 17:04:28 -06:00
Erin
434c70fd98 add a local_only column to the statuses table 2017-12-10 16:41:25 -06:00
Andrea Scarpino
6a82939adb Fix account and tag searches with leading/trailing spaces (#5965)
* Strip leading & trailing spaces from account query

* Strip leading & trailing spaces from tag search
2017-12-10 19:35:46 +01:00
cwm
abe95b614b add initial components based off of tootsuite pr #1507 2017-12-09 10:26:22 -06:00
Eugen Rochko
2f4c5f504f
Limit users to 50 lists, remove pagination from lists API (#5933) 2017-12-09 01:32:29 +01:00
Eugen Rochko
c36b9cc5a6
Ensure link thumbnails are not stretched to super low quality (#5932) 2017-12-09 00:56:16 +01:00
Jenkins
744447b3c0 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-07 04:17:14 +00:00
Yamagishi Kazutoshi
0c4ca3e549 Remove duplicate annotate (#5910)
* Remove duplicate annotate

* Remove blank line
2017-12-07 04:53:42 +01:00
Akihiko Odaki
c083816c24 Add embed_url to preview cards (#5775) 2017-12-07 03:37:43 +01:00
beatrix
81b0145759
Merge pull request #229 from glitch-soc/glitch-theme
Advanced Next-Level Flavours And Skins For Mastodon™
2017-12-06 17:44:07 -05:00
Jenkins
8ca91cef45 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-06 20:17:13 +00:00
Yamagishi Kazutoshi
b21db9bbde Using double splat operator (#5859) 2017-12-06 11:41:57 +01:00
Eugen Rochko
42bcbd36b7 Remove rabl dependency (#5894)
* Remove rabl dependency

* Replicate old Oj configuration
2017-12-06 15:04:49 +09:00
Eugen Rochko
d68868ca14
Lists redis clean-up (#5886)
* When list is deleted, remove feed from redis

* Clean up list feeds of inactive users
2017-12-05 23:20:27 +01:00
Eugen Rochko
e20895f251
Add list of lists component to web UI (#5811)
* Add list of lists component to web UI

* Add list adding

* Add list removing

* List editor modal

* Add API account search limited by following=true relation

* Rework list editor modal

* Remove mandatory pagination of GET /api/v1/lists/:id/accounts

* Adjust search input placeholder

* Fix rspec (#5890)

* i18n: (zh-CN) Add missing translations for #5811 (#5891)

* i18n: (zh-CN) yarn manage:translations -- zh-CN

* i18n: (zh-CN) Add missing translations for #5811

* Fix some issues

- Display loading/missing state for list timelines
- Order lists alphabetically in overview
- Fix async list editor reset
- Redirect to /lists after deleting unpinned list
- Redirect to / after pinning a list

* Remove dead list columns when a list is deleted or fetch returns 404
2017-12-05 23:02:27 +01:00
David Yip
d9800a5647
Merge branch 'gs-master' into glitch-theme 2017-12-04 11:07:01 -06:00
kibigo!
bc4fa6b198 Rename themes -> flavours ? ? 2017-12-03 23:26:40 -08:00
Jenkins
933eafdcd3 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-02 14:17:11 +00:00
Yamagishi Kazutoshi
74320971e2 Add invite filter (#5862) 2017-12-01 16:40:02 +01:00
kibigo!
541fe9b110 Skins support 2017-11-30 19:29:47 -08:00
Jenkins
ad46bc9772 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-30 03:17:12 +00:00
Eugen Rochko
85e97ecab6
Fix too many forwards (#5854)
* Avoid sending explicit Undo->Announce when original deleted

* Do not forward a reply back to the server that sent it

* Deduplicate inboxes of rebloggers' followers for delete forwarding

* Adjust test

* Fix wrong class, bad SQL, wrong variable, outdated comment
2017-11-30 03:50:05 +01:00
David Yip
95c270f5b1
Merge remote-tracking branch 'origin/master' into gs-master 2017-11-28 11:45:13 -06:00
Yamagishi Kazutoshi
15fab79cfa Show "expired" in invite index (#5838)
* Show "expired" in invite index

* "Force expire" -> "Deactivate"
2017-11-28 15:41:02 +01:00
aschmitz
eeaec39888 Allow hiding of reblogs from followed users (#5762)
* Allow hiding of reblogs from followed users

This adds a new entry to the account menu to allow users to hide
future reblogs from a user (and then if they've done that, to show
future reblogs instead).

This does not remove or add historical reblogs from/to the user's
timeline; it only affects new statuses.

The API for this operates by sending a "reblogs" key to the follow
endpoint. If this is sent when starting a new follow, it will be
respected from the beginning of the follow relationship (even if
the follow request must be approved by the followee). If this is
sent when a follow relationship already exists, it will simply
update the existing follow relationship. As with the notification
muting, this will now return an object ({reblogs: [true|false]}) or
false for each follow relationship when requesting relationship
information for an account. This should cause few issues due to an
object being truthy in many languages, but some modifications may
need to be made in pickier languages.

Database changes: adds a show_reblogs column (default true,
non-nullable) to the follows and follow_requests tables. Because
these are non-nullable, we use the existing MigrationHelpers to
perform this change without locking those tables, although the
tables are likely to be small anyway.

Tests included.

See also <https://github.com/glitch-soc/mastodon/pull/212>.

* Rubocop fixes

* Code review changes

* Test fixes

This patchset closes #648 and resolves #3271.

* Rubocop fix

* Revert reblogs defaulting in argument, fix tests

It turns out we needed this for the same reason we needed it in muting:
if nil gets passed in somehow (most usually by an API client not passing
any value), we need to detect and handle it.

We could specify a default in the parameter and then also catch nil, but
there's no great reason to duplicate the default value.
2017-11-28 15:00:35 +01:00
nullkal
1b57d4dd3a Fix account migration feature (#5837)
* Make removable account migration

* Fix error during update of account migration setting

* Add notice when update account migration setting
2017-11-28 14:31:23 +01:00
David Yip
7463d80ff4
Merge remote-tracking branch 'tootsuite/master' 2017-11-27 22:33:26 -06:00
Eugen Rochko
706e534455
Add UI for setting up account migration (#5832) 2017-11-27 22:47:06 +01:00
David Yip
e77c3996a5
Merge remote-tracking branch 'origin/master' into gs-master 2017-11-27 11:29:04 -06:00
Eugen Rochko
740f8a95a9
Add consumable invites (#5814)
* Add consumable invites

* Add UI for generating invite codes

* Add tests

* Display max uses and expiration in invites table, delete invite

* Remove unused column and redundant validator

- Default follows not used, probably bad idea
- InviteCodeValidator is redundant because RegistrationsController
  checks invite code validity

* Add admin setting to disable invites

* Add admin UI for invites, configurable role for invite creation

- Admin UI that lists everyone's invites, always available
- Admin setting min_invite_role to control who can invite people
- Non-admin invite UI only visible if users are allowed to

* Do not remove invites from database, expire them instantly
2017-11-27 16:07:59 +01:00
Jenkins
86f4f8e158 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-25 05:17:15 +00:00
William Pitcock
32987004c9 status: preserve visibility attribute when reblogging (infoleak fix) (#5789)
this should fix *all* remaining visibility-related mastodon ostatus infoleaks.
thanks to @csaurus@gnusocial.de for pointing out the infoleak.
2017-11-25 01:36:08 +01:00
abcang
269a445c0b Fix unnecessary order (#5807) 2017-11-24 18:41:07 +01:00
Jenkins
167fe2ab08 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-24 01:17:20 +00:00
Eugen Rochko
e84fecb7e9
Add logging of admin actions (#5757)
* Add logging of admin actions

* Update brakeman whitelist

* Log creates, updates and destroys with history of changes

* i18n: Update Polish translation (#5782)

Signed-off-by: Marcin Mikołajczak <me@m4sk.in>

* Split admin navigation into moderation and administration

* Redesign audit log page

* 🇵🇱 (#5795)

* Add color coding to audit log

* Change dismiss->resolve, log all outcomes of report as resolve

* Update terminology (e-mail blacklist) (#5796)

* Update terminology (e-mail blacklist)

imho looks better

* Update en.yml

* Fix code style issues

* i18n-tasks normalize
2017-11-24 02:05:53 +01:00
Jenkins
a36a2c1796 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-19 15:17:12 +00:00
abcang
53e95c4efc Fix N+1 at notification (#5752) 2017-11-19 15:32:48 +01:00
David Yip
d083f7741a Merge remote-tracking branch 'tootsuite/master' 2017-11-19 01:30:29 -06:00
Yamagishi Kazutoshi
bfdcf76a64 Add “staff” badge to admin user pages (#5735)
* Allow staff badge to change from setting to hidden

* i18n: Update Polish translation
2017-11-19 04:13:57 +01:00
Eugen Rochko
58cede4808
Profile redirect notes (#5746)
* Serialize moved accounts into REST and ActivityPub APIs

* Parse federated moved accounts from ActivityPub

* Add note about moved accounts to public profiles

* Add moved account message to web UI

* Fix code style issues
2017-11-18 19:39:02 +01:00
beatrix
dec960c828
Merge pull request #208 from yipdw/keyword-mute
Run keyword mutes on hashtags
2017-11-18 12:06:40 -05:00
David Yip
130aa90d55 Update annotations on Follow, FollowRequest, and Mute.
Follow and FollowRequest had conflicts in their schema annotations, so I
ran latest migrations and let annotate_models fix them up.
2017-11-17 17:41:15 -06:00
David Yip
1ab12ba38e Merge remote-tracking branch 'origin/master' into merge-upstream 2017-11-17 17:40:00 -06:00
Eugen Rochko
24cafd73a2
Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
David Yip
b28b405b97 Merge remote-tracking branch 'origin/master' into gs-master 2017-11-16 22:52:37 -06:00
Eugen Rochko
3e90987c8b Fix some rubocop style issues (#5730) 2017-11-17 10:06:26 +09:00
Eugen Rochko
2151fd3150
Display moderator badge (green), admin badge changed to red (#5728) 2017-11-17 01:22:38 +01:00
ysksn
9e3d24a150 Fix AccountFinderConcern::AccountFinder#with_usernames (#5722) 2017-11-16 12:13:03 +01:00
Surinna Curtis
35fbdc36f9 Merge tootsuite/master at 3023725936 2017-11-16 01:21:16 -06:00
David Yip
08652baab0 Replace =~ with #matches?. #208.
=~ made sense when we were passing it through to a regex, but we're no
longer doing that: TagMatcher looks at individual tags and returns a
value that *looks* like what you get out of #=~ but really isn't that
meaningful.  Probably a good idea to not subvert convention like this
and instead use a name with guessable intent.
2017-11-15 18:27:25 -06:00
David Yip
8fc54890e5 Prefix cache keys with the matcher type. #208.
We already know about one regex limitation, which is that they cannot
segment words in e.g. Japanese, Chinese, or Thai.  It may also end up
that regex matching is too slow compared to other methods.

However, the regex is an implementation detail.  We still want the
ability to switch between "occurs anywhere" and "match whole word", and
caching the matcher result is likely to still be important (since the
matcher itself won't change nearly as often as status ingress rate).
Therefore, we ought to be able to change the cache keys to reflect a
change of data structure.

(Old cache keys expire within minutes, so they shouldn't be too big of
an issue.  Old cache keys could also be explicitly removed by an
instance administrator.)
2017-11-15 18:12:31 -06:00
David Yip
cb4ef24ac9 Match keyword mute filter on hashtags. #208.
It is reasonable to expect someone to enter #foo to mute hashtag #foo.
However, tags are recorded on statuses without the preceding #.
To adjust for this, we build a separate tag matcher and use
Tag::HASHTAG_RE to extract a hashtag from the hashtag syntax.
2017-11-15 18:12:29 -06:00
beatrix
04508868b0
Merge pull request #212 from aschmitz/feat/mute-reblogs
Allow hiding reblogs on a per-follow basis
2017-11-15 12:01:17 -05:00
ysksn
19e8b861a2 Delegate some methods of User to @settings (#5706)
* Move some tests of User into Settings::ScopedSettings

* Add a test for User@settings
2017-11-15 16:05:20 +01:00
Surinna Curtis
031a5a8f92 Optional notification muting (#5087)
* Add a hide_notifications column to mutes

* Add muting_notifications? and a notifications argument to mute!

* block notifications in notify_service from hard muted accounts

* Add specs for how mute! interacts with muting_notifications?

* specs testing that hide_notifications in mutes actually hides notifications

* Add support for muting notifications in MuteService

* API support for muting notifications (and specs)

* Less gross passing of notifications flag

* Break out a separate mute modal with a hide-notifications checkbox.

* Convert profile header mute to use mute modal

* Satisfy eslint.

* specs for MuteService notifications params

* add trailing newlines to files for Pork :)

* Put the label for the hide notifications checkbox in a label element.

* Add a /api/v1/mutes/details route that just returns the array of mutes.

* Define a serializer for /api/v1/mutes/details

* Add more specs for the /api/v1/mutes/details endpoint

* Expose whether a mute hides notifications in the api/v1/relationships endpoint

* Show whether muted users' notifications are muted in account lists

* Allow modifying the hide_notifications of a mute with the /api/v1/accounts/:id/mute endpoint

* make the hide/unhide notifications buttons work

* satisfy eslint

* In probably dead code, replace a dispatch of muteAccount that was skipping the modal with launching the mute modal.

* fix a missing import

* add an explanatory comment to AccountInteractions

* Refactor handling of default params for muting to make code cleaner

* minor code style fixes oops

* Fixed a typo that was breaking the account mute API endpoint

* Apply white-space: nowrap to account relationships icons

* Fix code style issues

* Remove superfluous blank line

* Rename /api/v1/mutes/details -> /api/v2/mutes

* Don't serialize "account" in MuteSerializer

Doing so is somewhat unnecessary since it's always the current user's account.

* Fix wrong variable name in api/v2/mutes

* Use Toggle in place of checkbox in the mute modal.

* Make the Toggle in the mute modal look better

* Code style changes in specs and removed an extra space

* Code review suggestions from akihikodaki

Also fixed a syntax error in tests for AccountInteractions.

* Make AddHideNotificationsToMute Concurrent

It's not clear how much this will benefit instances in practice, as the
number of mutes tends to be pretty small, but this should prevent any
blocking migrations nonetheless.

* Fix up migration things

* Remove /api/v2/mutes
2017-11-15 03:56:41 +01:00
Yamagishi Kazutoshi
20150659e6 Add uniqueness to block email domains (#5692) 2017-11-14 20:37:17 +01:00
David Yip
656d54e945 Maintain case-insensitivity when merging multiple matchers (#213)
When given two regexps, Regexp.union preserves the options set (or not
set) on each regex; this meant that none of the multiline (m),
case-insensitivity (i), or extended syntax (x) options were set.  Our
regexps are written expecting the m, i, and x options were set on all of
them, so we need to make sure that we preserve that behavior.
2017-11-13 11:06:02 -06:00
Daniel Hunsaker
cf7e840990 Update model annotations to use BIGINT for IDs (#5461)
All the migrations have been updated to use BIGINTs for ID fields in the DB, but ActiveRecord needs to be told to treat those values as BIGINT as well. This PR does that.
2017-11-12 16:18:50 +01:00
aschmitz
5128c4261e Updates per code review
Thanks, @valerauko!
2017-11-11 14:37:23 -06:00
Eugen Rochko
7bb8b0b2fc
Add moderator role and add pundit policies for admin actions (#5635)
* Add moderator role and add pundit policies for admin actions

* Add rake task for turning user into mod and revoking it again

* Fix handling of unauthorized exception

* Deliver new report e-mails to staff, not just admins

* Add promote/demote to admin UI, hide some actions conditionally

* Fix unused i18n
2017-11-11 20:23:33 +01:00
aschmitz
b95c48748c Per-user reblog hiding implementation/fixes/tests
Note that this will only hide/show *future* reblogs by a user, and does
nothing to remove/add reblogs that are already in the timeline. I don't
think that's a particularly confusing behavior, and it's a lot easier
to implement (similar to mutes, I believe).
2017-11-10 22:04:54 -06:00
aschmitz
4944515020 "Show reblogs" per-follower UI/database changes
TODO:

* Tests (particularly for FollowRequests).
* Anything to respect the setting when putting reblogs in timelines.
2017-11-09 08:41:10 -06:00
ysksn
54b42901df Add and Remove tests for FollowRequest (#5622)
* Add a test for FollowRequest#authorize!

* Remove tests

There is no need to test
ActiveModel::Validations::ClassMethods#validates.

* Make an alias of destroy! as reject!

Instead of defining the method,
make an alias of destroy! as reject! because of reducing test.
2017-11-09 14:36:52 +01:00
ThibG
5d5c0f4f43 Twidere mention workaround (#5552)
* Work around Twidere and Tootdon bug

Tootdon and Twidere construct @user@domain handles from mentions in toots based
solely on the mention text and account URI's domain without performing any
webfinger call or retrieving account info from the Mastodon server.

As a result, when a remote user has WEB_DOMAIN ≠ LOCAL_DOMAIN, Twidere and
Tootdon will construct the mention as @user@WEB_DOMAIN. Now, this will usually
resolve to the correct account (since the recommended configuration is to have
WEB_DOMAIN perform webfinger redirections to LOCAL_DOMAIN) when processing
mentions, but won't do so when displaying them (as it does not go through the
whole account resolution at that time).

This change rewrites mentions to the resolved account, so that displaying the
mentions will work.

* Use lookbehind instead of non-capturing group in MENTION_RE

Indeed, substitutions with the previous regexp would erroneously eat any
preceding whitespace, which would lead to concatenated mentions in the
previous commit.

Note that users will “lose” up to one character space per mention for their
toots, as that regexp is also used to remove the domain-part of mentioned
users for character counting purposes, and it also erroneously removed the
preceding character if it was a space.
2017-11-07 19:08:14 +01:00
Eugen Rochko
1032f3994f
Add ability to disable login and mark accounts as memorial (#5615)
Fix #5597
2017-11-07 19:06:44 +01:00
nullkal
b6e2e999bd Show the local couterpart of emoji when it exists in /admin/custom_emojis (#5467)
* Show the local couterpart of emoji when it exists in admin/custom_emojis

* Fix indentation

* Fix error

* Add class table-action-link to Overwrite link

* Make it enable to overwrite emojis

* Make Code Climate happy
2017-11-07 14:49:32 +01:00
David Yip
870d71b78b Merge branch 'master' into gs-master 2017-10-27 09:45:25 -05:00
nullkal
781105293c Feature: Unlisted custom emojis (#5485) 2017-10-27 16:11:30 +02:00
beatrix
a2612d0d38 Merge pull request #179 from glitch-soc/keyword-mute
Keyword muting
2017-10-25 17:37:48 -04:00
David Yip
42f2045c21 Merge remote-tracking branch 'STJrInuyasha/feature/direct-timeline' into gs-direct-timeline 2017-10-25 16:01:20 -05:00
David Yip
e40fe4092d Remove nil check in Glitch::KeywordMute#=~.
@regex can no longer be nil, so we don't need to check it.
2017-10-24 19:03:59 -05:00
David Yip
f5a3283976 Switch to Regexp.union for building the mute expression.
Also make the keyword-building methods private: they always probably
should have been private, but now I have encoded enough fun and games
into them that it now seems wrong for them to *not* be private.
2017-10-24 18:31:34 -05:00
David Yip
8410d33b49 Only cache the regex text, not the regex itself.
It is possible to cache a Regexp object, but I'm not sure what happens
if e.g. that object remains in cache across two different Ruby versions.
Caching a string seems to raise fewer questions.
2017-10-23 19:31:59 -05:00
Matthew Walsh
3db80f75a6 Added a timeline for Direct statuses
* Lists all Direct statuses you've sent and received
* Displayed in Getting Started
* Streaming server support for direct TL
2017-10-22 18:35:14 -07:00
David Yip
af8f06413e KeywordMute matcher: more closely mimic Regexp#=~ behavior.
Regexp#=~ returns nil if it does not match.  An empty mute set does not
match any status, so KeywordMute::Matcher#=~ ought to return nil also.
2017-10-22 01:12:21 -05:00
David Yip
4b68e82a19 Don't add \b to whole-word keywords that don't start with word characters.
Ditto for ending with \b.

Consider muting the phrase "(hot take)".  I stipulate it is reasonable
to enter this with the default "match whole word" behavior.  Under the
old behavior, this would be encoded as

    \b\(hot\ take\)\b

However, if \b is before the first character in the string and the first
character in the string is not a word character, then the match will
fail.  Ditto for after.  In our example, "(" is not a word character, so
this will not match statuses containing "(hot take)", and that's a very
surprising behavior.

To address this, we only add leading and trailing \b to keywords that
start or end with word characters.
2017-10-22 00:38:54 -05:00
David Yip
ad86c86fa8 Apply keyword mutes to reblogs. 2017-10-21 15:44:47 -05:00
David Yip
670e6a33f8 Move KeywordMute into Glitch namespace.
There are two motivations for this:

1. It looks like we're going to add other features that require
   server-side storage (e.g. user notes).

2. Namespacing glitchsoc modifications is a good idea anyway: even if we
   do not end up doing (1), if upstream introduces a keyword-mute feature
   that also uses a "KeywordMute" model, we can avoid some merge
   conflicts this way and work on the more interesting task of
   choosing which implementation to use.
2017-10-21 14:54:36 -05:00
David Yip
4a64181461 Allow keywords to match either substrings or whole words.
Word-boundary matching only works as intended in English and languages
that use similar word-breaking characters; it doesn't work so well in
(say) Japanese, Chinese, or Thai.  It's unacceptable to have a feature
that doesn't work as intended for some languages.  (Moreso especially
considering that it's likely that the largest contingent on the Mastodon
bit of the fediverse speaks Japanese.)

There are rules specified in Unicode TR29[1] for word-breaking across
all languages supported by Unicode, but the rules deliberately do not
cover all cases.  In fact, TR29 states

    For example, reliable detection of word boundaries in languages such
    as Thai, Lao, Chinese, or Japanese requires the use of dictionary
    lookup, analogous to English hyphenation.

So we aren't going to be able to make word detection work with regexes
within Mastodon (or glitchsoc).  However, for a first pass (even if it's
kind of punting) we can allow the user to choose whether they want word
or substring detection and warn about the limitations of this
implementation in, say, docs.

[1]: https://unicode.org/reports/tr29/
     https://web.archive.org/web/20171001005125/https://unicode.org/reports/tr29/
2017-10-21 14:54:36 -05:00
David Yip
b4b657eb1d Invalidate cached matcher objects on KeywordMute commit. #164. 2017-10-21 14:54:36 -05:00
David Yip
693c66dfde Use more idiomatic string concatentation. #164.
The intent of the previous concatenation was to minimize object
allocations, which can end up being a slow killer.  However, it turns
out that under MRI 2.4.x, the shove-strings-in-an-array-and-join method
is not only arguably more common but (in this particular case) actually
allocates *fewer* objects than the string concatenation.

Or, at least, that's what I gather by running this:

    words = %w(palmettoes nudged hibernation bullish stockade's tightened Hades
    Dixie's formalize superego's commissaries Zappa's viceroy's apothecaries
    tablespoonful's barons Chennai tollgate ticked expands)

    a = Account.first

    KeywordMute.transaction do
      words.each { |w| KeywordMute.create!(keyword: w, account: a) }

      GC.start

      s1 = GC.stat

      re = String.new.tap do |str|
        scoped = KeywordMute.where(account: a)
        keywords = scoped.select(:id, :keyword)
        count = scoped.count

        keywords.find_each.with_index do |kw, index|
          str << Regexp.escape(kw.keyword.strip)
          str << '|' if index < count - 1
        end
      end

      s2 = GC.stat

      puts s1.inspect, s2.inspect

      raise ActiveRecord::Rollback
    end

vs this:

    words = %w( palmettoes nudged hibernation bullish stockade's tightened Hades Dixie's
    formalize superego's commissaries Zappa's viceroy's apothecaries tablespoonful's
    barons Chennai tollgate ticked expands
    )

    a = Account.first

    KeywordMute.transaction do
      words.each { |w| KeywordMute.create!(keyword: w, account: a) }

      GC.start

      s1 = GC.stat

      re = [].tap do |arr|
        KeywordMute.where(account: a).select(:keyword, :id).find_each do |m|
          arr << Regexp.escape(m.keyword.strip)
        end
      end.join('|')

      s2 = GC.stat

      puts s1.inspect, s2.inspect

      raise ActiveRecord::Rollback
    end

Using rails r, here is a comparison of the total_allocated_objects and
malloc_increase_bytes GC stat data:

                 total_allocated_objects        malloc_increase_bytes
string concat    3200241 -> 3201428 (+1187)     1176 -> 45216 (44040)
array join       3200380 -> 3201299 (+919)      1176 -> 36448 (35272)
2017-10-21 14:54:36 -05:00
David Yip
a4851100fd Make use of the regex attr_reader. #164.
It would also have been valid to get rid of the attr_reader, but I like
being able to reach inside KeywordMute::Matcher without resorting to
instance_variable_get tomfoolery.
2017-10-21 14:54:36 -05:00
David Yip
603cf02b70 Rework KeywordMute interface to use a matcher object; spec out matcher. #164.
A matcher object that builds a match from KeywordMute data and runs it
over text is, in my view, one of the easier ways to write examples for
this sort of thing.
2017-10-21 14:54:36 -05:00
David Yip
4745d6eeca Spec out KeywordMute interface. #164. 2017-10-21 14:54:21 -05:00
David Yip
9093e2de7a Add KeywordMute model.
Gist of the proposed keyword mute implementation:

Keyword mutes are represented server-side as one keyword per record.
For each account, there exists a keyword regex that is generated as one
big alternation of all keywords.  This regex is cached (in Redis, I
guess) so we can quickly get it when filtering in FeedManager.
2017-10-21 14:53:41 -05:00
David Yip
dbb1fce94d Merge remote-tracking branch 'upstream/master' into gs-master 2017-10-16 09:23:59 -05:00
Nolan Lawson
fa0be3f834 Add option to reduce motion (#5393)
* Add option to reduce motion

* Use HOC to wrap all Motion calls

* fix case-sensitive issue

* Avoid updating too frequently

* Get rid of unnecessary change to _simple_status.html.haml
2017-10-16 09:36:15 +02:00
David Yip
6cd5b3bbe5 Merge remote-tracking branch 'origin/master' into gs-master 2017-10-16 01:29:02 -05:00
unarist
a1c54220e8 Optimize Status#permitted_for 500x (account timeline) (#5373)
The main change of this PR is removing `order by visibility` hack.

This was introduced to force using of `index_statuses_on_account_id` instead of PK index, but it seems no longer needed probably due to `index_statuses_on_account_id_id`. Removing this avoids reading all rows, so really improves first fetching of the user who has lot of statuses.

I have also changed JOIN to IN + subquery, which slightly faster in most cases.
2017-10-13 16:53:43 +02:00
Eugen Rochko
b8db386e05 Fix UserTrackingConcern firing on every request, optimize some queries (#5368)
- For some reason, :if option on before_action did not work. It got
  executed every time, returned false, and the action run anyway,
  which led to the current_sign_in_at and sign_in_count being
  updated on every request
- Return "do not filter" early in FeedManager#filter_from_home? if
  the status is authored by receiver. Usually this method is not
  called for own statuses at all, but it is called when Feed#get
  uses the database
- Return early if #reload_stale_associations! has nothing to load
  to save a database query with WHERE 1=0
2017-10-13 16:44:29 +02:00
Lex Alexander
b8bae96647 Retoot count increases without reason (#5363)
* Retoot count increases without reason

-The store_uri method for Statuses was being called on after_create and causing reblogs to be incremented twice.
-This calls it when the transaction is finished by using after_create_commit.
-Fixes #4916.

* Added test case for after_create_commit callback for checking reblog count.

* Rewrote test to keep original, but added one for only the after_create_commit callback.
2017-10-13 02:52:09 +02:00
kibigo!
8d6b9ba494 Merge upstream 2.0ish #165 2017-10-11 10:43:10 -07:00
nullkal
6c54d2e583 foreign_key, non-nullable, dependent: destroy in account_moderation_notes (#5294)
* Add foreign key constraint to column `account` in `account_moderation_notes`

* Change account_id and target_account_id to non-nullable in account_moderation_notes

* Add dependent: :destroy to account and target_account in account_moderation_notes
2017-10-10 13:12:17 +02:00
beatrix
f0a2a6c875 try to tighten up local only toot stuff, like... properly (#163)
* try to tighten up local only toot stuff, like... properly

* try to un-break tests
2017-10-09 09:56:17 -04:00
Eugen Rochko
0717d9b3e6 Set snowflake IDs for backdated statuses (#5260)
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity
- Skip for statuses coming from inbox, aka delivered in real-time
- Skip for statuses that claim to be from the future
2017-10-08 17:34:34 +02:00
DJ Sundog
979b0d66a7 update indentation 2017-10-07 13:53:46 -07:00
DJ Sundog
6ca03a7f58 add faststart to audio transcoding 2017-10-07 19:59:22 +00:00
DJ Sundog
96ba3482b9 adding support for audio uploads, transcoded to mp4 videos 2017-10-07 19:54:10 +00:00
nullkal
633426b261 Add moderation note (#5240)
* Add moderation note

* Add frozen_string_literal

* Make rspec pass
2017-10-07 20:26:43 +02:00
Eugen Rochko
3a3475450e Encode custom emojis as resolveable objects in ActivityPub (#5243)
* Encode custom emojis as resolveable objects in ActivityPub

* Improve code style
2017-10-07 17:43:42 +02:00
Eugen Rochko
49cc0eb3e7 Improve admin UI for custom emojis, add copy/disable/enable (#5231) 2017-10-05 23:42:05 +02:00
Eugen Rochko
b9c76e2edb When processing custom emoji, ensure a non-animated version exists (#5230)
Use the non-animated version in web UI, but return both in API
2017-10-05 23:41:47 +02:00
utam0k
b3af3f9f8c Implement EmailBlackList (#5109)
* Implement BlacklistedEmailDomain

* Use Faker::Internet.domain_name

* Remove note column

* Add frozen_string_literal comment

* Delete unnecessary codes

* Sort alphabetically

* Change of wording

* Rename BlacklistedEmailDomain to EmailDomainBlock
2017-10-04 15:16:10 +02:00
aschmitz
468523f4ad Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356faf2d5c7e66e3ec715502796c875cd from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
aschmitz
97c02c3389 Make IdsToBigints (mostly!) non-blocking (#5088)
* Make IdsToBigints (mostly!) non-blocking

This pulls in GitLab's MigrationHelpers, which include code to make
column changes in ways that Postgres can do without locking. In general,
this involves creating a new column, adding an index and any foreign
keys as appropriate, adding a trigger to keep it populated alongside
the old column, and then progressively copying data over to the new
column, before removing the old column and replacing it with the new
one.

A few changes to GitLab's MigrationHelpers were necessary:

* Some changes were made to remove dependencies on other GitLab code.
* We explicitly wait for index creation before forging ahead on column
  replacements.
* We use different temporary column names, to avoid running into index
  name length limits.
* We rename the generated indices back to what they "should" be after
  replacing columns.
* We rename the generated foreign keys to use the new column names when
  we had to create them. (This allows the migration to be rolled back
  without incident.)

# Big Scary Warning

There are two things here that may trip up large instances:

1. The change for tables' "id" columns is not concurrent. In
   particular, the stream_entries table may be big, and does not
   concurrently migrate its id column. (On the other hand, x_id type
   columns are all concurrent.)
2. This migration will take a long time to run, *but it should not
   lock tables during that time* (with the exception of the "id"
   columns as described above). That means this should probably be run
   in `screen` or some other session that can be run for a long time.
   Notably, the migration will take *longer* than it would without
   these changes, but the website will still be responsive during that
   time.

These changes were tested on a relatively large statuses table (256k
entries), and the service remained responsive during the migration.
Migrations both forward and backward were tested.

* Rubocop fixes

* MigrationHelpers: Support ID columns in some cases

This doesn't work in cases where the ID column is referred to as a
foreign key by another table.

* MigrationHelpers: support foreign keys for ID cols

Note that this does not yet support foreign keys on non-primary-key
columns, but Mastodon also doesn't yet have any that we've needed to
migrate.

This means we can perform fully "concurrent" migrations to change ID
column types, and the IdsToBigints migration can happen with effectively
no downtime. (A few operations require a transaction, such as renaming
columns or deleting them, but these transactions should not block for
noticeable amounts of time.)

The algorithm for generating foreign key names has changed with this,
and therefore all of those changed in schema.rb.

* Provide status, allow for interruptions

The MigrationHelpers now allow restarting the rename of a column if it
was interrupted, by removing the old "new column" and re-starting the
process.

Along with this, they now provide status updates on the changes which
are happening, as well as indications about when the changes can be
safely interrupted (when there are at least 10 seconds estimated to be
left before copying data is complete).

The IdsToBigints migration now also sorts the columns it migrates by
size, starting with the largest tables. This should provide
administrators a worst-case scenario estimate for the length of
migrations: each successive change will get faster, giving admins a
chance to abort early on if they need to run the migration later. The
idea is that this does not force them to try to time interruptions
between smaller migrations.

* Fix column sorting in IdsToBigints

Not a significant change, but it impacts the order of columns in the
database and db/schema.rb.

* Actually pause before IdsToBigints
2017-10-02 21:28:59 +02:00
Eugen Rochko
f4ca116ea8 After 7 days of repeated delivery failures, give up on inbox (#5131)
- A successful delivery cancels it out
- An incoming delivery from account of the inbox cancels it out
2017-09-29 03:16:20 +02:00
Eugen Rochko
887cd94e96 Increase attachment descriptions to 420 characters (#5139)
Blaze it
2017-09-29 02:30:00 +02:00
Eugen Rochko
76f360c625 If HTTP signature is wrong and webfinger cache is stale, retry with resolve (#5129)
If the signature could not be verified and the webfinger of the account
was last retrieved longer than the cache period, try re-resolving the
account and then attempting to verify the signature again
2017-09-28 17:50:14 +02:00
Eugen Rochko
4ec1771165 Add ability to specify alternative text for media attachments (#5123)
* Fix #117 - Add ability to specify alternative text for media attachments

- POST /api/v1/media accepts `description` straight away
- PUT /api/v1/media/:id to update `description` (only for unattached ones)
- Serialized as `name` of Document object in ActivityPub
- Uploads form adjusted for better performance and description input

* Add tests

* Change undo button blend mode to difference
2017-09-28 15:31:31 +02:00
Ondřej Hruška
daad07b1d5
Merge remote-tracking branch 'origin/better-themeing' into merging-upstream 2017-09-28 09:20:59 +02:00
Ondřej Hruška
83bda6c1a8
Merge commit 'b9f59ebcc68e9da0a7158741a1a2ef3564e1321e' into merging-upstream 2017-09-28 09:18:35 +02:00
Ondřej Hruška
9330ea1f4d
Merge commit '4aea3f88a6d30f102a79c2da7fcfac96465ba1a8' into merging-upstream 2017-09-28 09:12:17 +02:00
Eugen Rochko
3caf0ba923 Fix empty query sent to postgres for custom emojis (#5121) 2017-09-27 04:14:03 +02:00
kibigo!
86e617a839 Better themeing support!! 2017-09-25 19:24:32 -07:00
unarist
5c82c2b75f Increase max height of preview card image (#5092)
We added horizontal layout to preview card for wide image. However, max height of the thumbnail is still limited to 120px and it makes nearly square images to too small for that layout.

This PR increases max height as well as max width.
2017-09-25 17:10:49 +02:00
Eugen Rochko
1e02ba111a Add emoji autosuggest (#5053)
* Add emoji autosuggest

Some credit goes to glitch-soc/mastodon#149

* Remove server-side shortcode->unicode conversion

* Insert shortcode when suggestion is custom emoji

* Remove remnant of server-side emojis

* Update style of autosuggestions

* Fix wrong emoji filenames generated in autosuggest item

* Do not lazy load emoji picker, as that no longer works

* Fix custom emoji autosuggest

* Fix multiple "Custom" categories getting added to emoji index, only add once
2017-09-23 14:47:32 +02:00
Eugen Rochko
293972f716 New API: GET /api/v1/custom_emojis to get a server's custom emojis (#5051) 2017-09-23 01:57:23 +02:00
Eugen Rochko
9c8e602163 Fix custom emojis not detected when used in content warning (#5049) 2017-09-23 01:50:17 +02:00
Akihiko Odaki
0de82dd316 Do not filter statuses with unknown languages (#5045) 2017-09-22 19:33:17 +02:00