Commit Graph

443 Commits

Author SHA1 Message Date
aschmitz 669fe9ee06 Change IDs to strings rather than numbers in API JSON output (#5019)
* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Back out RelationshipsController Change

This was made to make a test a bit less flakey, but has nothing to
do with this branch.

* Change internal streaming payloads to stringified IDs as well

Per
https://github.com/tootsuite/mastodon/pull/5019#issuecomment-330736452
we need these changes to send deleted status IDs as strings, not
integers.
2017-09-20 14:53:48 +02:00
Akihiko Odaki bb4d005a83 Introduce OStatus::TagManager (#5008) 2017-09-19 18:08:08 +02:00
Eugen Rochko 41e6c8b151 Fix incomplete account records being read (#4998)
* Fix incomplete account records being read

- Put account processing into redis lock
- Do not save until record is complete

* Fix spaces
2017-09-19 06:53:16 +02:00
Eugen Rochko 09a94b575e Admin interface for listing, adding and removing custom emojis (#5002)
* Admin interface for listing, adding and removing custom emojis

* Only display local ones in the list
2017-09-19 03:52:38 +02:00
unarist 1eab53ee10 Fix an error when actor json couldn't be fetched in ResolveRemoteAccountService (#4979)
* Fix an error when actor json couldn't be fetched in ResolveRemoteAccountService

* Add specs
2017-09-17 11:54:23 +02:00
Akihiko Odaki 48d77ea1eb Fix filterable_languages method of SettingsHelper (#4966) 2017-09-16 14:59:41 +02:00
ふぁぼ原 3816943e6b Enable to recognize most kinds of characters as URL paths (#4941) 2017-09-14 18:03:20 +02:00
Eugen Rochko 596dab06e9 Support OpenGraph video embeds (#4897)
* Support OpenGraph video embeds

It's not really OpenGraph, it's twitter:player property, but it's
not OEmbed so that fits. For example, this allows Twitch clips to
be displayed as embeds.

Also, fixes glitch-soc/mastodon#135

* Fix invalid OpenGraph cards being saved through attaching and
revisit URLs after 14 days
2017-09-14 04:11:36 +02:00
ThibG af00220d79 Fix refollowing (#4931)
* Make RefollowWorker ActivityPub-only to avoid potential identifier mismatches

* Don't call RefollowWorker on new accounts
2017-09-14 00:05:25 +02:00
Eugen Rochko b9d241c6f5 Fix #4917 - Add missing suspend checks (#4921) 2017-09-13 11:05:02 +02:00
ThibG f29918e707 [WiP] Whenever a remote keypair changes, unfollow them and re-subscribe to … (#4907)
* Whenever a remote keypair changes, unfollow them and re-subscribe to them

In Mastodon (it could be different for other OStatus or AP-enabled software),
a keypair change is indicative of whole user (or instance) data loss. In this
situation, the “new” user might be different, and almost certainly has an empty
followers list. In this case, Mastodon instances will disagree on follower
lists, leading to unreliable delivery and “shadow followers”, that is users
believed by a remote instance to be followers, without the affected user
knowing.

Drawbacks of this change are:
1. If an user legitimately changes public key for some reason without losing
   data (not possible in Mastodon at the moment), they will have their remote
   followers unsubscribed/re-subscribed needlessly.
2. Depending of the number of remote followers, this may generate quite some
   traffic.
3. If the user change is an attempt at usurpation, the remote followers will
   unknowingly follow the usurper. Note that this is *not* a change of
   behavior, Mastodon already behaves like that, although delivery might be
   unreliable, and the usurper would not have known the former user's
   followers.

* Rename ResubscribeWorker to RefollowWorker

* Process followers in batches
2017-09-12 23:10:40 +02:00
Eugen Rochko 7d7844a47f Default follows for new users (#4871)
When a new user confirms their e-mail, bootstrap their home timeline
by automatically following a set of accounts. By default, all local
admin accounts (that are unlocked). Can be customized by new admin
setting (comma-separated usernames, local and unlocked only)
2017-09-10 09:58:38 +02:00
Eugen Rochko 90712d4293 Fix errors preventing UnsubscribeService from working (#4866) 2017-09-09 17:36:27 +02:00
Eugen Rochko 9e15eeec63 Add missing reject_media check before avatar download via ActivityPub (#4862) 2017-09-09 13:41:45 +02:00
Eugen Rochko 4b460bc571 Fix #4852 - Check if already requested from FollowService (#4855) 2017-09-09 02:02:44 +02:00
Eugen Rochko 7ca173be47 Fix #4850 - When visibility missing from API call to toot, fallback to user preference (#4861) 2017-09-09 02:02:29 +02:00
Eugen Rochko a4caa7eb62 Fetch statuses/following/followers numbers from ActivityPub collections (#4840) 2017-09-08 12:00:17 +02:00
Eugen Rochko e821c00e74 Fix mentions in direct statuses not being delivered via AP (#4806) 2017-09-05 20:55:25 +02:00
Eugen Rochko 9b50a9dd83 Fix some ActivityPub JSON bugs (#4796)
- Fix assumption that `url` is always a string. Handle it if it's an
  array of strings, array of objects, object, or string, both for
  accounts and for objects
- `sharedInbox` is actually supposed to be under `endpoints`, handle
  both cases and adjust the serializer
2017-09-04 18:26:33 +02:00
Eugen Rochko 37fdddd927 Rename "locked" to "manuallyApprovesFollowers" in ActivityPub (#4779)
See: <https://www.w3.org/wiki/Activity_Streams_extensions#as:manuallyApprovesFollowers>
2017-09-02 23:13:35 +02:00
Eugen Rochko d3b6746173 Make "unfollow" undo pending outgoing follow request too (#4781)
* Make "unfollow" undo pending outgoing follow request too

* Add cancel button to web UI when awaiting follow request approval

* Make the hourglass button do the cancelling
2017-09-02 20:44:41 +02:00
Eugen Rochko 1b5806b744 Define missing JSON-LD properties (#4767)
Using _: property names is discouraged, as in the future,
canonicalization may throw an error when encountering that instead
of discarding it silently like it does now.

We are defining some ActivityStreams properties which we expect
to land in ActivityStreams eventually, to ensure that future versions
of Mastodon will remain compatible with this even once that happens.
Those would be `locked`, `sensitive` and `Hashtag`

We are defining a custom context inline for some properties which we
do not expect to land in any other context. `atomUri`, `inReplyToAtomUri`
and `conversation` are part of the custom defined OStatus context.
2017-09-02 14:01:23 +02:00
Eugen Rochko 390bfec6da Avoid sending some ActivityPub payloads if the receiver will get them through distribution (#4739) 2017-09-01 21:26:01 +02:00
Eugen Rochko 7dc5035031 Make PreviewCard records reuseable between statuses (#4642)
* Make PreviewCard records reuseable between statuses

**Warning!** Migration truncates preview_cards tablec

* Allow a wider thumbnail for link preview, display it in horizontal layout (#4648)

* Delete preview cards files before truncating

* Rename old table instead of truncating it

* Add mastodon:maintenance:remove_deprecated_preview_cards

* Ignore deprecated_preview_cards in schema definition

* Fix null behaviour
2017-09-01 16:20:16 +02:00
unarist f7937d903c Don't process ActivityPub payload if signature is invalid (#4752)
* Don't process ActivityPub payload if signature is invalid

* Fix style issue
2017-08-31 17:18:49 +02:00
James 6b2be5dbfb Guarantee Subscription service first account has proper URL details (#4732)
* Guarantee Subscription service first account has proper URL details

Subscription Service potentially could break if the first user suspended
themselves, creating a situation where the urls that populate throughout
subscription service's PuSH request would cause the remote API to throw 503 errors.

Guaranteeing that the first account picked is not suspended prevents this problem.

* Fix style issue
2017-08-31 15:44:00 +02:00
Eugen Rochko 7b8f262840 Forward ActivityPub creates that reply to local statuses (#4709)
* Forward ActivityPub creates that reply to local statuses

* Fix test

* Fix wrong signers
2017-08-30 15:37:02 +02:00
Eugen Rochko 4c76402ba1 Serialize ActivityPub alternate link into OStatus deletes, handle it (#4730)
Requires moving Atom rendering from DistributionWorker (where
`stream_entry.status` is already nil) to inline (where
`stream_entry.status.destroyed?` is true) and distributing that.

Unfortunately, such XML renderings can no longer be easily chained
together into one payload of n items.
2017-08-29 16:11:05 +02:00
unarist 7876aed134 Fix deletion of status which has been reblogged (#4728) 2017-08-28 21:38:59 +02:00
Eugen Rochko 2a2698e450 Add ActivityPub serializer for Undo of Announce (#4703) 2017-08-26 15:32:40 +02:00
Eugen Rochko 00840f4f2e Add handling of Linked Data Signatures in payloads (#4687)
* Add handling of Linked Data Signatures in payloads

* Add a way to sign JSON, fix canonicalization of signature options

* Fix signatureValue encoding, send out signed JSON when distributing

* Add missing security context
2017-08-26 13:47:38 +02:00
masarakki 649a20ab46 authorize-follow-requests-after-unlocking (#4658) 2017-08-26 12:40:03 +02:00
Eugen Rochko cf615abbf9 Add configuration to disable private status federation over PuSH (#4582) 2017-08-24 17:51:32 +02:00
unarist b01a19fe39 Fetch reblogs as Announce activity instead of Note object (#4672)
* Process Create / Announce activity in FetchRemoteStatusService

* Use activity URL in ActivityPub for reblogs

* Redirect to the original status on StatusesController#show
2017-08-24 16:21:42 +02:00
unarist 760cfe328f Fix accessing to XML attribute in FetchAtomService (#4668) 2017-08-23 12:25:57 +02:00
unarist 5927b43c0f Ignore empty response in ActivityPub::FetchRemoteStatusService (#4661)
* Ignore empty response in ActivityPub::FetchRemoteStatusService

This fixes `NoMethodError: undefined method `[]' for nil:NilClass` error.

* Check json.nil? in JsonLdHelper#supported_context?
2017-08-22 20:00:49 +02:00
unarist d63de55ef8 Fix bugs which OStatus accounts may detected as ActivityPub ready (#4662)
* Fallback to OStatus in FetchAtomService

* Skip activity+json link if that activity is Person without inbox
* If unsupported activity was detected and all other URLs failed, retry with ActivityPub-less Accept header

* Allow mention to OStatus account in ActivityPub

* Don't update profile with inbox-less Person object
2017-08-22 18:30:15 +02:00
Eugen Rochko 72bb3e03fd Support more variations of ActivityPub keyId in signature (#4630)
- Tries to avoid performing HTTP request if the keyId is an actor URI
- Likewise if the URI is a fragment URI on top of actor URI
- Resolves public key, returns owner if the owner links back to the key
2017-08-21 22:57:34 +02:00
Eugen Rochko 3534e115e5 Do not try to re-subscribe to unsubscribed accounts (#4653) 2017-08-21 17:32:41 +02:00
Eugen Rochko 6e9eda5331 ActivityPub migration procedure (#4617)
* ActivityPub migration procedure

Once one account is detected as going from OStatus to ActivityPub,
invalidate WebFinger cache for other accounts from the same domain

* Unsubscribe from PuSH updates once we receive an ActivityPub payload

* Re-subscribe to PuSH unless already unsubscribed, regardless of protocol
2017-08-21 01:14:40 +02:00
Eugen Rochko fe5b66aa08 Handle duplicate ActivityPub activities (#4639)
* Handle duplicate ActivityPub activities

Only perform side-effects when record processed for the first time

* Fast-forward repeat follow requests
2017-08-20 16:53:47 +02:00
Yamagishi Kazutoshi 5f22c0189d Add support for searching AP users (#4599)
* Add support for searching AP users

* use JsonLdHelper
2017-08-14 14:08:34 +02:00
Eugen Rochko 26d26644ac Require "inbox" to be set on actor to be ActivityPub-ready (#4595) 2017-08-14 11:27:25 +02:00
Yamagishi Kazutoshi 96e9ed13de Fix search (regression from #4589) (#4594) 2017-08-14 04:50:56 +02:00
Eugen Rochko 4e75f0d889 Hook up URL-based resource look-up to ActivityPub (#4589) 2017-08-14 02:29:36 +02:00
Eugen Rochko b7370ac8ba ActivityPub delivery (#4566)
* Deliver ActivityPub Like

* Deliver ActivityPub Undo-Like

* Deliver ActivityPub Create/Announce activities

* Deliver ActivityPub creates from mentions

* Deliver ActivityPub Block/Undo-Block

* Deliver ActivityPub Accept/Reject-Follow

* Deliver ActivityPub Undo-Follow

* Deliver ActivityPub Follow

* Deliver ActivityPub Delete activities

Incidentally fix #889

* Adjust BatchedRemoveStatusService for ActivityPub

* Add tests for ActivityPub workers

* Add tests for FollowService

* Add tests for FavouriteService, UnfollowService and PostStatusService

* Add tests for ReblogService, BlockService, UnblockService, ProcessMentionsService

* Add tests for AuthorizeFollowService, RejectFollowService, RemoveStatusService

* Add tests for BatchedRemoveStatusService

* Deliver updates to a local account to ActivityPub followers

* Minor adjustments
2017-08-13 00:44:41 +02:00
Eugen Rochko ccdd5a9576 Add serializing/unserializing of "locked" actor attribute (#4585) 2017-08-12 17:41:03 +02:00
Eugen Rochko dd7ef0dc41 Add ActivityPub inbox (#4216)
* Add ActivityPub inbox

* Handle ActivityPub deletes

* Handle ActivityPub creates

* Handle ActivityPub announces

* Stubs for handling all activities that need to be handled

* Add ActivityPub actor resolving

* Handle conversation URI passing in ActivityPub

* Handle content language in ActivityPub

* Send accept header when fetching actor, handle JSON parse errors

* Test for ActivityPub::FetchRemoteAccountService

* Handle public key and icon/image when embedded/as array/as resolvable URI

* Implement ActivityPub::FetchRemoteStatusService

* Add stubs for more interactions

* Undo activities implemented

* Handle out of order activities

* Hook up ActivityPub to ResolveRemoteAccountService, handle
Update Account activities

* Add fragment IDs to all transient activity serializers

* Add tests and fixes

* Add stubs for missing tests

* Add more tests

* Add more tests
2017-08-08 21:52:15 +02:00
Yamagishi Kazutoshi 3e7a541e09 Change RuboCop rules to loose (#4464) 2017-07-31 15:19:13 +02:00
nullkal 6654c30033 Fix an error when a user tries to search nonexistent remote user (regression from #4275) (#4400) 2017-07-27 15:11:59 +02:00