Commit Graph

56 Commits

Author SHA1 Message Date
Eugen Rochko 6cf57c6765
Refactor how Redis locks are created (#18400)
* Refactor how Redis locks are created

* Fix autorelease duration on account deletion lock
2022-05-13 00:02:35 +02:00
Eugen Rochko 3917353645
Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
Eugen Rochko 7de0ee7aba
Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Claire 9a19227f17
Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
Takeshi Umeda c403c3695b
Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Eugen Rochko 2c77d97e0d
Add joined date to profiles in web UI (#16169) 2021-05-07 14:33:19 +02:00
ThibG a60d9335d8
Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-18 23:26:26 +01:00
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
ThibG fb586584f2
Fix account processing failing because of large collections (#15027)
Fixes #15025
2020-10-23 14:21:31 +02:00
Eugen Rochko 64aac30733
Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Eugen Rochko 5d8398c8b8
Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
ThibG b154428e14
Add federation support for the "hide network" preference (#11673)
* Change ActivityPub follower/following collections to not link first page

* Add support for hiding followers and following of remote users

* Switch to using a single `hide_collections` column

* Address code style remarks
2020-03-09 00:10:29 +01:00
ThibG 3704402dcc
Fix duplicate accounts being created when fetching an account for its key only (#13147)
Fixes #13136

When a user's canonical acct domain is different from its id's domain
(WEB_DOMAIN ≠ LOCAL_DOMAIN), two webfinger queries are required to find the
canonical domain from the URI. However, we skip webfinger queries when
updating only the key of a remote user, which led to the creation of a
duplicate account, using the URI's domain instead of the canonical acct: one.
2020-02-24 21:15:53 +01:00
Eugen Rochko cb447b28c4
Add profile directory to web UI (#11688)
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Eugen Rochko 4e92183227
Refactor domain block checks (#11268) 2019-07-09 03:27:35 +02:00
Eugen Rochko 2cfa427ea7
Change domain block behaviour to prevent creation of accounts from suspended domains (#11219) 2019-07-01 19:13:30 +02:00
Eugen Rochko 707ddf7808
Change domain blocks to automatically support subdomains (#11138)
* Change domain blocks to automatically support subdomains

If a more authoritative domain is blocked (example.com), then the
same block will be applied to a subdomain (foo.example.com)

* Match subdomains of existing accounts when blocking/unblocking domains

* Improve code style
2019-06-22 00:13:10 +02:00
ThibG 14f6ce2885 Record account suspend/silence time and keep track of domain blocks (#10660)
* Record account suspend/silence time and keep track of domain blocks

* Also unblock users who were suspended/silenced before dates were recorded

* Add tests

* Keep track of suspending date for users suspended through the CLI

* Show accurate number of accounts that would be affected by unsuspending an instance

* Change migration to set silenced_at and suspended_at

* Revert "Also unblock users who were suspended/silenced before dates were recorded"

This reverts commit a015c65d2d1e28c7b7cfab8b3f8cd5fb48b8b71c.

* Switch from using suspended and silenced to suspended_at and silenced_at

* Add post-deployment migration script to remove `suspended` and `silenced` columns

* Use Account#silence! and Account#suspend! instead of updating the underlying property

* Add silenced_at and suspended_at migration to post-migration

* Change account fabricator to translate suspended and silenced attributes

* Minor fixes

* Make unblocking domains always retroactive
2019-05-14 19:05:02 +02:00
Eugen Rochko 1714ea5978
Add ActivityPub representation for identity proofs (#10414)
* Add ActivityPub representation for identity proofs

* Add tests
2019-03-30 02:12:06 +01:00
abcang e186bd2fb0 Fix Tombstone.delete_all ArgumentError (#9978) 2019-02-06 02:50:52 +01:00
ThibG 75b1488cf4 Add tombstones for remote statuses (#9830)
* Add Tombstone model to remember object deletion

* Do not recreate a status if it has been deleted

* Record Tombstone for remote deleted items

Also, only record deleted items from same-host actors

* Clear an user's tombstones when their key change
2019-01-18 15:56:55 +01:00
ThibG 28b482874a Improvements to signature verification (#9667)
* Refactor signature verification a bit

* Rescue signature verification if recorded public key is invalid

Fixes #8822

* Always re-fetch AP signing key when HTTP Signature verification fails

But when the account is not marked as stale, avoid fetching collections and
media, and avoid webfinger round-trip.

* Apply stoplight to key/account update as well as initial key retrieval
2019-01-07 21:45:13 +01:00
Eugen Rochko 0f938ff29c
Add handler for Move activity (#9629) 2018-12-29 02:24:36 +01:00
ThibG 7f5a4be580 Fix emoji update date processing (#9255) 2018-11-10 23:59:51 +01:00
Eugen Rochko f4d549d300
Redesign forms, verify link ownership with rel="me" (#8703)
* Verify link ownership with rel="me"

* Add explanation about verification to UI

* Perform link verifications

* Add click-to-copy widget for verification HTML

* Redesign edit profile page

* Redesign forms

* Improve responsive design of settings pages

* Restore landing page sign-up form

* Fix typo

* Support <link> tags, add spec

* Fix links not being verified on first discovery and passive updates
2018-09-18 16:45:58 +02:00
Eugen Rochko cabdbb7f9c
Add CLI task for rotating keys (#8466)
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
2018-08-26 20:21:03 +02:00
ThibG 42573b76f1 Do not crash if remote custom emoji does not define updated date (fixes #8376) (#8377) 2018-08-23 00:27:58 +02:00
Eugen Rochko cc94b1d95a Prevent ActivityPub movedTo recursion (#8092)
Fix #8051
2018-07-28 17:14:55 -04:00
Akihiko Odaki 55fd55714a Raise Mastodon::RaceConditionError if Redis lock failed (#7511)
An explicit error allows user agents to know the error and Sidekiq to
retry.
2018-05-16 12:29:45 +02:00
Wiktor 7fe2993b87 Fix account URI when updating ActivityPub account (#7488)
Updates account `uri` field on each call to `update_account` instead of
only once during `create_account` to mirror the same behavior in OStatus
`ResolveAccountService` class [0].

ActivityPub accounts are identified using `@username` and `@domain` pair
instead of URI since #6842.

This fixes #7479: a bug when the account identified by `@username` and
`@domain` changes its URI.

[0]:
03b69ebc45/app/services/resolve_account_service.rb (L121)
2018-05-14 22:56:45 +02:00
Surinna Curtis 01dfd6dbc8 Take the first recognized actor_type. (#7410) 2018-05-08 13:30:04 +02:00
Eugen Rochko 42cd363542
Bot nameplates (#7391)
* Store actor type in database

* Add bot nameplate to web UI, add setting to preferences, API, AP
Fix #7365

* Fix code style issues
2018-05-07 09:31:07 +02:00
Surinna Curtis dc786c0cf4 Support Actors/Statuses with multiple types (#7305)
* Add equals_or_includes_any? helper in JsonLdHelper

* Support arrays in JSON-LD type fields for actors/tags/objects.

* Spec for resolving accounts with extension types

* Style tweaks for codeclimate
2018-05-02 12:40:24 +02:00
Eugen Rochko 78ed4ab75f
Add bio fields (#6645)
* Add bio fields

- Fix #3211
- Fix #232
- Fix #121

* Display bio fields in web UI

* Fix output of links and missing fields

* Federate bio fields over ActivityPub as PropertyValue

* Improve how the fields are stored, add to Edit profile form

* Add rel=me to links in fields

Fix #121
2018-04-14 12:41:08 +02:00
Eugen Rochko 778562c223
Ensure SynchronizeFeaturedCollectionWorker is unique and clean up (#7043)
* Ensure SynchronizeFeaturedCollectionWorker is unique and clean up

Fix #7041

* Fix code style issue
2018-04-13 01:27:22 +02:00
Nolan Lawson 6ff3b3e4db Fix nil account issue in ProcessAccountService (#7019) 2018-04-03 13:08:11 +02:00
David Underwood 123a343d11 [WIP] Enable custom emoji on account pages and in the sidebar (#6124)
Federate custom emojis with accounts
2018-04-01 23:55:42 +02:00
Eugen Rochko 9381a7d9d5
Use username/domain to match existing accounts in ActivityPub (#6842)
See also: #6837, #6667
2018-03-20 14:57:46 +01:00
Eugen Rochko 9110db41c5
Federate pinned statuses over ActivityPub (#6610)
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix #6117

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
2018-03-04 09:19:11 +01:00
Eugen Rochko e4a241abef
Fix bad URL schemes being accepted (#6219)
* Fix actors accepting invalid URI schemes or different host between URI and URL

* Fix statuses accepting invalid URI scheme or different host to actor

* Adjust tests to new requirements

* Improve readability of mismatching_origin?/invalid_origin? methods
2018-01-08 05:00:23 +01:00
Eugen Rochko d319b3dbe4
Update moved-to property when it's removed too (#6160)
* Fix #6140 - Update moved-to property when it's removed too

* Remove trailing whitespace
2018-01-03 00:38:20 +01:00
Eugen Rochko 4c6b5dbe96
Add semi-support for Video/Image objects in ActivityPub (#5848)
* Add semi-support for Video/Image objects in ActivityPub

Video and Image objects will create corresponding status records
with manually crafted text contents (title + URL)

* Extract html-url-finding logic into JsonLdHelper

* Fallback to id when url missing, extract supported object types
2017-11-30 04:06:20 +01:00
Eugen Rochko 58cede4808
Profile redirect notes (#5746)
* Serialize moved accounts into REST and ActivityPub APIs

* Parse federated moved accounts from ActivityPub

* Add note about moved accounts to public profiles

* Add moved account message to web UI

* Fix code style issues
2017-11-18 19:39:02 +01:00
Akihiko Odaki 63f0979799 Validate id of ActivityPub representations (#5114)
Additionally, ActivityPub::FetchRemoteStatusService no longer parses
activities.
OStatus::Activity::Creation no longer delegates to ActivityPub because
the provided ActivityPub representations are not signed while OStatus
representations are.
2017-10-04 01:13:48 +02:00
Eugen Rochko 41e6c8b151 Fix incomplete account records being read (#4998)
* Fix incomplete account records being read

- Put account processing into redis lock
- Do not save until record is complete

* Fix spaces
2017-09-19 06:53:16 +02:00
ThibG af00220d79 Fix refollowing (#4931)
* Make RefollowWorker ActivityPub-only to avoid potential identifier mismatches

* Don't call RefollowWorker on new accounts
2017-09-14 00:05:25 +02:00
ThibG f29918e707 [WiP] Whenever a remote keypair changes, unfollow them and re-subscribe to … (#4907)
* Whenever a remote keypair changes, unfollow them and re-subscribe to them

In Mastodon (it could be different for other OStatus or AP-enabled software),
a keypair change is indicative of whole user (or instance) data loss. In this
situation, the “new” user might be different, and almost certainly has an empty
followers list. In this case, Mastodon instances will disagree on follower
lists, leading to unreliable delivery and “shadow followers”, that is users
believed by a remote instance to be followers, without the affected user
knowing.

Drawbacks of this change are:
1. If an user legitimately changes public key for some reason without losing
   data (not possible in Mastodon at the moment), they will have their remote
   followers unsubscribed/re-subscribed needlessly.
2. Depending of the number of remote followers, this may generate quite some
   traffic.
3. If the user change is an attempt at usurpation, the remote followers will
   unknowingly follow the usurper. Note that this is *not* a change of
   behavior, Mastodon already behaves like that, although delivery might be
   unreliable, and the usurper would not have known the former user's
   followers.

* Rename ResubscribeWorker to RefollowWorker

* Process followers in batches
2017-09-12 23:10:40 +02:00
Eugen Rochko 9e15eeec63 Add missing reject_media check before avatar download via ActivityPub (#4862) 2017-09-09 13:41:45 +02:00
Eugen Rochko a4caa7eb62 Fetch statuses/following/followers numbers from ActivityPub collections (#4840) 2017-09-08 12:00:17 +02:00
Eugen Rochko 9b50a9dd83 Fix some ActivityPub JSON bugs (#4796)
- Fix assumption that `url` is always a string. Handle it if it's an
  array of strings, array of objects, object, or string, both for
  accounts and for objects
- `sharedInbox` is actually supposed to be under `endpoints`, handle
  both cases and adjust the serializer
2017-09-04 18:26:33 +02:00