Commit Graph

1254 Commits

Author SHA1 Message Date
Claire
8ec4be4233 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream-updated dependency (redis) textually too
  close to glitch-soc-only dependecy.
  Updated redis gem like upstream did.
2021-08-09 23:25:49 +02:00
Claire
4ac78e2a06
Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Claire
763ab0c7eb
Fix owned account notes not being deleted when an account is deleted (#16579)
* Add account_notes relationship

* Add tests

* Fix owned account notes not being deleted when an account is deleted

* Add post-migration to clean up orphaned account notes
2021-08-08 15:29:57 +02:00
Claire
b5e10d07ff Merge branch 'main' into glitch-soc/merge-upstream 2021-07-25 12:48:48 +02:00
Claire
1d67acb72f
Fix scoped order warning in RemoveStatusService (#16531)
Fixes “Scoped order is ignored, it's forced to be batch order.”
2021-07-24 14:41:46 +02:00
Claire
d8629e7b86
Add logging of S3-related errors (#16381) 2021-07-21 18:34:39 +02:00
Claire
5f334807d2 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-22 13:11:31 +02:00
Claire
9a19227f17
Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
Claire
dc58d02192 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-12 11:03:05 +02:00
Takeshi Umeda
c403c3695b
Fix to be able to redownload avatar and header (#16190)
* Fix to reset if header and avatar download fails

* Add RedownloadAvatarWorker and RedownloadHeaderWorker
2021-05-11 14:19:22 +02:00
Claire
7ac8bcf209 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-10 21:45:32 +02:00
Claire
afb7882189
Fix blocking someone not clearing up list feeds (#16205) 2021-05-10 17:31:55 +02:00
Claire
87c3a0d0b3 Merge branch 'main' into glitch-soc/merge-upstream 2021-05-09 10:41:28 +02:00
Takeshi Umeda
9da5e0b350
Fix webfinger_update_due to run WebFinger on stale activitypub-account (#16182) 2021-05-08 16:22:18 +02:00
Claire
50b430d9a2 Merge branch 'main' into glitch-soc/merge-upstream
- `app/views/statuses/_simple_status.html.haml`:
  Small markup change in glitch-soc, on a line that has been modified by
  upstream. Ported upstream changes.
2021-05-07 18:21:59 +02:00
Eugen Rochko
74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Eugen Rochko
2c77d97e0d
Add joined date to profiles in web UI (#16169) 2021-05-07 14:33:19 +02:00
Eugen Rochko
fab65848d2
Fix empty home feed before first follow has finished processing (#16152)
Change queue of merge worker from pull to default
2021-05-04 04:45:08 +02:00
Eugen Rochko
f627d2eb93
Fix trying to fetch key from empty URI when verifying HTTP signature (#16100) 2021-05-01 23:18:59 +02:00
Eugen Rochko
daccc07dc1
Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Claire
536892b8ae Merge branch 'main' into glitch-soc/merge-upstream 2021-04-21 13:52:03 +02:00
Claire
0b36e3419d
Fix processing of remote Delete activities (#16084)
* Add tests

* Ensure deleted statuses are marked as such

* Save some redis memory by not storing URIs in delete_upon_arrival values

* Avoid possible race condition when processing incoming Deletes

* Avoid potential duplicate Delete forwards

* Lower lock durations to reduce issues in case of hard crash of the Rails process

* Check for `lock.aquired?` and improve comment

* Refactor RedisLock usage in app/lib/activitypub

* Fix using incorrect or non-existent sender for relaying Deletes
2021-04-21 04:46:09 +02:00
Claire
e2a2bc9021 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Upstream updated copyright year, we don't mention it so kept our version.
- `app/controllers/admin/dashboard_controller.rb`:
  Not really a conflict, upstream change (removing the spam checker) too close
  to glitch-soc changes. Ported upstream changes.
- `app/models/form/admin_settings.rb`:
  Same.
- `app/services/remove_status_service.rb`:
  Same.
- `app/views/admin/settings/edit.html.haml`:
  Same.
- `config/settings.yml`:
  Same.
- `config/environments/production.rb`:
  Not a real conflict, upstream added a default HTTP header, but we have
  extra headers in glitch-soc.
  Added the header.
2021-04-20 12:17:14 +02:00
Eugen Rochko
ca3bc1b09f
Refactor StatusReachFinder to handle followers and relays as well (#16051) 2021-04-17 15:41:57 +02:00
Eugen Rochko
6d6000f61f
Fix remote reporters not receiving suspend/unsuspend activities (#16050) 2021-04-17 14:55:46 +02:00
Eugen Rochko
dde8739020
Fix reports of already suspended accounts being recorded (#16047) 2021-04-16 22:01:05 +02:00
Eugen Rochko
619fad6cf8
Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Claire
f60c99a8fb Merge branch 'main' into glitch-soc/merge-upstream 2021-03-24 10:53:50 +01:00
Claire
cbd0ee1d07
Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire
1b6a21e6bc Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/validators/status_length_validator.rb`:
  Conflict due to glitch-soc's configurable maximum toot chars.
  Ported upstream changes.
2021-03-11 16:08:15 +01:00
Claire
5614e6724e
Fix URL scanning in note length validator and preview card fetching (#15827)
* Add tests

* Fix URL scanning in note length validator and preview card fetching
2021-03-04 00:12:26 +01:00
Claire
d8fdbb054e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/validators/status_length_validator.rb`:
  Upstream changes too close to glitch-soc MAX_CHARS changes, but not a real
  conflict.
  Applied upstream changes.
- `package.json`:
  glitch-soc-only dependency textually too close to a dependency updated
  upstream, not a real conflict.
  Applied upstream changes.
2021-03-02 12:06:58 +01:00
Claire
65db262550
Update twitter-text from 1.14 to 3.1.0 and fix toot character counting (#15382)
* Update twitter-text from 1.14 to 3.1.0

* Disable emoji parsing

* Properly depend on twitter-text for url detection

* Fix some URLs being wrongly detected client-side

* Add test for server-side validation of non-autolinkable URLs

* Fix server-side status length counting
2021-03-02 12:02:56 +01:00
Claire
5f4c0b79c2
Change ResolveAccountService's handling of skip_webfinger (#15750)
* Change ResolveAccountService's handling of skip_webfinger

Change it so it never makes any webfinger query, as the name would imply.

* Add tests

* Change FollowService to not take an URI for target_account

* Restore domain-block check in FollowService

* Fix tests
2021-02-24 06:32:13 +01:00
Claire
5e11f3a6e1 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/styles/mastodon/modal.scss`:
  For some reason we changed the file loading path in glitch-soc,
  but now upstream has completely changed how the logo is loaded.
  Applied upstream changes.
2021-02-12 10:28:32 +01:00
Eugen Rochko
f8972d4503
Fix YouTube embeds failing due to YouTube serving wrong OEmbed URLs (#15716) 2021-02-12 05:45:38 +01:00
Claire
b81710c02c Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `CONTRIBUTING.md`:
  Not a real conflict, glitch-soc quotes the upstream file, which has been
  changed. Update the quote.
2021-01-26 14:01:30 +01:00
Takeshi Umeda
7f1c56954b
Fix first return value of FetchLinkCardService.html method (#15630) 2021-01-25 09:22:41 +01:00
luigi
7ea9588520
Use Enumerable#filter_map in more places (#15527) 2021-01-22 16:28:15 +01:00
luigi
eb51e43fb4
Optimize some regex matching (#15528)
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
2021-01-22 10:09:08 +01:00
Claire
33d30632fb Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/models/public_feed.rb`:
  Upstream refactored a bit, glitch-soc had specific code for local-only
  statuses.
  Updated glitch-soc's specific code accordingly.
2021-01-11 11:55:42 +01:00
luigi
087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
abcang
efffdd3778
Fix rubocop config and warnings (#15503)
* disable NewCops

* update TargetRubyVersion

* Fix Lint/MissingSuper for ActiveModelSerializers::Model

* Fix Lint/MissingSuper for feed

* Fix Lint/FloatComparison

* Do not use instance variables
2021-01-07 09:40:55 +01:00
Claire
90528f43bc Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/webpack/configuration.js`:
  Upstream updated the `js-yaml` dependency, which changed how to call it.
  Those changes conflicted because that code is pretty different in glitch-soc
  which has to deal with its more complex theming system.
  Proceeded to the same compatibility changes in glitch-soc's code.
- `package.json` and `yarn.lock`:
  Not really a conflict, just glitch-soc-specific dependencies textually too
  close to some dependencies updated upstream.
2021-01-06 18:08:06 +01:00
ThibG
f1f96ebf02
Fix being able to import more than allowed number of follows (#15384)
* Fix being able to import more than allowed number of follows

Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.

This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).

* Catch validation errors to not re-queue failed follows

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-26 23:52:46 +01:00
Eugen Rochko
e89648574f
Fix error when changing ACL on missing objects during suspension (#15420) 2020-12-24 15:53:45 +01:00
Eugen Rochko
ba0b79fc5c
Fix undefined method error when batch-removing statuses with ElasticSearch on (#15421) 2020-12-24 15:53:26 +01:00
ThibG
de57efd055
Fix mentions not being deleted efficiently (#15416)
As a regression from the recent optimizations, mentions were left untouched
until `account.destroy`, which would then delete them individually,
and executing queries to find and delete associated notifications, resulting
in a massive slowdown.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 16:43:11 +01:00
Eugen Rochko
473abc35a1
Fix trying to privatize empty media attachments (#15414) 2020-12-23 15:47:03 +09:00
Claire
537afa00f3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/lib/feed_manager.rb`:
  Not a real conflict, glitch-soc-only DM-related method
  too close to changed upstream stuff.
  Ported upstream changes.
- `app/services/batched_remove_status_service.rb`:
  Additional logic in glitch-soc to clear DMs from timelines.
  Ported upstream changes and fixed the DM TL clearing logic.
- `app/workers/scheduler/feed_cleanup_scheduler.rb`:
  Additional code in glitch-soc to clear DM timelines.
  Ported upstream changes.
2020-12-23 01:47:45 +01:00
ThibG
814b7775fb
Improve performances of deleting favourites when deleting accounts (#15412)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 01:35:02 +01:00
ThibG
2ab3e91eaf
Fix BatchedRemoveStatusService not working without ES in rails console (#15408)
Not a huge deal but may cause surprising failures in custom scripts and
development.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-23 00:04:52 +01:00
ThibG
3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
ThibG
f18349640b
Fix batch order warnings in BatchedRemoveStatusService (#15409)
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 20:24:55 +01:00
Eugen Rochko
9915d11c0d
Fix unnecessary queries when batch-removing statuses, 100x faster (#15387) 2020-12-22 17:13:55 +01:00
Claire
38bfaf8855 Merge branch 'master' into glitch-soc/merge-upstream 2020-12-21 18:28:23 +01:00
ThibG
43961035a9
Fix some notifications not being deleted on poll/status deletion (#15402)
* Fix deleting polls not deleting notifications

* Fix fav notification deletion when deleting a toot

* Refactor DeleteAccountService spec

* Add DeleteAccountService tests for other associations and notifications

* Add favourite handling spec in status removal

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-21 18:22:17 +01:00
ThibG
7bf3c6e57b
Fix AccountDeletionWorker crashing and clogging sidekiq queues (#15380)
* Fix account deletion workers being queued multiple times for a single account

* Fix poll votes being unnecessarily instantiated on poll deletion

* Fix favourites being unnecessarily instantiated on status deletion

* Remove inaccurate comments

* Delete polls instead of destroying them

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-20 18:25:00 +01:00
Claire
f9d000ebae Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/resolve_url_service.rb`:
  The private toot search by URL hack has been revamped upstream.
  Took upstream's version.
2020-12-19 00:55:12 +01:00
ThibG
a60d9335d8
Fix resolving accounts sometimes creating duplicate records for a given AP id (#15364)
* Fix ResolveAccountService accepting mismatching acct: URI

* Set attributes that should be updated regardless of suspension

* Fix key fetching

* Automatically merge remote accounts with duplicate `uri`

* Add tests

* Add "tootctl accounts fix-duplicates"

Finds duplicate accounts sharing a same ActivityPub `id`, re-fetch them and
merge them under the canonical `acct:` URI.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-18 23:26:26 +01:00
Eugen Rochko
eb35be0431
Fix follow limit preventing re-following of a moved account (#14207) 2020-12-18 09:18:31 +01:00
ThibG
b1feb47055
Improve searching for private toots from URL (#14856)
* Improve searching for private toots from URL

Most of the time, when sharing toots, people use the toot URL rather than
the toot URI, which makes sense since it is the user-facing URL.

In Mastodon's case, the URL and URI are different, and Mastodon does not
have an index on URL, which means searching a private toot by URL is done
with a slow query that will only succeed for very recent toots.

This change gets rid of the slow query, and attempts to guess the URI from
URL instead, as Mastodon's are predictable.

* Add tests

* Only return status with guessed uri if url matches

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-17 06:51:49 +01:00
Claire
e4f8679eae Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/models/form/admin_settings.rb`:
  New setting added upstream. Ported it.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream removed RTL classes. Did the same.
- `config/settings.yml`:
  New setting added upstream. Ported it.
2020-12-15 14:27:06 +01:00
ThibG
1390cc194b
Add indication to admin UI of whether a report has been forwarded (#13237)
* Add indication to admin UI of whether a report has been forwarded

* Rework how forwarded status is displayed

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-15 04:30:15 +01:00
Claire
1978f7265e Fix processing of Delete activities on direct messages 2020-12-14 23:05:32 +01:00
Claire
f8d867bac4 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/remove_status_service.rb`:
  Conflict caused by us having a distinc Direct timeline.
  Ported upstream changes.
- `app/javascript/mastodon/features/compose/components/compose_form.js`:
  Conflict between glitch-soc's variable character limit and upstream
  refactoring that part of the code.
  Ported upstream changes.
2020-12-05 17:33:37 +01:00
Eugen Rochko
e7e099d1a0
Fix deletes not reaching every server that interacted with status (#15200)
Extract logic for determining ActivityPub inboxes to send deletes
to to its own class and explicitly include the person the status
replied to (even if not mentioned), people who favourited it, and
people who replied to it (though that one is still not recursive)
2020-11-27 15:48:31 +01:00
Thibaut Girka
e71b16ae2b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  glitch-soc-only dependency textually too close to updated upstream
  dependencies. Updated to upsteam dependencies.
2020-11-25 15:31:45 +01:00
ThibG
66832cbf33
Undo outgoing follows when suspending a remote account (#15188) 2020-11-21 21:27:00 +01:00
Thibaut Girka
48f0f3ffee Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/ISSUE_TEMPLATE/bug_report.md`:
  Upstream added the `bug` label to bug reports.
  Did the same.
- `app/services/fan_out_on_write_service.rb`:
  Upstream put DMs back into timelines, glitch-soc was already doing it.
  Ignored upstream changes.
2020-11-20 13:27:48 +01:00
ThibG
8b8004a962
Fix webfinger redirect handling in ResolveAccountService (#15187)
* Fix webfinger redirect handling in ResolveAccountService

ResolveAccountService#process_webfinger! handled a one-step webfinger
redirection, but only accepting the result if it matched the exact URI passed
as input, defeating the point of a redirection check.

Instead, use the same logic as in `ActivityPub::FetchRemoteAccountService`,
updating the resulting `acct:` URI with the result of the first webfinger
query.

* Add tests
2020-11-19 19:52:06 +01:00
ThibG
96c1e71329
Add import/export feature for bookmarks (#14956)
* Add ability to export bookmarks

* Add support for importing bookmarks

* Add bookmark import tests

* Add bookmarks export test
2020-11-19 17:48:13 +01:00
ThibG
2f6831f318
Fix sending spurious Rejects when processing remote account deletion (#15104)
* Fix sending spurious Rejects when processing remote account deletion

* Make skip_side_effects imply skip_activitypub
2020-11-19 17:39:47 +01:00
Eugen Rochko
df1653174b
Add cache buster feature for media files (#15155)
Nginx can be configured to bypass proxy cache when a special header
is in the request. If the response is cacheable, it will replace
the cache for that request. Proxy caching of media files is
desirable when using object storage as a way of minimizing bandwidth
costs, but has the drawback of leaving deleted media files for
a configured amount of cache time. A cache buster can make those
media files immediately unavailable. This especially makes sense
when suspending and unsuspending an account.
2020-11-19 17:38:06 +01:00
ThibG
966593393e
Fix DMs not appearing into timelines (#15182)
Fixes #15179
2020-11-19 00:23:46 +01:00
Thibaut Girka
10bd6f415d Improve searching for private toots from URL
Most of the time, when sharing toots, people use the toot URL rather than
the toot URI, which makes sense since it is the user-facing URL.

In Mastodon's case, the URL and URI are different, and Mastodon does not
have an index on URL, which means searching a private toot by URL is done
with a slow query that will only succeed for very recent toots.

This change gets rid of the slow query, and attempts to guess the URI from
URL instead, as Mastodon's are predictable.
2020-11-12 23:44:30 +01:00
Thibaut Girka
ecd83f495c Merge branch 'master' into glitch-soc/merge-upstream 2020-11-09 10:53:53 +01:00
ThibG
1e64666662
Fix crash in SuspendAccountWorker (#15106)
* Fix crash in SuspendAccountWorker

`follows` is an array thanks to `to_a`

* Fix code style issue

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2020-11-08 18:29:48 +01:00
Thibaut Girka
0437d70628 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/follower_accounts_controller.rb`:
  Conflict due to upstream changing suspension logic while
  glitch-soc has an extra option to hide followers count.
  Ported upstream changes.
2020-11-08 14:20:35 +01:00
Eugen Rochko
3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
Thibaut Girka
412218af2e Merge branch 'master' into glitch-soc/merge-upstream 2020-11-07 18:23:02 +01:00
ThibG
ee8cf246cf
Fix crashes in SuspendAccountService/UnsuspendAccountService (#15100)
* Fix crashes in SuspendAccountService/UnsuspendAccountService

* Catch filesystem errors
2020-11-07 13:16:54 +01:00
ThibG
f84239ecab
Fix suspension/unsuspension not working because of FeedManager change (#15099) 2020-11-07 13:16:00 +01:00
Thibaut Girka
9c34090db8 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/process_mentions_service.rb`:
  glitch-soc local-only handling, ported upstream changes
2020-10-26 15:48:16 +01:00
ThibG
1f945e7390
Fix followers synchronization mechanism not being triggered on mentions (#15026)
e.g. if someone on an instance that previously had followers gets mentioned
in a private toot, before this PR, they would not receive a
Collection-Synchronization header and may show the toot to the former followers
in addition to the mentioned person.
2020-10-23 14:22:16 +02:00
ThibG
fb586584f2
Fix account processing failing because of large collections (#15027)
Fixes #15025
2020-10-23 14:21:31 +02:00
Thibaut Girka
ec49aa8175 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.github/dependabot.yml`:
  Updated upstream, we deleted it to not be flooded by Depandabot.
  Kept deleted.
- `Gemfile.lock`:
  Puma updated on both sides, went for the most recent version.
- `app/controllers/api/v1/mutes_controller.rb`:
  Upstream updated the serializer to support timed mutes, while
  glitch-soc added a custom API ages ago to get information that
  is already available elsewhere.
  Dropped the glitch-soc-specific API, went with upstream changes.
- `app/javascript/core/admin.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/javascript/packs/public.js`:
  Conflict due to changing how assets are loaded. Went with upstream.
- `app/models/mute.rb`:
  🤷
- `app/models/user.rb`:
  New user setting added upstream while we have glitch-soc-specific
  user settings. Added upstream's user setting.
- `config/settings.yml`:
  Upstream added a new user setting close to a user setting we had
  changed the defaults for. Added the new upstream setting.
- `package.json`:
  Upstream dependency updated “too close” to a glitch-soc-specific
  dependency. No real conflict. Updated the dependency.
2020-10-21 19:10:50 +02:00
ThibG
ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
OSAMU SATO
96761752ec
Add duration parameter to muting. (#13831)
* Adding duration to muting.

* Remove useless checks
2020-10-13 01:01:14 +02:00
Eugen Rochko
5e1364c448
Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Eugen Rochko
7d985f2aac
Remove dependency on goldfinger gem (#14919)
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.

Fix #14091
2020-10-08 00:34:57 +02:00
Thibaut Girka
a7aedebc31 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
- `app/controllers/oauth/authorized_applications_controller.rb`:
  Upstream changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/base_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/controllers/settings/sessions_controller.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's theming system.
  Ported upstream changes.
- `app/models/user.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc not preventing moved accounts from logging
  in.
  Ported upstream changes while keeping the ability for moved accounts to log
  in.
- `app/policies/status_policy.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `app/serializers/rest/account_serializer.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's ability  to hide followers count.
  Ported upstream changes.
- `app/services/process_mentions_service.rb`:
  Upstream refactored and changed the logic surrounding suspended accounts.
  Minor conflict due to glitch-soc's local-only toots.
  Ported upstream changes.
- `package.json`:
  Not a real conflict, upstream updated dependencies that were too close to
  glitch-soc-only ones in the file.
2020-09-28 14:13:30 +02:00
Eugen Rochko
974b1b79ce
Add option to be notified when a followed user posts (#13546)
* Add bell button

Fix #4890

* Remove duplicate type from post-deployment migration

* Fix legacy class type mappings

* Improve query performance with better index

* Fix validation

* Remove redundant index from notifications
2020-09-18 17:26:45 +02:00
Eugen Rochko
ed099d8bdc
Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Thibaut Girka
9748f074a3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/controllers/api/v1/timelines/public_controller.rb
- app/lib/feed_manager.rb
- app/models/status.rb
- app/services/precompute_feed_service.rb
- app/workers/feed_insert_worker.rb
- spec/models/status_spec.rb

All conflicts are due to upstream refactoring feed management and us having
local-only toots on top of that. Rewrote local-only toots management for
upstream's changes.
2020-09-08 16:26:47 +02:00
Eugen Rochko
65760f59df
Refactor feed manager (#14761) 2020-09-08 03:41:16 +02:00
Eugen Rochko
e8bc187845
Refactor how public and tag timelines are queried (#14728) 2020-09-07 11:02:04 +02:00
Thibaut Girka
3fefb7909b Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/webpack/shared.js`:
  Upstream has changed how Tesseract.js gets included and dropped a dependency.
  The conflict is caused by glitch-soc having different code due to its
  theming system.
  Ported upstream changes.
- `lib/mastodon/version.rb`:
  Upstream refactor/code style change in a place we replaced upstream's
  repo URL with ours.
  Ported upstram changes, keeping our repo URL.
- `yarn.lock`:
  Upstream dropped dependencies, one of which was textually too close to
  a glitch-soc-specific dependency. Not a real conflict.
2020-09-01 17:02:18 +02:00
Eugen Rochko
1c308af84c
Change own direct-visibility statuses to be in the home feed again (#14711)
And remove highlighting in web UI

Full circle from #8940
2020-09-01 13:30:42 +02:00
dependabot[bot]
8972e5f7f6
Bump rubocop from 0.86.0 to 0.88.0 (#14412)
* Bump rubocop from 0.86.0 to 0.88.0

Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0.
- [Release notes](https://github.com/rubocop-hq/rubocop/releases)
- [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Fix for latest RuboCop

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-09-01 03:04:00 +02:00
Takeshi Umeda
b1bb1906be
Fix limited follower id in fan-out-on-write service (#14709) 2020-08-31 18:11:27 +02:00
Thibaut Girka
8c3c27bf06 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/accounts_controller.rb`:
  Upstream change too close to a glitch-soc change related to
  instance-local toots. Merged upstream changes.
- `app/services/fan_out_on_write_service.rb`:
  Minor conflict due to glitch-soc's handling of Direct Messages,
  merged upstream changes.
- `yarn.lock`:
  Not really a conflict, caused by glitch-soc-only dependencies
  being textually too close to updated upstream dependencies.
  Merged upstream changes.
2020-08-30 16:13:08 +02:00
Eugen Rochko
ce8f33dd2f
Fix inefficiencies in fan-out-on-write service (#14682) 2020-08-30 12:33:59 +02:00
Akihiko Odaki
41eeb9ebaa
Use Status.group instead of Status.distinct in HashQueryService (#14662)
DISTINCT clause removes duplicated records according to all the selected
attributes. In reality, it can remove duplicated records only looking at
statuses.id, but the clause confuses the query planner and yields
insufficient performance.
The behavior is also problematic if the scope produced by HashQueryService
is used to query columns without id (using pluck method, for example). The
scope is expected to contain unique statuses, but the uniquness will be
evaluated with some arbitrary columns other than id.

GROUP BY clause resolves those problem by explicitly specifying the
column to take into account for the record distinction.

A workaround for the problem of DISTINCT clause in
Api::V1::Timelines::TagController is no longer necessary and removed.
2020-08-25 13:39:35 +02:00
Thibaut Girka
2cfa142961 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `yarn.lock`:
  No real conflict, just a glitch-soc-only dependency textually too close to
  one updated upstream. Ported upstream changes.
2020-08-01 20:24:23 +02:00
Takeshi Umeda
156af70e4d
Fix following restriction not working when exact match in account search (#14394) 2020-07-26 23:58:36 +02:00
Thibaut Girka
4d950c71e9 Merge branch 'master' into glitch-soc/master 2020-07-24 11:26:40 +02:00
Takeshi Umeda
939a47393c
Fix duplicate notification and delivery of mentions (#14378) 2020-07-23 15:41:49 +02:00
Thibaut Girka
b9f351d845 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/actions/compose.js`:
  Not a “real” conflict, but change too close to a change we made to
  fix the vanilla WebUI locally pushing authored local-only toots in the
  public TL view.
2020-07-22 13:38:17 +02:00
Takeshi Umeda
7540e235a2
Fix movie width and frame_rate returning nil (#14357)
* Fix movie width and frame_rate returning nil

* Add StreamValidationError and raise

* Fix code style
2020-07-19 22:28:27 +02:00
Thibaut Girka
1c6c40d17c Merge branch 'master' into glitch-soc/merge-upstream 2020-07-16 14:43:50 +02:00
ThibG
d658af7ff8
Fix removing allowed domains being done synchronously (#14302)
* Fix removing allowed domains being done synchronously

* Add tests
2020-07-15 21:08:19 +02:00
Thibaut Girka
3f60b096b5 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `config/routes.rb`:
  Upstream disabled E2EE routes, which we did earlier, but slightly
  differently. Took upstream's version.
2020-07-15 15:32:40 +02:00
Eugen Rochko
98b3b80d6b
Fix account domain block not properly unfollowing accounts from domain (#14304)
Fix #14303
2020-07-13 19:33:10 +02:00
ThibG
ee5a4030f7
Fix removing a DomainAllow rule wiping known accounts in open federation mode (#14298)
Fixes #14296
2020-07-12 22:11:49 +02:00
Thibaut Girka
e9ad99bc93 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `package.json`:
  Not really a conflict, just some glitch-soc-specific dependency
  too close to an upstream-updated one.
2020-07-07 15:58:45 +02:00
Eugen Rochko
ed04697510
Change User-Agent of link preview fetching service to include "Bot" (#14248)
This forces Twitter to render OpenGraph tags in the response
2020-07-07 10:55:18 +02:00
Thibaut Girka
9c3b9d5ab9 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `.env.production.sample`:
  Upstream deleted it, I decided to keep it.
- `package.json`:
  Upstream updated a dependency textually too close to wavesurfer.js
  which has been deleted from upstream but is kept in glitch-soc for now.
2020-06-30 23:00:20 +02:00
Eugen Rochko
64aac30733
Add customizable thumbnails for audio and video attachments (#14145)
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Thibaut Girka
12c8ac9e14 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/activitypub/collections_controller.rb`:
  Conflict due to glitch-soc having to take care of local-only
  pinned toots in that controller.
  Took upstream's changes and restored the local-only special
  handling.
- `app/controllers/auth/sessions_controller.rb`:
  Minor conflicts due to the theming system, applied upstream
  changes, adapted the following two files for glitch-soc's
  theming system:
  - `app/controllers/concerns/sign_in_token_authentication_concern.rb`
  - `app/controllers/concerns/two_factor_authentication_concern.rb`
- `app/services/backup_service.rb`:
  Minor conflict due to glitch-soc having to handle local-only
  toots specially. Applied upstream changes and restored
  the local-only special handling.
- `app/views/admin/custom_emojis/index.html.haml`:
  Minor conflict due to the theming system.
- `package.json`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
- `yarn.lock`:
  Upstream dependency updated, too close to a glitch-soc-only
  dependency in the file.
2020-06-09 10:39:20 +02:00
ThibG
89f40b6c3e
Make domain block/silence/reject-media code more robust (#13424)
* Split media cleanup from reject-media domain blocks to its own service

* Slightly improve ClearDomainMediaService error handling

* Lower DomainClearMediaWorker to lowest-priority queue

* Do not catch ActiveRecord::RecordNotFound in domain block workers

* Fix DomainBlockWorker spec labels

* Add some specs

* Change domain blocks to immediately mark accounts as suspended

Rather than doing so sequentially, account after account, while cleaning
their data. This doesn't change much about the time the block takes to
complete, but it immediately prevents interaction with the blocked domain,
while up to now, it would only be guaranteed when the process ends.
2020-06-09 10:32:00 +02:00
Takeshi Umeda
d890abfcab
Fix performance of follow import (#13836) 2020-06-09 10:26:58 +02:00
Takeshi Umeda
643065799b
Fix activity not being signed (#13948) 2020-06-03 08:33:30 +02:00
Eugen Rochko
5d8398c8b8
Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Thibaut Girka
b07c1e615b Fix backup service when user has posted instance-local toots, again 2020-05-21 20:05:46 +02:00
Thibaut Girka
27b5143dc4 Merge branch 'master' into glitch-soc/merge-upstream 2020-05-20 18:22:42 +02:00
ThibG
27ea7c13a5
Fix hashtag search performing account search as well (#13758) 2020-05-14 23:37:37 +02:00
Thibaut Girka
3baacf6993 Merge branch 'master' into glitch-soc/merge-upstream 2020-05-13 22:11:49 +02:00
Takeshi Umeda
d8bad8fbf6
Fix not publishing update of remote timeline (#13745)
* Fix not publishing update of remote timeline

* fix @ missing

* if/unless to if/else
2020-05-12 15:24:35 +02:00
Thibaut Girka
4a70792b4a Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  Not a real conflict, just a glitch-soc-only dependency too close to a
  dependency that got updated upstream. Updated as well.
- `app/models/status.rb`:
  Not a real conflict, just a change too close to glitch-soc-changed code
  for optionally showing boosts in public timelines.
  Applied upstream changes.
- `app/views/layouts/application.html.haml`:
  Upstream a new, static CSS file, conflict due to glitch-soc's theming
  system, include the file regardless of the theme.
- `config/initializers/content_security_policy.rb`:
  Upstream dropped 'unsafe-inline' from the 'style-src' directive, but
  both files are very different. Removed 'unsafe-inline' as well.
2020-05-10 16:19:56 +02:00
Eugen Rochko
4b766f9846
Refactor monkey-patching of Goldfinger (#12561) 2020-05-10 11:41:43 +02:00
Thibaut Girka
a22e6a3683 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/statuses_controller.rb`:
  Upstream disabled the embed controller for reblogs.
  Not a real conflict, but glitch-soc has an extra line to deal
  with its theming system.
  Ported upstream changes.
- `app/javascript/packs/public.js`:
  Upstream made changes to get rid of most inline CSS, this changes
  javascript for public pages, which in glitch are split between
  different files. Ported those changes.
- `app/models/status.rb`:
  Upstream changed the block check in `Status#permitted_for` to
  include domain-block checks. Not a real conflict with glitch-soc,
  but our scope is slightly different, as our scope for
  unauthenticated access do not include instance-local toots.
  Ported upstream changes.
- `app/serializers/rest/instance_serializer.rb`:
  Not a real conflict, upstream added a new field to the instance
  serializer, the conflict is one line above since we added more of
  that.
  Ported upstream changes.
- `app/views/settings/profiles/show.html.haml`:
  Upstream got rid of most inline CSS and moved hidden elements
  to data attributes in the process, in fields were we have
  different values.
  Ported upstream changes while keeping our glitch-specific
  values.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream got rid of inline CSS on an HAML line we treat
  differently, stripping empty text nodes.
  Ported upstream changes to the style attribute, keeping
  the empty text node stripping behavior.
2020-05-03 21:23:49 +02:00
Eugen Rochko
2744f61696
Fix not being able to resolve public resources in development environment (#13505) 2020-04-25 22:01:08 +02:00
Thibaut Girka
f3eff922a3 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`:
  Upstream bumped poll option character limit, but we already had
  a higher one, kept ours.
- `app/validators/poll_validator.rb`:
  Upstream bumped poll option character limit, but we already had
  a higher one, kept ours.
- `config/initializers/content_security_policy.rb`:
  Upstream added a rule, the way we compute ours is different, but
  that added rule has been ported.
- `package.json`:
  No real conflict, dependency update. Performed the same update.
- `yarn.lock`:
  No real conflict, dependency update. Performed the same update.
2020-04-02 20:32:00 +02:00
Eugen Rochko
c3965e28b3
Fix returning results when searching for URL with non-zero offset (#13377)
Fix #13083
2020-04-02 03:39:37 +02:00
Takeshi Umeda
11169367e4
Fix incorrect deletion of local accounts imported by overwriting (#13350) 2020-03-30 20:32:34 +02:00
Thibaut Girka
d98fabf2ee Fix crash when posting with a CW but no text nor media 2020-03-27 22:28:39 +01:00
Thibaut Girka
7b435fd9bf Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/services/post_status_service.rb`:
  CW/sensitive logic fixed upstream, but different in glitch-soc.
  Ported the changes accordingly.
2020-03-27 21:54:44 +01:00
ThibG
f08f880f58
Fix media not being marked sensitive when client sets a CW but no text (#13277)
Mastodon enforces the “sensitive” flag on media attachments whenever a toot
is posted with a Content Warning. However, it does so *after* potentially
converting the Content Warning to toot text (when there is no toot text),
which leads to inconsistent and surprising behavior for API clients.
This commit fixes this inconsistency.
2020-03-25 22:40:58 +01:00
Thibaut Girka
9abb227250 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Our README.md files are completely different. Discarded upstream changes.
- `app/javascript/core/admin.js`:
  Updating rails-ujs, no real conflict, but a comment to close to changed
  code. Various glitch-soc-only files have been updated to match those changes,
  though.
- `package.json`:
  No real conflict, just an additional dependency in glitch-soc that was too
  close to something updated upstream. Took upstream's changes.
2020-03-22 16:10:44 +01:00
ThibG
cb12a2cdd3
Fix some timeouts when searching URLs by limiting some database queries (#13253)
Only look up private toots from database if the request failed because of 401,
403 or 404 errors, as those may indicate a private toot, rather than something
that isn't a toot or cannot be processed.
2020-03-12 23:06:43 +01:00
ThibG
b154428e14
Add federation support for the "hide network" preference (#11673)
* Change ActivityPub follower/following collections to not link first page

* Add support for hiding followers and following of remote users

* Switch to using a single `hide_collections` column

* Address code style remarks
2020-03-09 00:10:29 +01:00
Eugen Rochko
9660aa4543
Change local media attachments to perform heavy processing asynchronously (#13210)
Fix #9106
2020-03-08 23:56:18 +01:00
Thibaut Girka
c790ecb14d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/api/v1/statuses_controller.rb`:
  Conflict due to upstream adding a new parameter (with_rate_limit),
  too close to glitch-soc's own additional parameter (content_type).
  Added upstream's parameter.
- `app/services/post_status_service.rb`:
  Conflict due to upstream adding a new parameter (rate_limit),
  too close to glitch-soc's own additional parameter (content_type).
  Added upstream's parameter.
- `app/views/settings/preferences/appearance/show.html.haml`:
  Conflict due to us not exposing theme settings here (as we have
  a different flavour/skin menu).
  Took upstream change, while still not exposing theme settings.
- `config/webpack/shared.js`:
  Coding style fixes for a part we have rewritten.
  Discarded upstream changes.
2020-03-08 19:38:53 +01:00
ThibG
5284e29e2f
Fix public posts from silenced accounts not being changed to unlisted visibility (#13096) 2020-03-08 16:11:49 +01:00
ThibG
6185bff4b3
Fix error when searching for URLs that contain the mention syntax (#13151)
Fixes #13150
2020-03-08 15:42:20 +01:00
Eugen Rochko
339ce1c4e9
Add specific rate limits for posting and following (#13172) 2020-03-08 15:17:39 +01:00
Thibaut Girka
85933bc9ff Merge branch 'master' into glitch-soc/merge-upstream 2020-02-25 14:28:13 +01:00
ThibG
cf4fe6caef
Fix misleading error when attempting to re-send a pending follow request (#13133)
Fixes #13131
2020-02-24 21:19:19 +01:00
ThibG
d91946ae13
Fix backups failing when files are missing from media attachments (#13146)
Fixes #13123
2020-02-24 21:18:26 +01:00
ThibG
3704402dcc
Fix duplicate accounts being created when fetching an account for its key only (#13147)
Fixes #13136

When a user's canonical acct domain is different from its id's domain
(WEB_DOMAIN ≠ LOCAL_DOMAIN), two webfinger queries are required to find the
canonical domain from the URI. However, we skip webfinger queries when
updating only the key of a remote user, which led to the creation of a
duplicate account, using the URI's domain instead of the canonical acct: one.
2020-02-24 21:15:53 +01:00
ThibG
983ab40086
Fix previously OStatus-based accounts not being detected as ActivityPub (#13129) 2020-02-22 01:26:41 +01:00
Thibaut Girka
980c004f06 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `Gemfile.lock`:
  No real conflict, glitch-soc-only dependency (redcarpet) too close to an
  upstream one (rdf-normalize)
- `README.md`:
  we have different READMEs, discarded upstream's changes
- `app/views/admin/custom_emojis/index.html.haml`:
  No real conflict, different context because of glitch-soc theming
- `lib/mastodon/statuses_cli.rb`:
  Upstream added code to keep bookmarked statuses, we were already doing so
  with slightly different code. Discarded upstream's changes.
- `package.json`:
  No real conflict, glitch-soc-only dependency (favico.js) too close to
  an upstream one
2020-01-12 15:57:34 +01:00
ThibG
d386d89179 Fix invalid votes from the API being accepted (#12601)
* Fix invalid votes from the API being accepted

Fixes #12556

- Ensure `choice` is an integer instead of silently converting to 0
- Ensure `choice` corresponds to an actual choice of the poll

* Please CodeClimate
2020-01-12 14:17:03 +01:00
Daigo 3 Dango
206dfd7dad Clarify keyword arguments with ** (#12769)
This change is to suppress the warning below on on ruby-2.7.0:

- warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/
2020-01-11 05:57:05 +09:00
ThibG
c306978190 Use quality values in Accept field to explicitly prefer JSON over HTML (#12806) 2020-01-10 00:10:29 +01:00
Thibaut Girka
01eaeab56d Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/controllers/application_controller.rb`:
  Conflict due to theming system.
- `app/controllers/oauth/authorizations_controller.rb`:
  Conflict due to theming system.
2020-01-04 23:04:42 +01:00
ThibG
e4d75f238b Fix URL search not returning private toots user has access to (#12742) 2020-01-03 05:01:45 +01:00
Eugen Rochko
f86ee4b59f
Fix IDN mentions not being processed, IDN domains not being rendered (#12715)
This changes the REST API to return unicode domains in the `acct`
attribute instead of punycode, and to render unicode instead of
punycode on public HTML pages as well.

Fix #7812, fix #12246
2019-12-30 19:20:43 +01:00
chr v1.x
7e074610a6 Fix backups stopping due to read timeouts (#12281)
* Make BackupService resilient to read timeouts

If an attachment read times out, assume that the resources is
inaccessible and continue the backup without it. This fixes #12280.

* Both errors on one line
2019-12-30 03:17:54 +01:00
Thibaut Girka
ab5ff638f2 Merge branch 'master' into glitch-soc/master
Conflicts:
- `config/locales/en.yml`
  No real conflict, upstream added a translatable string “too close” to
  one specific to glitch-soc
- `lib/mastodon/statuses_cli.rb`
  Fixes made upstream, while changed in glitch-soc to keep bookmarked statuses
- `package.json`
  No real conflict, additional dependency in glitch-soc
2019-12-19 13:52:54 +01:00
ThibG
b364175e1b Fix link crawler not specifying accepted content-type (#12646)
The link crawler expects HTML documents, so set the `Accept`
header accordingly.

Fixes #12618
2019-12-18 16:56:06 +01:00
Takeshi Umeda
68070e82cc Add signature to exported status (#12649) 2019-12-18 16:55:21 +01:00
Takeshi Umeda
a391eaf4d8 Fix an error when ActivityPub::FetchRemoteStatusService url is called with nil (#12652) 2019-12-18 16:54:03 +01:00
ThibG
2ee5a9d9c3 Clean up OStatus-related codepaths (#12173)
* Remove “protocol” argument and return value, as only ActivityPub is supported

* Remove FetchRemoteAccountService, only use ActivityPub::FetchRemoteAccountService

* Fix tests
2019-12-17 13:32:57 +01:00
Alice Gaudon
668f698077 Admin setting to disable default follows (#12566) 2019-12-16 23:55:50 +01:00
Thibaut Girka
abcba5c198 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- package.json
  Not really a conflict, caused by an additional dependency in glitch-soc.
- yarn.lock
  Not really a conflict, caused by an additional dependency in glitch-soc.
2019-12-07 12:28:59 +01:00
trwnh
1653ae91ce Fix account search with no query (#12549)
* Fix account search with no query

Modeled after #12541. Fix #12548

* fix codeclimate
2019-12-06 19:44:23 +01:00
Eugen Rochko
bbf926b1c0
Fix error when using search API with no query (#12541)
Fix #12462
2019-12-04 04:34:08 +01:00
ThibG
911cc14481 Add follow_request notification type (#12198)
* Add follow_request notification type

The notification type already existed in the backend but was never pushed
to the front-end. This also means translation strings were also available
for the backend, from the notification mailer.

Unlike other notification types, these are off by default, to match what
I remember of Gargron's view on the topic: that follow requests should not
clutter notifications and should instead be reviewed at the user's own
leisure in the dedicated column.

Since follow requests have their own column, I've deemed it unnecessary to
add a specific tab for them in the notification quick filter.

* Show follow request link in single-column if there are pending requests, even if account isn't locked

* Push follow requests from notifications to the follow_requests list

* Offer to accept or reject follow request from the notification

* Redesign follow request notification
2019-12-01 17:25:29 +01:00
Thibaut Girka
2b7158427f Merge branch 'master' into glitch-soc/merge-upstream 2019-11-30 17:29:44 +01:00
ThibG
00c219aa45 Fix empty poll options not being filtered on remote poll update (#12484)
If a poll contains empty options (which is apparently possible on Pleroma),
it is created without them. However, the poll update code doesn't filter
empty options, and thus:
1. Clear known votes, as it assumes the set of options has changed
2. Errors out because it tries adding empty options, which fails validation

This commit fixes that by filtering them out the same way they are filtered
out at poll creation time.
2019-11-26 22:46:31 +01:00
Thibaut Girka
c0f7e21685 Merge branch 'master' into glitch-soc/merge-upstream 2019-11-23 19:03:01 +01:00
Eugen Rochko
a60b602ade
Fix OEmbed discovery not handling different URL variants in query (#12439)
Fix #12433
2019-11-21 19:03:07 +01:00
ThibG
c656cc2191 Fix FetchLinkCardServices crashing on a tags without a target (#12159)
* Add test for links without targets

* Fix FetchLinkCardServices crashing on a tags without a target
2019-11-21 16:04:52 +01:00
ThibG
c4d45765e3 Change domain block behavior to update user records before deleting data (#12247)
When suspending or silencing a domain, one probably wants the silence or
suspend parts to take effect as soon as possible. Deleting media files can
take a while (possibly days), so perform silencing as soon as possible, and
clean up media afterwards.
2019-11-21 11:36:15 +01:00
Thibaut Girka
ff67385cfb Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- README.md
  discarded upstream changes
- app/controllers/api/v1/bookmarks_controller.rb
  finally merged upstream, some code style fixes
  and slightly changed pagination code
- app/controllers/application_controller.rb
  changed upstream to always return HTML error pages
  slight conflict caused by theming code
- app/models/bookmark.rb
  finally merged upstream, no real conflict
- spec/controllers/api/v1/bookmarks_controller_spec.rb
  finally merged upstream, slightly changed pagination code
2019-11-20 15:36:09 +01:00
Eugen Rochko
d14e74eff5
Add cache for OEmbed endpoints to avoid extra HTTP requests (#12403)
* add youtube oembed endpoint

* add check for oembed endpoint

* change unless for a more readable if

* clear blank lines

* endpoint via https

* Fix string literal in condition

* use cache for endpoints

* use cache for endpoints

* clean up and adding check

* clean up and remove redundant return

* add html check

* add false to return

* use double quotes

* use double quotes

* Clean up
2019-11-17 18:40:33 +01:00
ThibG
dfea7368c9 Add bookmarks (#7107)
* Add backend support for bookmarks

Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.

* Add spec for bookmark endpoints

* Add front-end support for bookmarks

* Introduce OAuth scopes for bookmarks

* Add bookmarks to archive takeout

* Fix migration

* Coding style fixes

* Fix rebase issue

* Update bookmarked_statuses to latest UI changes

* Update bookmark actions to properly reflect status changes in state

* Add bookmarks item to single-column layout

* Make active bookmarks red
2019-11-13 23:02:10 +01:00
ThibG
0d5f3ae331 Fix already-known remote private toots not being searchable by URL (#12336)
* Fix already-known remote private toots not being searchable by URL

Whenever an URL search fails, and an account is provided, try getting a
private status matching that URL from database.

* Fix searching by `id` field instead of `uri`
2019-11-11 00:04:48 +02:00
Thibaut Girka
83774367d2 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`
  conflict because of the poll option limit being different than upstream's
2019-11-07 13:43:56 +01:00
ThibG
66c1fe0495 Fix various issues with account migration (#12301)
* Fix being able to follow oneself by moving to an account that was following the old one

* Add specs

* Add spec to catch MoveWorker issue with local followers following both accounts

* Fix move worker breaking when a local account follows both source and target accounts

* Fix migration from remote to local account not sending Undo Follow

* Fix show_reblogs not being preserved for moved account's followers
2019-11-07 09:05:07 +02:00
Thibaut Girka
0ae7d84273 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- `package.json`
2019-11-06 14:38:28 +01:00
ThibG
65e13cfacf Add abilityto add oneself to lists (#12271)
* Add ability to add oneself to lists

* Change search results to include oneself when searching through followers

* Mark follow relation as optional in ListAccount
2019-11-04 13:02:01 +01:00
Thibaut Girka
41a98b6543 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- README.md
- app/helpers/statuses_helper.rb
  Upstream moved account helpers to their own file, we had extra
  helpers there, moved too.
- app/lib/sanitize_config.rb
- app/models/user.rb
- app/serializers/initial_state_serializer.rb
- config/locales/simple_form.en.yml
- spec/lib/sanitize_config_spec.rb
2019-10-27 12:14:01 +01:00
nightpool
9762fe382c microformat mentions can have an implicit property (#12189)
See the first example here: http://microformats.org/wiki/microformats2#hyperlinked_person
2019-10-24 22:46:15 +02:00
Thibaut Girka
61631f4751 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- README.md
- app/javascript/styles/mastodon/components.scss
  conflicts caused by image URLs being different
- app/models/status.rb
  as_home_timeline removed, kept glitch-soc-only as_direct_timeline
- app/views/statuses/_simple_status.html.haml
- config/locales/en.yml
  some strings were changed upstream
- spec/models/status_spec.rb
  as_home_timeline removed, kept glitch-soc-only as_direct_timeline
2019-10-10 17:26:08 +02:00
Eugen Rochko
8386d9ec10
Fix hashtag timeline REST API accepting too many hashtags (#12091) 2019-10-07 03:56:21 +02:00
Thibaut Girka
1bbba0016a Merge tag 'v3.0.0' into glitch-soc/merge-upstream 2019-10-04 19:49:43 +02:00
Eugen Rochko
eb83d6256e
Add reason param to POST /api/v1/accounts REST API (#12064)
For approval-required registrations mode
2019-10-03 17:50:59 +02:00
Thibaut Girka
a84aedb7a7 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- app/models/media_attachment.rb
  Upstream raised max image size from 8MB to 10MB while our limit is
  configurable. Raised the default to 10MB.
2019-10-03 11:10:12 +02:00
ThibG
3a4d994c40 Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid (#12037)
* Add test to handle suspended and missing users in BootstrapTimelineService

* Fix BootstrapTimelineService crashing when bootstrapped accounts are invalid
2019-10-01 15:10:00 +02:00
Eugen Rochko
9ba40a6bfd
Remove HEAD request from fetching link previews (#12028)
It is not really necessary and we need to reduce requests
2019-10-01 04:54:10 +02:00
abcang
9ef5532a7e Fix follow requests N+1 (#12004) 2019-09-30 17:02:03 +02:00
Thibaut Girka
16ff7c5627 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
- Gemfile
- Gemfile.lock
- app/controllers/about_controller.rb
- app/controllers/auth/sessions_controller.rb
2019-09-30 12:23:57 +02:00
ThibG
3babf8464b Add voters count support (#11917)
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko
368a87755b
Fix account migration not affecting followers on origin server (#11980) 2019-09-29 16:23:13 +02:00
Eugen Rochko
3ec80c7aec
Fix preview card image not being re-fetched even if link is re-posted (#11981)
Fix #11956
2019-09-28 01:33:16 +02:00
Eugen Rochko
ab33c4df94
Add exclude_unreviewed param to GET /api/v2/search REST API (#11977)
Make it so normal search returns even unreviewed matches, but
autosuggestions do not.

Fix #11960
2019-09-28 01:02:21 +02:00
ThibG
18b451c0e6 Change silences to always require approval on follow (#11975)
* Change silenced accounts to require approval on follow

* Also require approval for follows by people explicitly muted by target accounts

* Do not auto-accept silenced or muted accounts when switching from locked to unlocked

* Add `follow_requests_count` to verify_credentials

* Show “Follow requests” menu item if needed even if account is locked

* Add tests

* Correctly reflect that follow requests weren't auto-accepted when local account is silenced

* Accept follow requests from user-muted accounts to avoid leaking mutes
2019-09-27 21:13:51 +02:00