Commit graph

821 commits

Author SHA1 Message Date
Bearice Ren 28cda42af5
fixes ArgumentError when proxy is used (#20420)
* fixes ArgumentError when proxy is used

* Update app/lib/request.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-11 21:31:03 +01:00
Eugen Rochko f8e8e622e5
Change incoming activity processing to happen in ingress queue (#20264) 2022-11-10 14:21:51 +01:00
James Tucker 78a6b871fe
Improve performance by avoiding regex construction (#20215)
```ruby
10.times { p /#{FOO}/.object_id }
10.times { p FOO_RE.object_id }
```
2022-11-10 05:49:30 +01:00
Claire c989faaa62
Change Request connection logic to try both IPv6 and IPv4 when available (#20108)
Fixes #19751
2022-11-08 16:36:26 +01:00
James Tucker 833d9c2f1c
Improve performance by avoiding method cache busts (#19957)
Switch to monkey-patching http.rb rather than a runtime extend of each
response, so as to avoid busting the global method cache. A guard is
included that will provide developer feedback in development and test
environments should the monkey patch ever collide.
2022-11-08 04:00:27 +01:00
Claire 3114c826a7
Fix filter handling in status cache hydration (#19963) 2022-11-07 19:47:48 +01:00
Claire bb89f83cc0
Fix additional issues with status cache hydration (#19747)
* Spare one SQL query when hydrating polls

* Improve tests

* Fix more discrepancies

* Fix possible crash when the status has no application set
2022-11-04 20:01:33 +01:00
Claire 03b991de6c
Fix various issues with store hydration (#19746)
- Improve tests
- Fix possible crash when application of a reblogged post isn't set
- Fix discrepancies around favourited and reblogged attributes
- Fix discrepancies around pinned attribute
- Fix polls not being hydrated
2022-11-04 19:33:16 +01:00
Claire c2170991c7
Fix reblogs being discarded after the reblogged status (#19731) 2022-11-04 16:31:44 +01:00
Eugen Rochko 5f9e47be34
Add caching for payload serialization during fan-out (#19642) 2022-11-04 13:21:06 +01:00
Eugen Rochko 1546538de9
Fix improperly checking for blocked domain on followed hashtags (#19472)
Fix #19469
2022-10-26 20:40:56 +02:00
Eugen Rochko bf0ab3e0fa
Fix vacuum scheduler missing lock, locks never expiring (#19458)
Remove vacuuming of orphaned preview cards
2022-10-26 12:10:48 +02:00
Eugen Rochko 8f07381856
Revert "Remove preference to aggregate reblogs in home/list feeds (#18112)" (#19463)
This reverts commit af396fa35f.
2022-10-26 09:10:18 +02:00
Claire 30453fab80
Add mention of the translation provider when translating a post (#19433) 2022-10-24 18:37:57 +02:00
Claire 8046cf34d6
Change “Translate” button to only show up when a translation backend is configured (#19434)
* Change “Translate” button to only show up when a translation backend is configured

Fixes #19346

* Add `translation` attribute to /api/v2/instance to expose whether the translation feature is enabled

Fixes #19328
2022-10-24 18:30:58 +02:00
Takeshi Umeda 1d34eff63f
Add featured tag add/remove activity handler (#19408) 2022-10-22 11:49:41 +02:00
Eugen Rochko 839f893168
Change public accounts pages to mount the web UI (#19319)
* Change public accounts pages to mount the web UI

* Fix handling of remote usernames in routes

- When logged in, serve web app
- When logged out, redirect to permalink
- Fix `app-body` class not being set sometimes due to name conflict

* Fix missing `multiColumn` prop

* Fix failing test

* Use `discoverable` attribute to control indexing directives

* Fix `<ColumnLoading />` not using `multiColumn`

* Add `noindex` to accounts in REST API

* Change noindex directive to not be rendered by default before a route is mounted

* Add loading indicator for detailed status in web UI

* Fix missing indicator appearing while account is loading in web UI
2022-10-20 14:35:29 +02:00
Eugen Rochko 43b5d5e38d
Add logged-out access to the web UI (#18961) 2022-09-29 04:39:33 +02:00
Yamagishi Kazutoshi d86dd067ce
Fix auto detect language for translate service (#19244) 2022-09-27 23:33:56 +02:00
Eugen Rochko 5c9abdeff1
Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
Eugen Rochko 0d6b878808
Add user content translations with configurable backends (#19218) 2022-09-23 23:00:12 +02:00
Claire 8cf7006d4e
Refactor ActivityPub handling to prepare for non-Account actors (#19212)
* Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService

ActivityPub::FetchRemoteAccountService is kept as a wrapper for when the actor is
specifically required to be an Account

* Refactor SignatureVerification to allow non-Account actors

* fixup! Move ActivityPub::FetchRemoteAccountService to ActivityPub::FetchRemoteActorService

* Refactor ActivityPub::FetchRemoteKeyService to potentially return non-Account actors

* Refactor inbound ActivityPub payload processing to accept non-Account actors

* Refactor inbound ActivityPub processing to accept activities relayed through non-Account

* Refactor how Account key URIs are built

* Refactor Request and drop unused key_id_format parameter

* Rename ActivityPub::Dereferencer `signature_account` to `signature_actor`
2022-09-21 22:45:57 +02:00
Eugen Rochko 50948b46aa
Add ability to filter followed accounts' posts by language (#19095) 2022-09-20 23:51:21 +02:00
Claire 1145dbd327
Improve error reporting and logging when processing remote accounts (#15605)
* Add a more descriptive PrivateNetworkAddressError exception class

* Remove unnecessary exception class to rescue clause

* Remove unnecessary include to JsonLdHelper

* Give more neutral error message when too many webfinger redirects

* Remove unnecessary guard condition

* Rework how “ActivityPub::FetchRemoteAccountService” handles errors

Add “suppress_errors” keyword argument to avoid raising errors in
ActivityPub::FetchRemoteAccountService#call (default/previous behavior).

* Rework how “ActivityPub::FetchRemoteKeyService” handles errors

Add “suppress_errors” keyword argument to avoid raising errors in
ActivityPub::FetchRemoteKeyService#call (default/previous behavior).

* Fix Webfinger::RedirectError not being a subclass of Webfinger::Error

* Add suppress_errors option to ResolveAccountService

Defaults to true (to preserve previous behavior). If set to false,
errors will be raised instead of caught, allowing the caller to be
informed of what went wrong.

* Return more precise error when failing to fetch account signing AP payloads

* Add tests

* Fixes

* Refactor error handling a bit

* Fix various issues

* Add specific error when provided Digest is not 256 bits of base64-encoded data

* Please CodeClimate

* Improve webfinger error reporting
2022-09-20 23:30:26 +02:00
Eugen Rochko af396fa35f
Remove preference to aggregate reblogs in home/list feeds (#18112) 2022-08-25 23:38:36 +02:00
Jeong Arm 861b35dd54
Support "http_hidden_proxy" ENV var for hidden service only proxy (#18427)
* Support "http_hidden_proxy" ENV var for hidden service only proxy

* Fallback to http_proxy if http_hidden_proxy is not set
2022-08-25 04:41:14 +02:00
Mashiro 656ac99ef0
Fix ambiguous column names in tootctl search deploy (#18993) 2022-08-17 22:07:12 +01:00
Eugen Rochko c3f0621a59
Add ability to follow hashtags (#18809) 2022-07-17 13:49:29 +02:00
Eugen Rochko e7aa2be828
Change how hashtags are normalized (#18795)
* Change how hashtags are normalized

* Fix tests
2022-07-13 15:03:28 +02:00
Eugen Rochko 44b2ee3485
Add customizable user roles (#18641)
* Add customizable user roles

* Various fixes and improvements

* Add migration for old settings and fix tootctl role management
2022-07-05 02:41:40 +02:00
Claire 02851848e9
Revamp post filtering system (#18058)
* Add model for custom filter keywords

* Use CustomFilterKeyword internally

Does not change the API

* Fix /filters/edit and /filters/new

* Add migration tests

* Remove whole_word column from custom_filters (covered by custom_filter_keywords)

* Redesign /filters

Instead of a list, present a card that displays more information and handles
multiple keywords per filter.

* Redesign /filters/new and /filters/edit to add and remove keywords

This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.

* Add /api/v2/filters to edit filter with multiple keywords

Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
  `keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`

API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
  `keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
  `keywords_attributes` can also be passed to edit, delete or add keywords in
   one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
   filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword

* Change from `irreversible` boolean to `action` enum

* Remove irrelevent `irreversible_must_be_within_context` check

* Fix /filters/new and /filters/edit with update for filter_action

* Fix Rubocop/Codeclimate complaining about task names

* Refactor FeedManager#phrase_filtered?

This moves regexp building and filter caching to the `CustomFilter` class.

This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.

* Perform server-side filtering and output result in REST API

* Fix numerous filters_changed events being sent when editing multiple keywords at once

* Add some tests

* Use the new API in the WebUI

- use client-side logic for filters we have fetched rules for.
  This is so that filter changes can be retroactively applied without
  reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
  (e.g. network error, or initial timeline loading)

* Minor optimizations and refactoring

* Perform server-side filtering on the streaming server

* Change the wording of filter action labels

* Fix issues pointed out by linter

* Change design of “Show anyway” link in accordence to review comments

* Drop “irreversible” filtering behavior

* Move /api/v2/filter_keywords to /api/v1/filters/keywords

* Rename `filter_results` attribute to `filtered`

* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer

* Fix systemChannelId value in streaming server

* Simplify code by removing client-side filtering code

The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
2022-06-28 09:42:13 +02:00
Takeshi Umeda 9c571a95db
Fix missing , (#18660) 2022-06-23 23:10:03 +02:00
Claire 28212bd2be
Fix TagManager#local_url? erroring out on invalid URL (#18580) 2022-05-31 18:25:24 +02:00
Eugen Rochko 0a1992430d
Fix errors when rendering RSS feeds (#18531) 2022-05-26 23:02:42 +02:00
Eugen Rochko 96129c2f10
Fix confirmation redirect to app without Location header (#18523) 2022-05-26 22:03:54 +02:00
Eugen Rochko a9b64b24d6
Change algorithm of tootctl search deploy to improve performance (#18463) 2022-05-22 22:16:43 +02:00
Yamagishi Kazutoshi c3fac61f56
Fix Mastodon::RaceConditionError that occurs when external status is reblogged (#18424)
* Fix `Mastodon::RaceConditionError` that occurs when external status is reblogged

* Replace to `@object`
2022-05-16 01:00:09 +02:00
Eugen Rochko 6cf57c6765
Refactor how Redis locks are created (#18400)
* Refactor how Redis locks are created

* Fix autorelease duration on account deletion lock
2022-05-13 00:02:35 +02:00
Eugen Rochko 2b8dc58b7f
Change RSS feeds (#18356)
* Change RSS feeds

- Use date and time for titles instead of ellipsized text
- Use full content in body, even when there is a content warning
- Use media extensions

* Change feed icons and add width and height attributes to custom emojis

* Fix custom emoji animate on hover breaking

* Fix tests
2022-05-09 07:43:08 +02:00
Eugen Rochko 6da648227e
Fix error caused by missing subject in Webfinger response (#18204) 2022-05-01 00:37:46 +02:00
Eugen Rochko 7b0fe4aef9
Fix opening and closing Redis connections instead of using a pool (#18171)
* Fix opening and closing Redis connections instead of using a pool

* Fix Redis connections not being returned to the pool in CLI commands
2022-04-29 22:43:07 +02:00
Eugen Rochko 3917353645
Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
Eugen Rochko b93e2ae0a4
Fix n+1 query on feed merge and populate operations (#18111) 2022-04-26 22:06:31 +02:00
Claire b9197486e5
Fix UnmergeWorker being exceptionally slow in some conditions (#18110) 2022-04-26 21:48:36 +02:00
Claire 0360135d4d
Fix PeerTube videos appearing with an erroneous “Edited at” marker (#18100)
* Fix PeerTube videos appearing with an erroneous “Edited at” marker

PeerTube videos have an `updated` field equal to `published`.
When processing an incoming activity that has the same value for `updated` and
`published`, assume this doesn't represent an actual edit.

* Please CodeClimate
2022-04-26 21:25:26 +02:00
dogelover911 4c9949560d
FeedManager: skip account when target_account's last status is too old (#18009)
Co-authored-by: dogelover911 <dogelover911@localhost>
2022-04-11 21:17:03 +02:00
0x2019 012537452a
Fix error resposes for from search prefix (#17963)
* Fix error responses in `from` search prefix (addresses mastodon/mastodon#17941)

Using unsupported prefixes now reports a 422; searching for posts from an
account the instance is not aware of reports a 404. TODO: The UI for this
on the front end is abysmal.

Searching `from:username@domain` now succeeds when `domain` is the local
domain; searching `from:@username(@domain)?` now works as expected.

* Remove unused methods on new Error classes as they are not being used

Currently when `raise`d there are error messages being supplied, but
this is not actually being used. The associated `raise`s have been
edited accordingly.

* Remove needless comments

* Satisfy rubocop

* Try fixing tests being unable to find AccountFindingConcern methods

* Satisfy rubocop

* Simplify `from` prefix logic

This incorporates @ClearlyClaire's suggestion (see
https://github.com/mastodon/mastodon/pull/17963#pullrequestreview-933986737).

Accepctable account strings in `from:` clauses are more lenient than
before this commit; for example, `from:@user@example.org@asnteo +cat`
will not error, and return posts by @user@example.org containing the
word "cat". This is more consistent with how Mastodon matches mentions
in statuses. In addition, `from` clauses will not be checked for
syntatically invalid usernames or domain names, simply 404ing when
`Account.find_remote!` raises ActiveRecord::NotFound.

New code for this PR that is no longer used has been removed.
2022-04-08 21:21:49 +02:00
Eugen Rochko 8e20e16cf0
Change e-mail notifications to only be sent when recipient is offline (#17984)
* Change e-mail notifications to only be sent when recipient is offline

Change the default for follow and mention notifications back on

* Add preference to always send e-mail notifications

* Change wording
2022-04-08 18:03:31 +02:00
Eugen Rochko 6221b36b27
Remove sign-in token authentication, instead send e-mail about new sign-in (#17970) 2022-04-06 20:58:12 +02:00
Claire ef196c913c
Fix error MethodError in Chewy::Strategy::Sidekiq::Worker (#17912)
Also refactor a bit to reduce code duplication.
2022-03-31 00:49:24 +02:00
Eugen Rochko bbc7afa2a2
Fix being able to post URLs longer than 4096 characters (#17908) 2022-03-30 14:46:03 +02:00
Claire 2c45859ca9
Refactor account and status formatting (#17877)
* Refactor status formatting

* Add account formatting helpers

* Remove StatusFormatter

* Fixup

* Fix copied typo
2022-03-28 01:17:17 +02:00
Eugen Rochko cefa526c6d
Refactor formatter (#17828)
* Refactor formatter

* Move custom emoji pre-rendering logic to view helpers

* Move more methods out of Formatter

* Fix code style issues

* Remove Formatter

* Add inline poll options to RSS feeds

* Remove unused helper method

* Fix code style issues

* Various fixes and improvements

* Fix test
2022-03-26 02:53:34 +01:00
Eugen Rochko 71f2b95106
Fix edits with no actual changes being allowed (#17843)
* Fix edits with no actual changes being allowed locally

* Fix edits with no actual changes being allowed through ActivityPub

* Fix false positive changes caused by description processing in model

* Fix not recording poll expiration update

* Fix test

* Revert changes to ProcessStatusUpdateService

* Various fixes and improvements

* Fix code style issues

* Various changes and improvements

* Add guard clause
2022-03-26 00:38:44 +01:00
Claire f65eaa5aae
Add admin dashboard checks for Elasticsearch version (#17863) 2022-03-25 21:00:59 +01:00
Eugen Rochko b58db8f12e
Add workaround for YouTube Shorts links (#17869)
* Add workaround for YouTube Shorts links

* Update link_details_extractor_spec.rb
2022-03-25 19:31:35 +01:00
Claire c9b3cf8510
Fix searching already-known converted activities by URL (#17814)
Fixes #17813
2022-03-17 00:46:49 +01:00
Eugen Rochko ddbe906c25
Fix not updating a status when newer version is fetched manually (#17745) 2022-03-12 09:11:36 +01:00
Eugen Rochko d17fb70131
Change how changes to media attachments are stored for edits (#17696)
* Change how changes to media attachments are stored for edits

Fix not being able to re-order media attachments

* Fix not broadcasting updates when polls/media is changed through ActivityPub

* Various fixes and improvements

* Update app/models/report.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add tracking of media attachment description changes

* Change poll in status edit to have a structure closer to the real one

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-09 09:06:17 +01:00
Eugen Rochko bd53dd5210
Change design of federation pages in admin UI (#17704)
* Change design of federation pages in admin UI

* Fix query performance in instance media attachments measure

* Fix reblogs being included in instance languages dimension
2022-03-09 08:52:32 +01:00
Eugen Rochko edf09ec747
Add /api/v1/accounts/familiar_followers to REST API (#17700)
* Add `/api/v1/accounts/familiar_followers` to REST API

* Change hide network preference to be stored consistently for local and remote accounts

* Add dummy classes to migration

* Apply suggestions from code review

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-07 09:36:47 +01:00
Eugen Rochko 2ea754b861
Fix duplicate notifications being possible after poll expiration (#17697) 2022-03-04 01:06:33 +01:00
Eugen Rochko 04c3ac896a
Fix infinite re-forwarding of update activities temporarily (#17698) 2022-03-04 01:00:13 +01:00
Eugen Rochko 085f286750
Fix returning empty <p> tag for blank account note in REST API (#17687) 2022-03-03 06:45:46 +01:00
Eugen Rochko 50ea54b3ed
Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Eugen Rochko 0dc57ab6ed
Fix status updates not being forwarded like deletes through ActivityPub (#17648)
Fix #17521
2022-02-26 17:51:59 +01:00
Eugen Rochko 27965ce5ed
Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
luzpaz 73f5e4a1d9
Fix various typos (#17621)
Found via `codespell -q 3 -S ./CHANGELOG.md,./AUTHORS.md,./config/locales,./app/javascript/mastodon/locales -L ba,keypair,medias,ro`
2022-02-22 20:14:17 +01:00
Claire 166f6e4b50
Fix some media attachments being converted with too high framerates (#17619)
Video files with variable framerates are converted to constant framerate videos
and the output framerate picked by ffmpeg is based on the original file's
container framerate (which can be different from the average framerate).

This means that an input video with variable framerate with about 30 frames per
second on average, but a maximum of 120 fps will be converted to a constant 120
fps file, which won't be processed by other Mastodon servers.

This commit changes it so that input files with VFR and a maximum framerate
above the framerate threshold are converted to VFR files with the maximum frame
rate enforced.
2022-02-22 17:11:22 +01:00
Eugen Rochko b377022cf9
Add caching layer to metrics (#17617) 2022-02-22 15:27:08 +01:00
Claire 8f537a1168
Change relays handling to not record boosts (#17571)
* Change relays handling to not record boosts

* Update tests
2022-02-16 14:36:44 +01:00
Claire 73fce8d311
Fix performance of server-side filtering (#17575)
Fixes #17567
2022-02-16 14:28:45 +01:00
Jeong Arm 9a015e43ef
Add from: query operator to search syntax (#16526)
* Add 'by:userhandle' parameter to search api

* Use search syntax for "by" prefix

* Codeclimate

* Use 'from' instead of 'by'
2022-02-14 00:17:09 +01:00
Eugen Rochko 3aebe711fd
Change languages to be listed under standard instead of native name in admin UI (#17485) 2022-02-09 04:15:38 +01:00
Eugen Rochko fd3a45e348
Add edit history to web UI (#17390)
* Add edit history to web UI

* Change history reducer to store items per status

* Fix missing loading prop
2022-02-09 01:17:07 +01:00
Eugen Rochko b6d7726ecb
Remove language detection through cld3 (#17478)
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Eugen Rochko f1f6ddd536
Fix structured data parsing from links choking on bad data (#17403)
* Fix structured data parsing from links choking on bad data

- Fix og:url meta tag being prioritized over canonical link tag
- Fix structured data parsing choking on commented-out CDATA declarations
- Fix HTML entities in title, description, provider_name, author_name
- Change structured data parsing to attempt every JSON-LD script tag

* Remove unnecessary slash escapes from CDATA regex pattern
2022-02-07 18:16:31 +01:00
Claire d1ecc323e7
Compact JSON-LD signed incoming activities (#17426)
Co-authored-by: Puck Meerburg <puck@puck.moe>
2022-02-03 14:07:29 +01:00
Claire 03d59340da
Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Claire a63495230a
Change percent to rate in retention metrics API (#16910) 2022-01-23 16:01:25 +01:00
Eugen Rochko 1060666c58
Add support for editing for published statuses (#16697)
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Eugen Rochko 14f436c457
Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Claire d5c9feb7b7
Add support for private pinned posts (#16954)
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Eugen Rochko fe71548844
Fix warnings on Rails boot (#16946) 2021-12-27 00:47:20 +01:00
Jeong Arm 4a109ec1ba
Fix server graph on admin/tags/:id (#17066) 2021-11-28 23:13:07 +01:00
Eugen Rochko 7de0ee7aba
Remove Keybase integration (#17045) 2021-11-26 05:58:18 +01:00
Eugen Rochko 6e50134a42
Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 3517867b76
Fix confusing error when webfinger request returns empty document (#16986)
For some reason, some misconfigured servers return an empty document when
queried over webfinger. Since an empty document does not lead to a parse
error, the error is not caught properly and triggers uncaught exceptions
later on.

This PR fixes that by immediately erroring out with `Webfinger::Error` on
getting an empty response.
2021-11-14 21:55:40 +01:00
Eugen Rochko 5d7c852283
Fix no link previews being generated for pages with invalid structured data (#16979)
Fix #16955
2021-11-13 23:07:13 +01:00
Claire ef8deb0878
Fix OpenGraph/LinkedData embeds having incorrectly-generated iframes (#16978) 2021-11-13 03:30:27 +01:00
Eugen Rochko 39cdf61ab7
Add support for structured data and more OpenGraph tags to link cards (#16938)
Save preview cards under their canonical URL

Increase max redirects to follow from 2 to 3
2021-11-05 23:23:05 +01:00
Claire a184c860ca
Fix retention rate computation (#16854) 2021-10-14 22:20:37 +02:00
Eugen Rochko 07341e7aa6
Add graphs and retention metrics to admin dashboard (#16829) 2021-10-14 20:44:59 +02:00
Claire fc3ae1343d
Switch from unmaintained paperclip to kt-paperclip (#16724)
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Eugen Rochko 52e5c07948
Change routing paths to use usernames in web UI (#16171) 2021-09-26 05:46:13 +02:00
Claire 12cd097e7c
Fix addressing of remote groups' followers (#16700)
Fixes #16699
2021-09-08 23:33:23 +02:00
Takeshi Umeda 709876bd6c
Fix invalid blurhash handling in Create activity (#16583) 2021-08-09 13:33:19 +02:00
Claire d8629e7b86
Add logging of S3-related errors (#16381) 2021-07-21 18:34:39 +02:00
Claire 211d5c3c30
Fix inefficiencies in auto-linking code (#16506)
The auto-linking code basically rewrote the whole string escaping non-ascii
characters in an inefficient way, and building a full character offset map
between the unescaped and escaped texts before sending the contents to
TwitterText's extractor.

Instead of doing that, this commit changes the TwitterText regexps to include
valid IRI characters in addition to valid URI characters.
2021-07-15 15:56:58 +02:00
Claire 9a19227f17
Fix some RedisLocks auto-releasing too fast (#16276)
* Fix Delete and Create-related locks expiring too fast

Fixes #16238

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes #15645

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
abcang b5ad787ebf
Fix rubocop warning (#16214) 2021-05-11 14:19:01 +02:00
Claire afb7882189
Fix blocking someone not clearing up list feeds (#16205) 2021-05-10 17:31:55 +02:00
Eugen Rochko 74081433d0
Change trending hashtags to be affected be reblogs (#16164)
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Takeshi Umeda 7cb34b32f8
Add management of delivery availability in Federation settings (#15771)
* Add management of delivery availavility in Federation settings

* fix translate

* Remove useless object creation

* Fix DeepSource issue

* Add shortcut for all

* Fix DeepSource(skipcq)

* Change 'remove' to 'clear'

* Fix style

* Change class method name (exhausted_deliveries_key_by)
2021-05-05 23:39:02 +02:00
Eugen Rochko 036556d350
Fix media processing getting stuck on too much stdin/stderr (#16136)
* Fix media processing getting stuck on too much stdin/stderr

See thoughtbot/terrapin#5

* Remove dependency on paperclip-av-transcoder gem

* Remove dependency on streamio-ffmpeg gem

* Disable stdin on ffmpeg process
2021-05-05 19:44:01 +02:00
Eugen Rochko daccc07dc1
Change auto-following admin-selected accounts, show in recommendations (#16078) 2021-04-24 17:01:43 +02:00
Eugen Rochko f4b7c6b619
Fix nil error when removing status caused by race condition (#16099) 2021-04-24 13:35:39 +02:00
Eugen Rochko 2eb17360df
Fix delete of local reply to local parent not being forwarded (#16096) 2021-04-22 14:26:11 +02:00
Claire 0b36e3419d
Fix processing of remote Delete activities (#16084)
* Add tests

* Ensure deleted statuses are marked as such

* Save some redis memory by not storing URIs in delete_upon_arrival values

* Avoid possible race condition when processing incoming Deletes

* Avoid potential duplicate Delete forwards

* Lower lock durations to reduce issues in case of hard crash of the Rails process

* Check for `lock.aquired?` and improve comment

* Refactor RedisLock usage in app/lib/activitypub

* Fix using incorrect or non-existent sender for relaying Deletes
2021-04-21 04:46:09 +02:00
Eugen Rochko ca3bc1b09f
Refactor StatusReachFinder to handle followers and relays as well (#16051) 2021-04-17 15:41:57 +02:00
Eugen Rochko 6d6000f61f
Fix remote reporters not receiving suspend/unsuspend activities (#16050) 2021-04-17 14:55:46 +02:00
Eugen Rochko dde8739020
Fix reports of already suspended accounts being recorded (#16047) 2021-04-16 22:01:05 +02:00
Eugen Rochko 3b8d085436
Fix app name, website and redirect URIs not having a maximum length (#16042)
Fix app scopes not being validated
2021-04-15 16:28:43 +02:00
Takeshi Umeda c968d22ee9
Fix an error with 'multiple mentions with same username' (#16038) 2021-04-14 15:48:49 +02:00
Eugen Rochko f7117646af
Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko 619fad6cf8
Remove spam check and dependency on nilsimsa gem (#16011) 2021-04-11 11:22:50 +02:00
Eugen Rochko 7183d9a113
Change multiple mentions with same username to render with domain (#15718)
Fix #15506
2021-04-10 11:51:02 +02:00
Claire c5fe0864d1
Fix SidekiqProcessCheck checking for a queue name that isn't used in Mastodon (#16002) 2021-04-05 23:41:37 +02:00
Eugen Rochko 487e37d6d4
Add system checks to dashboard in admin UI (#15989) 2021-04-03 14:12:30 +02:00
Claire cbd0ee1d07
Update Mastodon to Rails 6.1 (#15910)
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire 1c4dee4554
Fix Mastodon not understanding as:Public and Public (#15948)
Fixes #5551
2021-03-24 10:19:40 +01:00
Claire a4dcaef53b
Prepare Mastodon for zeitwerk autoloader (#15917)
* Prepare Mastodon for zeitwerk autoloader (Rails 6)

Add inflections and rename/move a few classes.

In particular, app/lib/exceptions.rb and app/lib/sanitize_config.rb
were manually loaded while still in autoload paths.

* Add inflection for Url → URL
2021-03-19 02:42:43 +01:00
Claire 5027abecd1
Fix cache_collection crashing when given an empty collection (#15921)
* Fix cache_collection crashing when given an empty collection

* Add tests
2021-03-18 00:41:32 +01:00
Claire 43eff898a0
Prepare Mastodon for Rails 6 (#15911)
* Fix misuse of foreign_type

* Fix use of removed "add_template_helper"

* Use response.media_type instead of response.content_type in tests

* Fix CSV export controller test on Rails 6

Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.

This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.

* Fix emoji formatting with Rails 6

* Make emoji output more idiomatic and robust

* Switch from redis-rails gem to built-in Rails redis cache storage
2021-03-17 10:09:55 +01:00
Claire 5614e6724e
Fix URL scanning in note length validator and preview card fetching (#15827)
* Add tests

* Fix URL scanning in note length validator and preview card fetching
2021-03-04 00:12:26 +01:00
Claire 65db262550
Update twitter-text from 1.14 to 3.1.0 and fix toot character counting (#15382)
* Update twitter-text from 1.14 to 3.1.0

* Disable emoji parsing

* Properly depend on twitter-text for url detection

* Fix some URLs being wrongly detected client-side

* Add test for server-side validation of non-autolinkable URLs

* Fix server-side status length counting
2021-03-02 12:02:56 +01:00
Eugen Rochko 9aa37b32c3
Add details to error response for POST /api/v1/accounts in REST API (#15803) 2021-03-01 04:59:13 +01:00
Justin Tracey c9e8e1739c
replace all instances of "ends_with?" with "end_with?" (#15745)
The "ends_with?" method is just a Rails alias of Ruby's "end_with?" method.
Using the latter makes the code less brittle.
2021-02-19 09:56:14 +01:00
Cecylia Bocovich e79f8dd85c
Onion service related changes to HTTPS handling (#15560)
* Enable secure cookie flag for https only

* Disable force_ssl for .onion hosts only

Co-authored-by: Aiden McClelland <me@drbonez.dev>
2021-02-11 04:40:13 +01:00
Claire be3b9f8151
Fix URI of repeat follow requests not being recorded (#15662)
* Fix URI of repeat follow requests not being recorded

In case we receive a “repeat” or “duplicate” follow request, we automatically
fast-forward the accept with the latest received Activity `id`, but we don't
record it.

In general, a “repeat” or “duplicate” follow request may happen if for some
reason (e.g. inconsistent handling of Block or Undo Accept activities, an
instance being brought back up from the dead, etc.) the local instance thought
the remote actor were following them while the remote actor thought otherwise.

In those cases, the remote instance does not know about the older Follow
activity `id`, so keeping that record serves no purpose, but knowing the most
recent one is useful if the remote implementation at some point refers to it
by `id` without inlining it.

* Add tests
2021-02-11 01:53:44 +01:00
ThibG a044ddac5b
Fix race conditions on account migration creation (#15597)
* Atomically check for processing lock in Move handler

* Prevent race condition when creating account migrations

Fixes #15595

* Add tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-02-02 14:49:57 +01:00
luigi 7ea9588520
Use Enumerable#filter_map in more places (#15527) 2021-01-22 16:28:15 +01:00
luigi eb51e43fb4
Optimize some regex matching (#15528)
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
2021-01-22 10:09:08 +01:00
ThibG 54d4e5252b
Use Rails' index_by where it makes sense (#15542)
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:27:38 +01:00
ThibG 7bed25f3ea
Fix processing of incoming Block activities (#15546)
Unlike locally-issued blocks, they weren't clearing follow
relationships in both directions, follow requests or notifications.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:25:01 +01:00
luigi 087ed84367
Optimize map { ... }.compact calls (#15513)
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
ThibG 3249d35bdc
Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
Eugen Rochko 1045549f85
Add stoplight for object storage failures, return HTTP 503 (#13043) 2020-12-15 12:55:29 +01:00
trwnh 127c543a6e
rename replies_policy enumerables (#15304) 2020-12-09 04:34:17 +01:00
Eugen Rochko e7e099d1a0
Fix deletes not reaching every server that interacted with status (#15200)
Extract logic for determining ActivityPub inboxes to send deletes
to to its own class and explicitly include the person the status
replied to (even if not mentioned), people who favourited it, and
people who replied to it (though that one is still not recursive)
2020-11-27 15:48:31 +01:00
ThibG 2f6831f318
Fix sending spurious Rejects when processing remote account deletion (#15104)
* Fix sending spurious Rejects when processing remote account deletion

* Make skip_side_effects imply skip_activitypub
2020-11-19 17:39:47 +01:00
Eugen Rochko df1653174b
Add cache buster feature for media files (#15155)
Nginx can be configured to bypass proxy cache when a special header
is in the request. If the response is cacheable, it will replace
the cache for that request. Proxy caching of media files is
desirable when using object storage as a way of minimizing bandwidth
costs, but has the drawback of leaving deleted media files for
a configured amount of cache time. A cache buster can make those
media files immediately unavailable. This especially makes sense
when suspending and unsuspending an account.
2020-11-19 17:38:06 +01:00
Eugen Rochko aa10200e58
Fix streaming API allowing connections to persist after access token invalidation (#15111)
Fix #14816
2020-11-12 23:05:24 +01:00
ThibG b0893291ef
Fix Move handler not being triggered when failing to fetch target (#15107)
When failing to fetch the target account, the ProcessingWorker fails
as expected, but since it hasn't cleared the `move_in_progress` flag,
the next attempt at processing skips the `Move` activity altogether.

This commit changes it to clear the flag when encountering any
unexpected error on fetching the target account. This is likely to
occur because, of, e.g., a timeout, when many instances query the
same actor at the same time.
2020-11-09 01:41:52 +01:00
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub (#14989) 2020-11-08 00:28:39 +01:00
Takeshi Umeda d6fe0c94ca
Add account sensitized (#14361)
* Add account sensitized

* Fix i18n normalize

* Fix description and spec

* Fix spec

* Fix wording
2020-11-04 20:45:01 +01:00
ThibG ca56527140
Add follower synchronization mechanism (#14510)
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
abcang 9649ca0fbe
Removed disabling comments for Style/MethodMissingSuper (#15014)
* Removed disabling comments for Style/MethodMissingSuper

* Update rubocop for codeclimate
2020-10-20 09:13:39 +02:00
Josh Leeb-du Toit 0c24f4dce2
Add support for Gemini urls (#15013)
This PR updates the `valid_url` regex and sanitizer allowlist to provide
support for Gemini urls.

Closes #14991
2020-10-19 17:02:13 +02:00
Takeshi Umeda b4c4af18dc
Fix a bear check when the activity object is nil (#14981) 2020-10-13 22:41:00 +02:00