I wanted to add a plugin to Discourse, so I followed the usual steps for doing so, which involves adding the plugin to the app configuration and then running ./launcher rebuild app.
However, I accidentally forgot that ./launcher rebuild app will also update Discourse — but the update failed. So, now I’m stuck trying to figure out why the update failed. Looking at the logs, I see:
I, [2025-04-02T08:51:34.524786 #1] INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
`/root` is not writable.
Bundler will use `/tmp/bundler20250402-1714-yuyf6i1714' as your home directory temporarily.
rake aborted!
Aws::S3::Errors::InvalidArgument: Unsupported header 'x-amz-checksum-crc32' received for this API call. (Aws::S3::Errors::InvalidArgument)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/client.rb:17315:in `put_object'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2994:in `block in put'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in `metric'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2993:in `put'
/var/www/discourse/lib/s3_helper.rb:82:in `upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `block in upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `open'
/var/www/discourse/lib/tasks/s3.rake:41:in `upload'
/var/www/discourse/lib/tasks/s3.rake:197:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/s3.rake:197:in `each'
/var/www/discourse/lib/tasks/s3.rake:197:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => s3:upload_assets
(See full trace by running task with --trace)
I, [2025-04-02T08:51:48.501828 #1] INFO -- : Installing CORS rules...
skipping
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.br.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.gz.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js.map
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.br.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.gz.js
Uploading: assets/scripts/discourse-test-listen-boot-c65930f97c9935680e942f8e32df616cc91ab7c9371b86db6e5ddf9ad868ae22.js
I, [2025-04-02T08:51:48.507455 #1] INFO -- : Terminating async processes
I, [2025-04-02T08:51:48.508153 #1] INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main pid: 41
2025-04-02 08:51:48.508 UTC [41] LOG: received fast shutdown request
I, [2025-04-02T08:51:48.508981 #1] INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 108
108:signal-handler (1743583908) Received SIGTERM scheduling shutdown...
108:M 02 Apr 2025 08:51:48.513 # User requested shutdown...
108:M 02 Apr 2025 08:51:48.514 * Saving the final RDB snapshot before exiting.
2025-04-02 08:51:48.518 UTC [41] LOG: aborting any active transactions
2025-04-02 08:51:48.533 UTC [41] LOG: background worker "logical replication launcher" (PID 55) exited with exit code 1
2025-04-02 08:51:48.539 UTC [50] LOG: shutting down
2025-04-02 08:51:48.542 UTC [50] LOG: checkpoint starting: shutdown immediate
2025-04-02 08:51:48.569 UTC [50] LOG: checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.013 s, sync=0.003 s, total=0.030 s; sync files=2, longest=0.002 s, average=0.002 s; distance=1 kB, estimate=23 kB
2025-04-02 08:51:48.588 UTC [41] LOG: database system is shut down
108:M 02 Apr 2025 08:51:48.620 * DB saved on disk
108:M 02 Apr 2025 08:51:48.621 # Redis is now ready to exit, bye bye...
FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 1712 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
6edbf13ce0bdc52df819524321e5e6b66bca7dd7ce2c5997262ba8008c6de371
Running discourse-doctor does not successfully diagnose the problem.
First, I looked up the “/root is not writable” error message and found this Discourse support thread. However, there was no resolution to this issue in the thread and the original poster appears to have a working Discourse instance, so I doubt this is the issue.
Looking up the error message about the unsupported header online, I find a Discourse support thread about the same issue dating back to Feb 24th. I use Backblaze for S3 storage and apparently, the AWS SDK developers introduced a breaking change with no regard to third party implementations and won’t fix it:
Hi - I’m Pat Patterson, Chief Technical Evangelist at Backblaze; I arrived on this thread because I have a self-hosted proof-of-concept Discourse forum, and I happened to bump into this exact issue today while configuring my forum to use Backblaze B2 for backups and uploads.
[…]
The problem is that a checksum (either the Content-MD5 header or one of the new checksum headers) is required (rather than just supported) for these operations, and this causes the current AWS SDKs to provide the new checksum header. As far as I know, there is no way to override this and have the SDK provide Content-MD5 as it used to.
It also appears that the AWS SDK will not be fixed, as the issue was raised but closed as “not planned”:
Judging by the context in the thread, I’m guessing that this is also required as part of the workaround. So, I try again with the ENV change and also the expire_missing_assets step commented out:
