I wanted to add a plugin to Discourse, so I followed the usual steps for doing so, which involves adding the plugin to the app configuration and then running ./launcher rebuild app.

However, I accidentally forgot that ./launcher rebuild app will also update Discourse -- but the update failed. So, now I'm stuck trying to figure out why the update failed. Looking at the logs, I see:

I, [2025-04-02T08:51:34.524786 #1]  INFO -- : > cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets
`/root` is not writable.
Bundler will use `/tmp/bundler20250402-1714-yuyf6i1714' as your home directory temporarily.
rake aborted!
Aws::S3::Errors::InvalidArgument: Unsupported header 'x-amz-checksum-crc32' received for this API call. (Aws::S3::Errors::InvalidArgument)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in `in_span'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in `span_wrapper'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/request.rb:72:in `send_request'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/client.rb:17315:in `put_object'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2994:in `block in put'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in `metric'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/aws-sdk-s3-1.182.0/lib/aws-sdk-s3/object.rb:2993:in `put'
/var/www/discourse/lib/s3_helper.rb:82:in `upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `block in upload'
/var/www/discourse/lib/tasks/s3.rake:41:in `open'
/var/www/discourse/lib/tasks/s3.rake:41:in `upload'
/var/www/discourse/lib/tasks/s3.rake:197:in `block (2 levels) in <main>'
/var/www/discourse/lib/tasks/s3.rake:197:in `each'
/var/www/discourse/lib/tasks/s3.rake:197:in `block in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => s3:upload_assets
(See full trace by running task with --trace)
I, [2025-04-02T08:51:48.501828 #1]  INFO -- : Installing CORS rules...
skipping
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.br.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.gz.js
Skipping: assets/break_string-cc617154cd957804f2f6a1f3bc68258c9cdca3d4b9a322bf777d145fed04790e.js.map
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.br.js
Skipping: assets/locales/i18n-3b40e842fd72b9bcc74ea83e094c823cd9ca535e4ecc5e78722e6f99d3656137.gz.js
Uploading: assets/scripts/discourse-test-listen-boot-c65930f97c9935680e942f8e32df616cc91ab7c9371b86db6e5ddf9ad868ae22.js

I, [2025-04-02T08:51:48.507455 #1]  INFO -- : Terminating async processes
I, [2025-04-02T08:51:48.508153 #1]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/15/bin/postmaster -D /etc/postgresql/15/main pid: 41
2025-04-02 08:51:48.508 UTC [41] LOG:  received fast shutdown request
I, [2025-04-02T08:51:48.508981 #1]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 108
108:signal-handler (1743583908) Received SIGTERM scheduling shutdown...
108:M 02 Apr 2025 08:51:48.513 # User requested shutdown...
108:M 02 Apr 2025 08:51:48.514 * Saving the final RDB snapshot before exiting.
2025-04-02 08:51:48.518 UTC [41] LOG:  aborting any active transactions
2025-04-02 08:51:48.533 UTC [41] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
2025-04-02 08:51:48.539 UTC [50] LOG:  shutting down
2025-04-02 08:51:48.542 UTC [50] LOG:  checkpoint starting: shutdown immediate
2025-04-02 08:51:48.569 UTC [50] LOG:  checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.013 s, sync=0.003 s, total=0.030 s; sync files=2, longest=0.002 s, average=0.002 s; distance=1 kB, estimate=23 kB
2025-04-02 08:51:48.588 UTC [41] LOG:  database system is shut down
108:M 02 Apr 2025 08:51:48.620 * DB saved on disk
108:M 02 Apr 2025 08:51:48.621 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && sudo -E -u discourse bundle exec rake s3:upload_assets failed with return #<Process::Status: pid 1712 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.2.1/lib/pups/exec_command.rb:132:in `spawn'
exec failed with the params {"cd"=>"$home", "cmd"=>["sudo -E -u discourse bundle exec rake s3:upload_assets", "sudo -E -u discourse bundle exec rake s3:expire_missing_assets"]}
bootstrap failed with exit code 1
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one.
./discourse-doctor may help diagnose the problem.
6edbf13ce0bdc52df819524321e5e6b66bca7dd7ce2c5997262ba8008c6de371

Running discourse-doctor does not successfully diagnose the problem.

First, I looked up the "/root is not writable" error message and found this Discourse support thread. However, there was no resolution to this issue in the thread and the original poster appears to have a working Discourse instance, so I doubt this is the issue.

Looking up the error message about the unsupported header online, I find a Discourse support thread about the same issue dating back to Feb 24th. I use Backblaze for S3 storage and apparently, the AWS SDK developers introduced a breaking change with no regard to third party implementations and won't fix it:

PatPatterson:

Hi - I’m Pat Patterson, Chief Technical Evangelist at Backblaze; I arrived on this thread because I have a self-hosted proof-of-concept Discourse forum, and I happened to bump into this exact issue today while configuring my forum to use Backblaze B2 for backups and uploads.

[...]

The problem is that a checksum (either the Content-MD5 header or one of the new checksum headers) is required (rather than just supported) for these operations, and this causes the current AWS SDKs to provide the new checksum header. As far as I know, there is no way to override this and have the SDK provide Content-MD5 as it used to.

It also appears that the AWS SDK will not be fixed, as the issue was raised but closed as "not planned":

Falco (Discourse team):

Yeah, and AWS SDK maintainers are only giving us the cold shoulder on this.

https://github.com/aws/aws-sdk-js-v3/issues/6819#issuecomment-2625426822

The fix appears to be to remove the expire_missing_assets step:

Falco (Discourse team):

It’s reported that the AWS SDK is not respecting those ENVs for DELETE operations, so you need to remove the

- sudo -E -u discourse bundle exec rake s3:expire_missing_assets

line for now.

However, this also fails with the same error. It appears that the s3:upload_assets step fails before it gets to expiring the missing assets.

Reading earlier in the thread, I see that a different suggestion was made first:

Falco (Discourse team)

Try adding to your ENV

AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED
AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED

Judging by the context in the thread, I'm guessing that this is also required as part of the workaround. So, I try again with the ENV change and also the expire_missing_assets step commented out:

env:
  AWS_REQUEST_CHECKSUM_CALCULATION: WHEN_REQUIRED
  AWS_RESPONSE_CHECKSUM_VALIDATION: WHEN_REQUIRED
  ## elided

## elided

hooks:
  ## elided  
  after_assets_precompile:
    - exec:
        cd: $home
        cmd:
          - sudo -E -u discourse bundle exec rake s3:upload_assets
##        - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

It works!

This does mean that assets won't be deleted from the bucket anymore...

Falco (Discourse team)

Yes, assets won’t be removed from the bucket anymore.

...but I think that's an okay workaround for now.