As it turns out, I missed this paragraph in the S3 setup instructions for Discourse:

meta.discourse.org

In order to store Discourse static assets in your Object Storage add this configuration on your app.yml under the hooks section:

 after_assets_precompile:
   - exec:
       cd: $home
       cmd:
         - sudo -E -u discourse bundle exec rake s3:upload_assets
         - sudo -E -u discourse bundle exec rake s3:expire_missing_assets

When using object storage, you also need a CDN to serve what gets stored in the bucket. I used StackPath CDN in my testing, and other than needing to set Dynamic Caching By Header: Accept-Encoding in their configuration it works ok.

DISCOURSE_CDN_URL is a CDN that points to you Discourse hostname and caches requests. It will be used mainly for pullable assets: CSS and other theme assets.

DISCOURSE_S3_CDN_URL is a CDN that points to your object storage bucket and caches requests. It will be mainly used for pushable assets: JS, images and user uploads.

We recommend those being different and for admins to set both.

So apparently, first I need to set up a CDN. I decided to use bunny.net. The instructions for setting it up with Backblaze can be found at support.bunny.net.