Description
RedownloadAvatarWorker (and likely RedownloadHeaderWorker) silently fails to save downloaded avatar files when the target directory structure doesn't exist on disk. The worker downloads the file successfully from the remote server but the file is never written to the filesystem.
This occurs after a media cache has been cleared (e.g., via tootctl media remove or manual cleanup). The database retains avatar_file_name and avatar_remote_url but account.avatar.exists? returns false.
Root Cause
The Sidekiq background worker doesn't create the directory structure before writing. In contrast, direct assignment via Rails (account.avatar = tempfile; account.save!) works correctly because Paperclip/ActiveStorage auto-creates directories during the assignment process.
Steps to Reproduce
- Have a running instance with federated accounts that have cached avatars
- Clear the media cache:
tootctl media remove --days 0 or delete files from /system/cache/accounts/avatars/
- Wait for
RedownloadAvatarWorker jobs to process (or trigger via tootctl accounts refresh)
- Check:
Account.where.not(domain: nil).where.not(avatar_file_name: nil).select { |a| !a.avatar.exists? }.count
- Result: Many accounts report
avatar.exists? = false despite having avatar_file_name set
Evidence
account = Account.find_by(username: 'example', domain: 'remote.instance')
account.avatar_file_name # => "abc123.png"
account.avatar.exists? # => false
File.directory?(File.dirname(account.avatar.path)) # => false (directory missing!)
# Manual fix works:
tempfile = URI.open(account.avatar_remote_url)
account.avatar = tempfile
account.save! # Creates directory + saves file
account.avatar.exists? # => true
Workaround
Run periodically via cron:
Account.where.not(domain: nil).where.not(avatar_remote_url: [nil, '']).find_each do |account|
next if account.avatar.exists?
tempfile = URI.open(account.avatar_remote_url, open_timeout: 5, read_timeout: 10)
account.avatar = tempfile
account.save!
rescue
nil
end
Expected Behavior
RedownloadAvatarWorker should ensure the target directory exists (via FileUtils.mkdir_p or equivalent) before attempting to write the downloaded file, matching the behavior of direct Paperclip assignment.
Environment
- Mastodon version: 4.4.16
- Storage: Local filesystem (no S3)
- Triggered after:
tootctl media remove --days 1 cleared avatar cache
- Scale: 29,334 accounts with
avatar_file_name set, only 3,717 files on disk
Description
RedownloadAvatarWorker(and likelyRedownloadHeaderWorker) silently fails to save downloaded avatar files when the target directory structure doesn't exist on disk. The worker downloads the file successfully from the remote server but the file is never written to the filesystem.This occurs after a media cache has been cleared (e.g., via
tootctl media removeor manual cleanup). The database retainsavatar_file_nameandavatar_remote_urlbutaccount.avatar.exists?returnsfalse.Root Cause
The Sidekiq background worker doesn't create the directory structure before writing. In contrast, direct assignment via Rails (
account.avatar = tempfile; account.save!) works correctly because Paperclip/ActiveStorage auto-creates directories during the assignment process.Steps to Reproduce
tootctl media remove --days 0or delete files from/system/cache/accounts/avatars/RedownloadAvatarWorkerjobs to process (or trigger viatootctl accounts refresh)Account.where.not(domain: nil).where.not(avatar_file_name: nil).select { |a| !a.avatar.exists? }.countavatar.exists? = falsedespite havingavatar_file_namesetEvidence
Workaround
Run periodically via cron:
Expected Behavior
RedownloadAvatarWorkershould ensure the target directory exists (viaFileUtils.mkdir_por equivalent) before attempting to write the downloaded file, matching the behavior of direct Paperclip assignment.Environment
tootctl media remove --days 1cleared avatar cacheavatar_file_nameset, only 3,717 files on disk