Hotfix anvilprod: Unfinished MP uploads to mirror buckets expire too soon (#7070)

[nadove-ucsc]

Linked issue: #7070

Checklist

Author

• PR is assigned to the author
• Status of PR is In progress
• Target branch is anvilprod
• Name of PR branch matches hotfixes/<GitHub handle of author>/<issue#>-<slug>-anvilprod
• PR is linked to the issue it hotfixes
• Status of linked issue is In progress
• PR description links to linked issue
• PR title is Hotfix anvilprod: followed by title of linked issue
• PR title references the linked issue

Author (hotfixes)

• Added h tag to commit title _{or this PR does not include a temporary hotfix}
• Added H tag to commit title _{or this PR does not include a permanent hotfix}
• Added hotfix label to PR
• This PR is labeled partial _{or represents a permanent hotfix}

Author (before every review)

• Rebased PR branch on anvilprod, squashed fixups from prior reviews
• Ran make requirements_update _{or this PR does not modify Dockerfile, environment, requirements*.txt, common.mk, Makefile or environment.boot}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• PR is not a draft
• PR is awaiting requested review from system administrator
• Status of PR is Review requested
• PR is assigned to only the system administrator and the author

System administrator (after approval)

• Actually approved the PR
• Decided if PR can be labeled no sandbox
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• N reviews label is accurate
• Status of PR is Approved
• PR is assigned to only the operator and the author

Operator

• Squashed PR branch and rebased onto anvilprod
• Sanity-checked history
• Pushed PR branch to GitHub

Operator (sandbox build)

• Added sandbox label _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab anvilprod _{or PR is labeled no sandbox}
• Build passes in hammerbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in hammerbox deployment _{or PR is labeled no sandbox}
• Deleted unreferenced indices in hammerbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices in hammerbox}
• Started reindex in hammerbox _{or this PR is not labeled reindex:anvilprod}
• Checked for failures in hammerbox _{or this PR is not labeled reindex:anvilprod}
• Started mirroring in hammerbox _{or this PR is not labeled mirror:anvilprod}
• Checked for failures in hammerbox _{or this PR is not labeled mirror:anvilprod}

Operator (merge the branch)

• All status checks passed and the PR is mergeable
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but excluded any p tags}
• Pushed merge commit to GitHub
• Status of PR is Merged stable

Operator (main build)

• Pushed merge commit to GitLab anvilprod
• Build passes on GitLab anvilprod
• Reviewed build logs for anomalies on GitLab anvilprod
• Deleted PR branch from GitHub
• PR is assigned to only the operator
• Deleted PR branch from GitLab anvilprod
• Status of linked issue is Stable

Operator (reindex)

• Deindexed all unreferenced catalogs in anvilprod _{or this PR is neither labeled reindex:partial nor reindex:anvilprod}
• Deindexed specific sources in anvilprod _{or this PR is neither labeled reindex:partial nor reindex:anvilprod}
• Indexed specific sources in anvilprod _{or this PR is neither labeled reindex:partial nor reindex:anvilprod}
• Started reindex in anvilprod _{or neither this PR nor a failed, prior promotion requires it}
• Checked for, triaged and possibly requeued messages in both fail queues in anvilprod _{or neither this PR nor a failed, prior promotion requires it}
• Emptied fail queues in anvilprod _{or neither this PR nor a failed, prior promotion requires it}
• Restarted the Data Browser pipeline for the ucsc/anvil/anvilprod branch on GitLab in anvilprod _{or neither this PR nor a failed, prior promotion requires it}
• Restarted deploy_browser job in the GitLab pipeline for this PR in anvilprod _{or neither this PR nor a failed, prior promotion requires it}
• Created backport PR and linked to it in a comment on this PR

Operator (mirroring)

• Started mirroring in anvilprod _{or neither this PR nor a failed, prior promotion is labelled mirror:anvilprod}
• Checked for, triaged and possibly requeued messages in mirror fail queue in anvilprod _{or neither this PR nor a failed, prior promotion is labelled mirror:anvilprod}
• Emptied mirror fail queue in anvilprod _{or neither this PR nor a failed, prior promotion is labelled mirror:anvilprod}

Operator

• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.67%. Comparing base (b12ca7d) to head (a40fdfa).

Additional details and impacted files

@@            Coverage Diff             @@
##           anvilprod    #8111   +/-   ##
==========================================
  Coverage      84.67%   84.67%           
==========================================
  Files            165      165           
  Lines          24167    24167           
==========================================
  Hits           20463    20463           
  Misses          3704     3704           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coveralls]

coverage: 84.753%. remained the same — hotfixes/nadove-ucsc/7070-unfinished-multipart-uploads-expire-too-soon-anvilprod into anvilprod

[hannes-ucsc]

Security design review

• Security design review completed; this PR does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with Terra
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below