Add seed file support to Browsertrix backend#2710

Merged

ikreymer merged 61 commits intoissue-2673-seed-filefrom

issue-2673-seed-file-backend-support

Jul 23, 2025

Commits on Jul 23, 2025

Add new file uplooads module with PUT and DELETE endpoints

tw4l
authored and
ikreymer
committed
Add temporary pylint disable comment

tw4l
authored and
ikreymer
committed
Add file upload tests

tw4l
authored and
ikreymer
committed
Add storage quota check

tw4l
authored and
ikreymer
committed
Fixups

tw4l
authored and
ikreymer
committed
Fix linting

tw4l
authored and
ikreymer
committed
WIP: Add seedFileId to workflow POST and PATCH

tw4l
authored and
ikreymer
committed
Pass seed file to crawler, add tests

tw4l
authored and
ikreymer
committed
Remove extra cruft

tw4l
authored and
ikreymer
committed
Use empty str instead of None for seed_file_url

tw4l
authored and
ikreymer
committed
Fix crawlconfig test expected value

tw4l
authored and
ikreymer
committed
Use all lowercase for upload type

tw4l
authored and
ikreymer
committed
Resolve internal path for presigned seed file url

tw4l
authored and
ikreymer
committed
Fix search values to not assume all workflows have seeds

tw4l
authored and
ikreymer
committed
Fix duplicate code lint warning for initing same class

tw4l
authored and
ikreymer
committed
Add slashes to end of page urls in test

tw4l
authored and
ikreymer
committed
Calculate firstSeed and seedCount for seed files on upload

tw4l
authored and
ikreymer
committed
Store seedCount and firstSeed in db for configs and crawls

tw4l
authored and
ikreymer
committed
Include seed file and thumbnail storage in org stats

tw4l
authored and
ikreymer
committed
Modify migration to be more efficient

tw4l
authored and
ikreymer
committed
Add weekly cleanup cron job

tw4l
authored and
ikreymer
committed
Throw exception on job failure so job is marked as unsuccessful

tw4l
authored and
ikreymer
committed
Add additional check on crawlconfig update

tw4l
authored and
ikreymer
committed
Fix docstring

tw4l
authored and
ikreymer
committed
Enforce min_seed_file_crawler_image

tw4l
authored and
ikreymer
committed
Add exception handler for reading cleanup cron job

tw4l
authored and
ikreymer
committed
Set test crawlerChannel to 1.7.0-beta.0

tw4l
authored and
ikreymer
committed
Fix some failing tests

tw4l
authored and
ikreymer
committed
Add back aggregation stage to unset config

tw4l
authored and
ikreymer
committed
Fix expected workflow name

tw4l
authored and
ikreymer
committed
Ensure seed file and thumbnail storage is added to org.bytesStored

tw4l
authored and
ikreymer
committed
Last test fix

tw4l
authored and
ikreymer
committed
Add list endpoint for files

tw4l
authored and
ikreymer
committed
Fix list_user_files default sortBy

tw4l
authored and
ikreymer
committed
Add missing Union import

tw4l
authored and
ikreymer
committed
Check to make sure seed file not in use before deleting

tw4l
authored and
ikreymer
committed
Fix migration

tw4l
authored and
ikreymer
committed
Don't reuse match_query variable name

tw4l
authored and
ikreymer
committed
Add test for deleting in-use seed file

tw4l
authored and
ikreymer
committed
Make seed file cleanup job cron schedule configurable

tw4l
authored and
ikreymer
committed
Add nightly test for seed file cleanup jobs

tw4l
authored and
ikreymer
committed
TEMP FOR TESTING: Run nightly tests on this branch

tw4l
authored and
ikreymer
committed
Fix handling of seed cleanup jobs

tw4l
authored and
ikreymer
committed
Fixup

tw4l
authored and
ikreymer
committed
Remove ops-configs volume from background cron job yaml

tw4l
authored and
ikreymer
committed
Try running job in default namespace

tw4l
authored and
ikreymer
committed
rolebinding: allow creating cronjobs in default namespace
ikreymer
committed
Add back ops-configs volume

tw4l
authored and
ikreymer
committed
Cleanup cronjobs in default namespace on reset

tw4l
authored and
ikreymer
committed
Fix cleanup cron job and tests for it

tw4l
authored and
ikreymer
committed
Rename ImageFile -> UserFile

tw4l
authored and
ikreymer
committed
Extend seed file cleanup test to include file not cleaned up

tw4l
authored and
ikreymer
committed
Remove extra space

tw4l
authored and
ikreymer
committed
Remove reundant resolve_internal_access_path call

tw4l
authored and
ikreymer
committed
Update .github/workflows/k3d-nightly-ci.yaml

ikreymer
and
tw4l
committed
- add resolve_relative_access_path() which will resolve relative urls either to current origin or to internal origin, if no headers provided
ikreymer
committed
make useruploadfile derive from userfile
ikreymer
committed
fix namespace on seed cleanup cronjob check
ikreymer
committed
pass headers for all collection ops
ikreymer
committed
further cleanup:
ikreymer
committed
tests: add host header to be sure of exact host
ikreymer
committed