feat: add option to delete old uploads to storacha space #36
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds an option to delete old uploads from Storacha before new builds are uploaded. Key changes include:
- Introducing a new input "storacha-delete-old-builds" in action.yml.
- Adding a new step in action.yml to delete old builds using the Storacha CLI.
- Updating the README to document the new input.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| action.yml | Added a new input and step to delete old builds from Storacha |
| README.md | Updated input table with the new "storacha-delete-old-builds" option |
|
I just got confirmation from Storacha that the |
| storacha-delete-old-builds: | ||
| description: 'Delete old builds from Storacha before uploading new one. Use "true" or "false" (as strings). Be careful with this option as it will delete all uploads to the space, not just the ones from this build.' |
There was a problem hiding this comment.
Is there some key/tag with storacha we could use to ensure only certain items are removed?
There was a problem hiding this comment.
I don't think storacha has an api/option to filter entries, but I suppose we could do client side filtering on the ls results.
There was a problem hiding this comment.
We should replace it with something that "keeps N latest items" or "purge items older than T"
See #36 (comment)
| w3 ls | while read -r cid; do | ||
| if [[ -n "$cid" ]]; then | ||
| echo "Removing $cid" | ||
| w3 rm "$cid" |
There was a problem hiding this comment.
Problem
I worry this will lead to race conditions in default state where people use our action without going extra mile to set up concurrency limits in their workflow.
Running on multiple PRs during busy day could lead to state where recently published preview on one PR is deleted by build from other PR.
Fix
I think if we are able to keep last N CIDs, or only delete "older than a week" then we could remove very old ones, but dont risk PRs rug-pulling each other by removing recently created ones.
Missing timestamps in w3 ?
@alanshaw perhaps i'm looking at sources of https://github.com/storacha/w3cli / https://github.com/storacha/upload-service with sleepy eyes, but the API seems to return timestamps (insertedAt, updatedAt), but then w3 CLI filters them out, even when I pass w3 ls --json (line 172-175 in lib.js explicitly maps only { root, shards })
Could pin creation timestamps be exposed, or am I missing something?
We need them to
- identify "old" uploads - timestamps exist in the API but aren't accessible via CLI
- implement "keep last N by age" - no way to sort by time
What
This is a new option to delete existing uploads in the Storacha space that the
storacha-proofhas permission for.Why
Storacha doesn't deduplicate across uploads. This means that every upload will use up space for the full size of the CAR. If each build is large, this can easily add up to a lot of space.
Open question
w3 delegation create did:key:DID_OF_KEY -c space/blob/add -c space/index/add -c filecoin/offer -c upload/add --base64space/blob/removeandupload/removeto make it possible.