Skip to content

Use IterWithAttributes in the stalePartialBlockLastModifiedTime function if possible #12395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

olegy2008
Copy link

@olegy2008 olegy2008 commented Aug 14, 2025

What this PR does

This PR updates stalePartialBlockLastModifiedTime function to use IterWithAttributes instead of Iter to reduce the amount of object attributes calls if bucket supports objstore.UpdatedAt IterOptionType.

Which issue(s) this PR fixes or relates to

Fixes #12359

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]. If changelog entry is not needed, please add the changelog-not-needed label to the PR.
  • about-versioning.md updated with experimental features.

@olegy2008 olegy2008 requested a review from a team as a code owner August 14, 2025 03:45
@CLAassistant
Copy link

CLAassistant commented Aug 14, 2025

CLA assistant check
All committers have signed the CLA.

@olegy2008
Copy link
Author

I didn't update any tests, since existing test TestStalePartialBlockLastModifiedTime uses filesystem bucket which supports UpdatedAt option, so the existing test covers the new code.

Please tell if I should add test or update existing one to cover the case when UpdatedAt option is not supported.

Copy link
Contributor

@dimitarvdimitrov dimitarvdimitrov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, generally looks ok

Please tell if I should add test or update existing one to cover the case when UpdatedAt option is not supported.

is there a way to wrap the buckt implmentation with our own which returns empty SupportedIterOptions?

Copy link
Contributor

@andyasp andyasp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Both branches should be unit tested.


// If bucket supports UpdatedAt IterOptionType, use IterWithAttributes
// to reduce the amount of object attributes calls.
if slices.Contains(userBucket.SupportedIterOptions(), objstore.UpdatedAt) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be evaluated once upon construction of the blocks cleaner instead of every call. That would also provide a way to unit test both paths even if the test bucket supports this type of listing.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate?
Do you want to add a new bool field to a BlocksCleaner struct, something like supportsUpdatedAt, set it in the NewBlocksCleaner function, and then use in the stalePartialBlockLastModifiedTime function?

@olegy2008
Copy link
Author

Applied suggestions from the review, updated test to also use a filesystem bucket wrapper which doesn't support UpdatedAt option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Idea: Minimize object storage calls in blocks cleaner
4 participants