feat(parquet querier): Introduce querier mode for querying parquet blocks #12295
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This update introduces a new `labels` field under the `gateway.ingress`
section in the `values.yaml` file, allowing users to specify custom
labels for the Ingress resource. The corresponding template in
`gateway-ingress.yaml` has been updated to include these labels in the
metadata.
- Added `labels: {}` to `values.yaml`
- Updated `gateway-ingress.yaml` to render labels if provided
This enhancement improves the flexibility of the Ingress configuration
for users deploying the Mimir distributed system.
Checklist
- [X] Tests updated
- [X] Documentation added
- [X] CHANGELOG.md updated - the order of entries should be [CHANGE],
[FEATURE], [ENHANCEMENT], [BUGFIX]
---------
Signed-off-by: unkls <benjamin.leveque97@gmail.com>
Signed-off-by: Vladimir Varankin <vladimir.varankin@grafana.com>
Co-authored-by: Vladimir Varankin <vladimir.varankin@grafana.com>
) Check the size of the entire response being sent to the query-frontend before sending it, not just the body. This fixes edge cases where a response body wasn't larger than the grpc limit but the entire request was. This resulted in timeouts in the query-frontend since the scheduler processor was never able to sent a response to the query-frontend. This is the same issue as #10154 but in the querier worker that pulls from the query-scheduler instead of the query-frontend. Related #10154 Required for #12200 --------- Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> Co-authored-by: Taylor C <41653732+tacole02@users.noreply.github.com>
Fix a copy/paste error from when MQE was added to the query-frontend. Related #11417
This change always configures the query-frontend and querier workers to use the query-scheduler. Part of #11884
#### What this PR does Parallelizes `updateBlock`. We having seen updating the block index take quite a while when there's a lot of new blocks. This seems like a naturally parallelizable problem. #### Checklist - [X] Tests updated. - [X] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features. --------- Co-authored-by: Claude <noreply@anthropic.com>
<!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Rebase your PR if it gets out of sync with main --> #### What this PR does This PR adds release notes for the Mimir Helm chart version 5.8 release. #### Which issue(s) this PR fixes or relates to Fixes grafana/mimir-squad#3152 #### Checklist - [ ] Tests updated. - [X] Documentation added. - [X] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features.
* bring in prometheus/parquet-common code to new package; replace efficient-go errors with pkg/errors; satisfy mimir-prometheus ChunkSeries interface * revert breaking upgrade to thanos/objstore * fix test require * attempt to update go version for strange errors * fix stringlabels issues * update license headers with AGPL and upstream attribution * fix errors.Is lints fix errors.Is lints * fix sort and cancel cause lints * correct go.mod & vendor in from main to solve conflicts * use env var to flag parquet promql acceptance * fix deps from main again * fix deps from main again * fix deps from main again * fix deps from main again implement new parquet-converter service (#11499) * bring in parquet-converter from parquet-mimir PoC * make docs * make reference-help * stop using the compactor's config * remove BlockRanges config, convert all levels of blocks * drop unused BlockWithExtension struct * rename ownBlock to own * move index fetch outside of for loop * lowercase logs * wording: compact => convert * some cleanup * skip blocks for which compaction mark failed download * simplfy convertBlock function * cleanup * Write Compact Mark * remove parquetIndex, we don't neeed it yet at least * use MetaFetcher to discover blocks * make reference-help and mark as experimental * cleanup: we don't need indexes anymore * revert index loader changes * basic TestParquetConverter * make reference-help * lint * happy linter * make docs * fix: correctly initialize memerlist KV for parquet converter * lint: sort lines * more wording fixes: compact => convert * licence header * version 1 * remove parquet-converter from 'backend' and 'all' modules it's experimental and meant to be run alone * address docs feedback * remove unused consts * increase timeout for a test TestPartitionReader_ShouldNotMissRecordsIfKafkaReturnsAFetchBothWithAnErrorAndSomeRecords parquet-converter: Introduce metrics and ring test (#11600) * parquet-converter: Introduce metrics and ring test This commit introduces a ring test to verify that sharding is working as expected. It also introduces metrics to measure total conversions, failures and durations. Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> converter: proper error handling to measure failures parquet converter in docker compose (#11633) * add parquet-converter to docker-compose microservices setup * format jsonnet fix(parquet converter): close TSDB block after conversion (#11635) parquet: vendor back from parquet-common (#11644) introduce store-gateway.parquet-enabled flag & docs (#11722) upgrade prometheus parquet-common dependency (#11723) parquet store-gateways introduce stores interface (#11724) * declare Stores interface satisfied by BucketStores and future Parquet store * add casts to for uses of existing impl which are not protected by interface * stub out parquet bucket stores implementation * most minimal initialization of Parquet Bucket Stores when flag is enabled * license header parquet: Scaffolding for parquet bucket store Series() (#11729) * parquet: Scaffolding for parquet bucket store * use parquetshardopener and be sure to close them * gci pkg/storegateway/parquet_bucket_stores.go Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> --------- Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> fix split between Parquet Stores and each tenant's Store (#11735) fix split between Parquet Stores and each tenant's Store parquet store-gateways blocks sync and lazy reader (#11759) parquet-bucket-store: finish implementing Stores interface (#11772) We're trying to mirror the existing bucket store structure for the parquet implementation and in this PR i'm just trying to implement some of the necessary methods starting with building up the series sets for labels calls. - Series - LabelNames - LabelValues --------- Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> Co-authored-by: Nicolas Pazos <nicolas.pazos-mendez@grafana.com> Co-authored-by: Nicolás Pazos <npazosmendez@gmail.com> fix(parquet): share `ReaderPoolMetrics` instance (#11851) We create multiple instances of `ReaderPool`, passing the registry and creating the metrics on the fly causes panics. fix(parquet store gateway): close things that should be closed (#11865) feat(parquet store gateway): support download labels file without validating (#11866) Signed-off-by: Nicolás Pazos <npazosmendez@gmail.com> Co-authored-by: francoposa <franco@francoposa.io> fix(parquet store gateway): pass blockReader to bucket block constructor (#11875) fix: don't stop nil services fix(parquet store gateways): correctly locate labels parquet files locally (#11894) parquet bucket store: add some debug logging (#11925) Adding few log statements to the existing code path with useful information to understand when and why we are returning 0 series. Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> parquet store gateways: several fixes and basic tests (#11929) Co-authored-by: francoposa <franco@francoposa.io> Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com> parquet converter: include user id in converter counter metrics (#11966) Adding user id to the converter metrics to better track converter progress through tenants. Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> Parquet converter: Implement priority queue for block conversion (#11980) This PR redesigns the parquet converter to use a non-blocking priority queue that prioritises recently uploaded blocks for conversion. * Priority Queue Implementation: - Replaces blocking nested loops with a thread-safe priority queue using container/heap - Blocks are prioritized by ULID timestamp, ensuring older blocks are processed first * Separate block discovery: - There is a new discovery goroutine that periodically discovers users and blocks, enqueuing them for processing - If the block was previously processed it will be marked as converted and skipped the next time its discovered. - There is a new configuration flag `parquet-converter.max-block-age` that allows us to have a rolling window of blocks so we dont queue up all the work at once. We can set this to 30 days and only blocks up to 30 days old will be converted, when the work is completed we can go and increase that window again. - There is a new processing goroutine that continuously consumes from the priority queue and converts blocks - Main Loop remains responsive and handles only service lifecycle events * New metrics - Since we added a priority queue, I added 5 new metrics for queue monitoring: - cortex_parquet_converter_queue_size - Current queue depth - cortex_parquet_converter_queue_wait_time_seconds - Time blocks spend queued - cortex_parquet_converter_queue_items_enqueued_total - Total blocks enqueued - cortex_parquet_converter_queue_items_processed_total - Total blocks processed - cortex_parquet_converter_queue_items_dropped_total - Total blocks dropped when queue closed The idea here is that by looking at the queue metrics we can have an idea of how much scaling up we need to deal with the pending work. Also, before this PR we had no idea of how much work was left to be done but now we will. --------- Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com> fix(parquet store gateway): obey query sharding matchers (#12018) Inefficient, but at least correct query sharding. The new test on sharding fails on the base branch. It's not trivial to add caching to the hashes like the main path does, because we don't have a `SeriesRef` to use as a cache key at the block level (to match what the main path does). We could in theory use something like the row number in the parquet file, but we don't have easy access to that in this part of the code. In any case, the priority right now is correctness, we'll work on optimizing later as appropriate. For referece, see how query sharding is handled on the main path: https://github.com/grafana/mimir/blob/604775d447c0a9e893fa6930ef8f2d403ebe6757/pkg/storegateway/series_refs.go#L1021-L1047 fix(parquet store gateway): panic in Series call with SkipChunks (#12020) `chunksIt` is `nil` when `SkipChunks` is `true`. parquet-converter debug log messages (#12021) Co-authored-by: Jesus Vazquez <jesus.vazquez@grafana.com> chore(parquet): Bump parquet-common dependency (#12023) Brings the last commit from parquet-common [0811a700a852759c16799358b4424d9888afec3f](prometheus-community/parquet-common@0811a70) See link for the diff between the two commits prometheus-community/parquet-common@76512c6...0811a70 --------- Co-authored-by: francoposa <franco@francoposa.io> feature(parquet): Implement store-gateway limits (#12040) This PR is based on the upstream work prometheus-community/parquet-common#81 The idea is to implement a set of basic quota limiters that can protect us against potential bad queries for the gateways. Note we had to bring bits of the code available in the querier in upstream because we have our own chunk querier in Mimir. --------- Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
<!-- Thanks for sending a pull request! Before submitting:
1. Read our CONTRIBUTING.md guide
2. Rebase your PR if it gets out of sync with main
-->
#### What this PR does
```
-blocks-storage.bucket-store.parquet-load-index-to-disk
True to download the Parquet labels file to disk before opening it. False to open it directly from the bucket. (default true)
```
#### Which issue(s) this PR fixes or relates to
Fixes #<issue number>
#### Checklist
- [ ] Tests updated.
- [ ] Documentation added.
- [ ] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [ ]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.
…oks (#12136) Leverage this feature prometheus-community/parquet-common#83 to optimize query sharding. Previously we were materializing all chunks and post-filtering them before sending through the wire. Now we can push down this filter to the materializer and avoid loading them entirely.
…l files (#12151) This PR: * Extends metadata cache to also cache the metadata of parquet label files. (We could add the metadata of parquet chunk files here in the future as well) * Updates the constructor of parquet bucket stores to initialize a caching bucket that its then passed down. In theory when the lazy reader invokes `ParquetBucketOpener.Open() -> bkt.Attributes()`, `CachingBucket.Attributes()` should check cache for `*.labels.parquet` files. --------- Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
…level and deletion marks (#12213) This PR: - Updates the converter with new converter opts as flags - Allows to skip blocks that are below a certain compaction level. This is useful for example if we only want to target 24 h blocks. - Makes converter iterate only through blocks that have not been marked for deletion yet. --------- Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
New flag to disable compression if desired. We're trying to test how much can we trade between CPU usage and latency. Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
…es (#12216) This PR: - Caches parquet labels and parquet chunks get-range calls - Modifies parquet bucket store syncBlocks() method to check whether it has been converted or not before adding it to the list of blocks that can be queried. We had too many errors like: ``` code = Unknown desc = error converting parquet series set to labels and chunks slice: get TSDB schema from lazy reader: lazy open parquet labels file: storage: object doesn't exist: googleapi: Error 404: No such object ```
Tiny k6 script that's helping me generate a bit of load to test the new
store-gateways
```
❯ k6 cloud -e API_URL=${MIMIR_URL} -e USERNAME=${MIMIR_TENANT_ID} -e PASSWORD=${MIMIR_PASSWORD} -e K6_PROJECT_ID=${K6_PROJECT_ID} -e TEST_NAME=parquetQueries k6-test.js
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: cloud
script: k6-test.js
output: https://mystack.grafana.net/a/k6-app/runs/5164848
scenarios: (100.00%) 1 scenario, 2 max VUs, 30m30s max duration (incl. graceful stop):
* parquetQueries: 2 looping VUs for 30m0s (exec: parquetQueries, gracefulStop: 30s)
INFO[0048] 🚀 Load test starting at: 2025-07-30T15:31:23.480Z detected_level=info instance_id=0 lz="amazon:us:columbus" service_name=unknown_service source=console test_run_id=5164848
INFO[0048] Querying CPU usage for random 12-hour window: 2025-05-05T17:29:13.000Z to 2025-05-06T05:29:13.000Z (cache buster: 1753889483) detected_level=info instance_id=0 lz="amazon:us:columbus" service_name=unknown_service source=console test_run_id=5164848
INFO[0048] Range querying: sum(rate(container_cpu_usage_seconds_total{namespace="mimir-dev-11", pod=~"ingester.*"}[5m])) + 1753889483 detected_level=info instance_id=0 lz="amazon:us:columbus" service_name=unknown_service source=console test_run_id=5164848
```
Use your personal tool of preference to have the following environment
variables available to run this:
```
export K6_TOKEN=redacted
export K6_PROJECT_ID=projectid
export MIMIR_URL=url
export MIMIR_TENANT_ID=tenantid
export MIMIR_PASSWORD=redacted
```
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
Signed-off-by: Jesus Vazquez <jesus.vazquez@grafana.com>
jesusvazquez
force-pushed
the
parquet-main
branch
from
ba1f1a3 to
e315f4f
Compare
francoposa
force-pushed
the
parquet-main
branch
3 times, most recently
from
0ddc3eb to
629fb22
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR reintroduces the approach we had at the beginning for querying the blocks from the queriers in https://github.com/grafana/mimir/tree/parquet-mimir but now using the upstream library and with a lot of inspiration from the work our peers at cortex did for managing the query fallback to store-gateways if some blocks have not been converted yet.
The idea is to test this approach and compare it to the results we're getting from the work we're doing in the store-gateways.