feat(parquet querier): Introduce querier mode for querying parquet blocks

CHANGELOG.md

@@ -12,12 +12,14 @@
 * [CHANGE] Query-frontend: Remove the CLI flag `-query-frontend.downstream-url` and corresponding YAML configuration and the ability to use the query-frontend to proxy arbitrary Prometheus backends. #12191
 * [CHANGE] Query-frontend: Remove experimental instant query splitting feature. #12267
 * [FEATURE] Distributor, ruler: Add experimental `-validation.name-validation-scheme` option to specify the validation scheme for metric and label names. #12215
+* [ENHANCEMENT] Compactor: Add `-compactor.update-blocks-concurrency` flag to control concurrency for updating block metadata during bucket index updates, separate from deletion marker concurrency. #12117
 * [ENHANCEMENT] Stagger head compaction intervals across zones to prevent compactions from aligning simultaneously, which could otherwise cause strong consistency queries to fail when experimental ingest storage is enabled. #12090
 * [ENHANCEMENT] Querier: Add native histogram definition to `cortex_bucket_index_load_duration_seconds`. #12094
 * [ENHANCEMENT] MQE: Add support for applying common subexpression elimination to range vector expressions in instant queries. #12236
 * [BUGFIX] Querier: Samples with the same timestamp are merged deterministically. Previously, this could lead to flapping query results when an out-of-order sample is ingested that conflicts with a previously ingested in-order sample's value. #8673
 * [BUGFIX] Store-gateway: Fix potential goroutine leak by passing the scoped context in LabelValues. #12048
 * [BUGFIX] Distributor: Fix pooled memory reuse bug that can cause corrupt data to appear in the err-mimir-label-value-too-long error message. #12048
+* [BUGFIX] Querier: Fix timeout responding to query-frontend when response size is very close to `-querier.frontend-client.grpc-max-send-msg-size`. #12261
 
 ### Jsonnet
 

cmd/mimir/help.txt.tmpl

@@ -543,6 +543,26 @@ Usage of ./cmd/mimir/mimir:
     	List of network interface names to look up when finding the instance IP address. (default [<private network interfaces>])
   -overrides-exporter.ring.store string
     	Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist")
+  -parquet-converter.conversion-interval duration
+    	The frequency at which the conversion runs. (default 1m0s)
+  -parquet-converter.data-dir string
+    	Directory to temporarily store blocks during conversion. This directory is not required to persist between restarts. (default "./data-parquet-converter/")
+  -parquet-converter.discovery-interval duration
+    	The frequency at which user and block discovery runs. (default 5m0s)
+  -parquet-converter.max-block-age duration
+    	Maximum age of blocks to convert. Blocks older than this will be skipped. Set to 0 to disable age filtering.
+  -parquet-converter.ring.consul.hostname string
+    	Hostname and port of Consul. (default "localhost:8500")
+  -parquet-converter.ring.etcd.endpoints string
+    	The etcd endpoints to connect to.
+  -parquet-converter.ring.etcd.password string
+    	Etcd password.
+  -parquet-converter.ring.etcd.username string
+    	Etcd username.
+  -parquet-converter.ring.instance-interface-names string
+    	List of network interface names to look up when finding the instance IP address. (default [<private network interfaces>])
+  -parquet-converter.ring.store string
+    	Backend storage to use for the ring. Supported values are: consul, etcd, inmemory, memberlist, multi. (default "memberlist")
   -print.config
     	Print the config and exit.
   -querier.cardinality-analysis-enabled

development/mimir-microservices-mode/docker-compose.jsonnet

@@ -35,6 +35,11 @@ std.manifestYamlDoc({
 
     // If true, a query-tee instance with a single backend is started.
     enable_query_tee: false,
+
+    // If true, start parquet-converter
+    enable_parquet: false,
+    // If true, emulate classic store-gateway behavior by loading the Parquet labels file to disk
+    parquet_store_gateway_load_index_to_disk: false,
   },
 
   // We explicitely list all important services here, so that it's easy to disable them by commenting out.
@@ -44,6 +49,7 @@ std.manifestYamlDoc({
     self.read_components +  // Querier, Frontend and query-scheduler, if enabled.
     self.store_gateways(3) +
     self.compactor +
+    (if $._config.enable_parquet then self.parquet_converter else {}) +
     self.rulers(2) +
     self.alertmanagers(3) +
     self.nginx +
@@ -142,6 +148,14 @@ std.manifestYamlDoc({
     }),
   },
 
+  parquet_converter:: {
+    'parquet-converter': mimirService({
+      name: 'parquet-converter',
+      target: 'parquet-converter',
+      httpPort: 8040,
+    }),
+  },
+
   rulers(count):: if count <= 0 then {} else {
     ['ruler-%d' % id]: mimirService({
       name: 'ruler-' + id,
@@ -164,12 +178,20 @@ std.manifestYamlDoc({
     for id in std.range(1, count)
   },
 
+  local extraStoreGatewayArgs = std.join(
+    ' ', [
+      (if $._config.enable_parquet then '-log.level=debug -store-gateway.parquet-enabled=true' else null),
+      (if ($._config.enable_parquet && !$._config.parquet_store_gateway_load_index_to_disk) then '-blocks-storage.bucket-store.parquet-load-index-to-disk=false' else null),
+    ]
+  ),
+
   store_gateways(count):: {
     ['store-gateway-%d' % id]: mimirService({
       name: 'store-gateway-' + id,
       target: 'store-gateway',
       httpPort: 8010 + id,
       jaegerApp: 'store-gateway-%d' % id,
+      extraArguments: extraStoreGatewayArgs,
     })
     for id in std.range(1, count)
   },

docs/sources/helm-charts/mimir-distributed/release-notes/v5.8.md

@@ -0,0 +1,32 @@
+---
+title: "Grafana Mimir Helm chart version 5.8 release notes"
+menuTitle: "V5.8 release notes"
+description: "Release notes for Grafana Mimir Helm chart version 5.8"
+weight: 300
+---
+
+# Grafana Mimir Helm chart version 5.8 release notes
+
+Grafana Labs is excited to announce version 5.8 of the Grafana Mimir Helm chart, which is compatible with Grafana Mimir v2.17 and Grafana Enterprise Metrics (GEM) v2.17. The `mimir-distributed` Helm chart is the best way to install Mimir on Kubernetes.
+
+The highlights that follow include the top features, enhancements, and bug fixes in this release. For a comprehensive list of changes, see the [Helm chart changelog](https://github.com/grafana/mimir/tree/main/operations/helm/charts/mimir-distributed/CHANGELOG.md).
+
+## Important changes
+
+KEDA autoscaling has changed `kedaAutoscaling.toPromQLLabelSelector` from an object to a list of strings, adding support for all PromQL operators.
+
+The Memcached timeout for the ruler-storage cache has increased from 200 milliseconds to 500 milliseconds.
+
+Grafana Mimir now exports OTel traces instead of Jaeger traces. As a result, the environment variable `JAEGER_REPORTER_MAX_QUEUE_SIZE` is no longer set by default. Components use OTel's default value of 2048 unless explicitly configured. You can still configure `JAEGER_REPORTER_MAX_QUEUE_SIZE` if you configure tracing using Jaeger environment variables, and you can always set `OTEL_BSP_MAX_QUEUE_SIZE` for OTel configuration. For more information, refer to [Configure Grafana Mimir tracing](https://grafana.com/docs/mimir/<MIMIR_VERSION>/configure/configure-tracing/) in the Mimir documentation.
+
+The rollout-operator has been updated to the latest release, which includes support for OTEL\_ tracing environment variables.
+
+Memberlist configuration for ingesters now enables `memberlist.abort-if-fast-join-fails`.
+
+Memcached now sets resource requests for the Memcached Prometheus exporter by default.
+
+A new `store_gateway.grpcMaxQueryResponseSizeBytes` value has been added to set the maximum store-gateway gRPC query response send size, and corresponding querier receive size, with a default of 200MB.
+
+## Features and enhancements
+
+There are new values for setting annotations and labels for the rollout-operator.