Issue with Query mode

[velu4689]

Hi Team,

Currently am facing the following issue:

internal: expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed)

Can you pls help me on providing an explanation under what circumstances, we receive this issue ?

Following is my config:

# Do not use this configuration in production.
# It is for demonstration purposes only.

# Do not use this configuration in production.
# It is for demonstration purposes only.

target: all,alertmanager,overrides-exporter

common:
  storage:
    backend: s3
    s3:
      endpoint: s3.us-west-2.amazonaws.com
      region: us-west-2
      bucket_name: <bucket_name>

blocks_storage:
  backend: s3
  storage_prefix: blocks
  bucket_store:
    sync_interval: 15m
  tsdb:
    retention_period: 2h
    ship_interval: 5m
    dir: /data/tsdb

memberlist:
  join_members: [<IP1>, <IP2>, <IP3>]

tenant_federation:
  enabled: true

compactor:
  data_dir: /data/compactor
  compaction_interval: 15m
  cleanup_interval: 15m
  sharding_ring:
    instance_addr: <IP1>
    kvstore:
      store: memberlist

querier:
  query_ingesters_within: 5h
  query_store_after: 2h
  max_query_into_future: 12h

frontend:
 #align_queries_with_step: true
 log_queries_longer_than: 10s

distributor:
  ring:
    instance_addr: <IP1>
    kvstore:
      store: memberlist
  ha_tracker:
    enable_ha_tracker: true

ingester:
  ring:
    instance_addr: <IP1>
    kvstore:
      store: memberlist
    replication_factor: 3

alertmanager:
  data_dir: /data/alertmanager
  external_url: http://<IP1>:9009/alertmanager

ruler:
  rule_path: /data/ruler
  alertmanager_url: http://<IP1>:8080/alertmanager
  ring:
    # Quickly detect unhealthy rulers to speed up the tutorial.
    heartbeat_period: 15s
    heartbeat_timeout: 1m

ruler_storage:
  backend: s3
  storage_prefix: ruler

server:
  http_listen_address: "<IP1>"
  http_listen_port: 9009
  log_level: info

store_gateway:
  sharding_ring:
    instance_addr: <IP1>
    kvstore:
      store: memberlist
      prefix: "/data/store_collectors"
    replication_factor: 3

[velu4689]

caller=retry.go:78 level=error user=spark-usw2-worker-monitor msg="error processing request" try=3 err="rpc error: code = Code(500) desc = {\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01H1Q6KRC0BEETPDG9AKE3TWJ5 01H1Q6KEN3D9B0SNK29CEP61PJ\"}

[pstibrany]

This error is returned when querier cannot read data from specified blocks. Querier asks store-gateway instances for chunks from those blocks, but no store-gateway replies with the data. It's possible that store-gateways were not yet fully initialized -- ie. they didn't load all the blocks.