streaming chunk from store gateway failing

[kadhamecha-conga]

hi team,

I have enable below config to reduce querier memory usage , but on large range query is failing with below error:

execution: attempted to read series at index 16757 from stream, but the stream has failed: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: read tcp x.x.x.x:52180->x.x..x.x:9095: read: connection reset by peer", received prior goaway: code: NO_ERROR, debug data: "max_age"

---

below two values i have added as querier config:
prefer_streaming_chunks_from_store_gateways: true
streaming_chunks_per_store_gateway_series_buffer_size: 512

can you please help me to undersatnd where the issue is ?
what should be the value set for streaming_chunks_per_store_gateway_series_buffer_size , if query range is higher ?

thanks.

[charleskorn]

That error message makes it sound as if the store-gateway the querier was reading from has crashed. Could you please check for any issues in the store-gateway logs around the same time?

[kadhamecha-conga]

hi,

there is no such error, but just below warnings in same sg logs:

caller=memcached_client.go:335 level=warn name=chunks-cache msg="failed to fetch items from memcached" numKeys=45 firstKey=subrange:anonymous/01HDMT2JB20N7P7CHPA88D6KNN/chunks/000002:85744000:85760000 err="memcache: connect timeout to 10.7.27.130:11211"

i'm running with below caching component:
chunk: 4
index: 2
metadata: 1
result: 2

i'm trying to use stream to reduce the querier memory usage and start facing this issue.
resources usage is under threshold.

let me know, if you need more info.

thanks.

[charleskorn]

The error message you posted contains a string like error reading from server: read tcp x.x.x.x:52180->y.y.y.y:9095: read: connection reset by peer. What component does the second address (y.y.y.y:9095) refer to? I'd check if that is crashing or experiencing any network-related issues.

[charleskorn]

current values of streaming_chunks_per_store_gateway_series_buffer_size 10000.

This is very high. The default value of 256 should be fine.

[kadhamecha-conga]

if i remove streaming option for sg query is failing with consistency-check-failed
and if i added stream large query failed with
execution: attempted to read series at index 16757 from stream, but the stream has failed

[charleskorn]

Could you please share your entire configuration file, with any sensitive values removed?

[kadhamecha-conga]

PFA config.

`

activity_tracker:
  filepath: /active-query-tracker/activity.log
alertmanager:
  data_dir: /data
  enable_api: true
  external_url: /alertmanager
  fallback_config_file: /configs/alertmanager_fallback_config.yaml
blocks_storage:
  backend: s3
  bucket_store:
    chunks_cache:
      backend: memcached
      memcached:
        addresses: dns+mimir-chunks-cache.dev.svc:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 10485760
        timeout: 10s
    index_cache:
      backend: memcached
      memcached:
        addresses: dns+mimir-index-cache.dev.svc:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 10485760
        timeout: 10s
    index_header_lazy_loading_enabled: true
    index_header_lazy_loading_idle_timeout: 1h
    max_chunk_pool_bytes: 12884901888
    metadata_cache:
      backend: memcached
      memcached:
        addresses: dns+mimir-metadata-cache.dev.svc:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 10485760
        timeout: 10s
    sync_dir: /data/tsdb-sync
  s3:
    bucket_name: XXXXXX.
  tsdb:
    dir: /data/tsdb
    flush_blocks_on_shutdown: false
    head_compaction_interval: 15m
    wal_replay_concurrency: 3
common:
  storage:
    backend: s3
    s3:
      endpoint: s3.dualstack.xxx.com
      region: us-east-2
compactor:
  compaction_interval: 30m
  data_dir: /data
  deletion_delay: 2h
  first_level_compaction_wait_period: 25m
  max_closing_blocks_concurrency: 2
  max_opening_blocks_concurrency: 4
  sharding_ring:
    wait_stability_min_duration: 1m
  symbols_flushers_concurrency: 4
frontend:
  align_queries_with_step: true
  cache_results: true
  log_queries_longer_than: 5s
  parallelize_shardable_queries: true
  query_sharding_target_series_per_shard: 2500
  query_stats_enabled: true
  results_cache:
    backend: memcached
    memcached:
      addresses: dns+mimir-results-cache.dev.svc:11211
      max_async_buffer_size: 10000000
      max_async_concurrency: 100
      max_get_multi_batch_size: 1000
      max_get_multi_concurrency: 750
      max_idle_connections: 750
      max_item_size: 10485760
      timeout: 5s
  scheduler_address: mimir-query-scheduler-headless.dev.svc:9095
  split_queries_by_interval: 24h0m0s
frontend_worker:
  grpc_client_config:
    max_send_msg_size: 419430400
  scheduler_address: mimir-query-scheduler-headless.dev.svc:9095
ingester:
  ring:
    final_sleep: 0s
    num_tokens: 512
    replication_factor: 1
    tokens_file_path: /data/tokens
    unregister_on_shutdown: false
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
limits:
  cardinality_analysis_enabled: true
  compactor_blocks_retention_period: 30d
  ingestion_burst_size: 1000000
  ingestion_rate: 200000
  max_cache_freshness: 10m
  max_fetched_chunks_per_query: 10000000
  max_global_exemplars_per_user: 100000
  max_global_series_per_metric: 0
  max_global_series_per_user: 10000000
  max_label_names_per_series: 50
  max_query_parallelism: 960
  max_total_query_length: 30d
  query_sharding_max_sharded_queries: 0
memberlist:
  abort_if_cluster_join_fails: false
  compression_enabled: false
  join_members:
  - dns+mimir-gossip-ring.dev.svc.cluster.local:7946
querier:
  max_concurrent: 48
  timeout: 5m
    --># prefer_streaming_chunks_from_ingesters: true
    --># streaming_chunks_per_ingester_series_buffer_size: 10000
    --># prefer_streaming_chunks_from_store_gateways: true
    --># streaming_chunks_per_store_gateway_series_buffer_size: 10000
query_scheduler:
  max_outstanding_requests_per_tenant: 800
ruler:
  alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.mimir-alertmanager-headless.dev.svc.cluster.local/alertmanager
  enable_api: true
  rule_path: /data
ruler_storage:
  cache:
    backend: memcached
    memcached:
      addresses: dns+mimir-metadata-cache.dev.svc:11211
      max_item_size: 10485760
runtime_config:
  file: /var/mimir/runtime.yaml
server:
  grpc_server_max_concurrent_streams: 1000
  grpc_server_max_connection_age: 2m
  grpc_server_max_connection_age_grace: 5m
  grpc_server_max_connection_idle: 1m
  grpc_server_max_recv_msg_size: 504857600
  grpc_server_max_send_msg_size: 504857600
  http_server_read_timeout: 5m0s
  http_server_write_timeout: 5m0s
store_gateway:
  sharding_ring:
    replication_factor: 2
    tokens_file_path: /data/tokens
    unregister_on_shutdown: false
    wait_stability_min_duration: 1m
usage_stats:
  installation_mode: helm
`

[charleskorn]

if i remove streaming option for sg query is failing with consistency-check-failed
and if i added stream large query failed with
execution: attempted to read series at index 16757 from stream, but the stream has failed

Could you please include the full error messages you see here? Both should include more information.

[kadhamecha-conga]

i'm running on 2.10.0.

[jnavarrof]

Hi @kadhamecha-conga. Hitting exactly the same error. Did you find the issue in your setup? Thanks