feat(entity-store-perf): parallel bulk ingest, ingest-rate, duration mode, and entity poll fixes

[gurevichdmitry]

Summary

Improves upload-perf-data-interval (and single upload-perf-data) for sustained load testing: higher log ingest throughput, optional rate limiting, duration-based runs, and reliable post-upload entity polling.

Changes

• Faster uploads (~800 → ~13k docs/sec observed): Read JSONL in 5k-doc batches and send parallel _bulk requests (pMap, default 8 concurrency, pipeline=_none, refresh=false) instead of a single helpers.bulk stream.
• --bulk-concurrency: CLI flag to tune parallel bulks (default 8; values above 8 did not improve throughput on our test cluster).
• --ingest-rate: Optional max docs/sec per upload (throttle pauses between batches). Logs achieved rate and notes when the cluster, not the flag, is the bottleneck.
• --duration: Wall-clock interval mode (mutually exclusive with --count); keeps uploading until the deadline with --interval pauses between passes.
• Docs: Updated entity_store_perf/README.md for new flags and behavior.

yarn start upload-perf-data-interval medium --deleteData --noTransforms --interval 2 --duration 5m --ingest-rate 25000 --bulk-concurrency 8

[Copilot]

Pull request overview

This PR enhances the entity_store_perf upload commands to better support sustained load/performance testing by increasing ingest throughput, adding optional throttling and duration-based execution, and making post-upload entity polling more reliable (with timeouts and clearer progress).

Changes:

• Add configurable parallel bulk ingest (--bulk-concurrency) and optional per-upload ingest throttling (--ingest-rate), plus achieved ingest-rate logging.
• Add duration-based interval runs (--duration, mutually exclusive with --count) and refactor the interval loop into a shared helper.
• Improve post-upload entity polling with timeout support, match_all counting when --deleteData is used, and progress/status logs; update docs accordingly.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
src/commands/shared/elasticsearch.ts	Adds optional pipeline to bulkUpsert, extends streamingBulkIngest with defaults + concurrency/pipeline support.
src/commands/entity_store_perf/index.ts	Adds CLI flags for bulk concurrency, ingest rate, and duration; updates argument handling/validation.
src/commands/entity_store_perf/entity_store_perf.ts	Implements new upload batching + parallel bulk indexing, ingest-rate throttling, duration mode, and improved entity polling logic.
src/commands/entity_store_perf/README.md	Documents new flags (--bulk-concurrency, --ingest-rate, --duration) and updated interval/polling behavior.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.