feat(multi-turn): honor --random-range-ratio and bimodal prefix-cache flags - #9
feat(multi-turn): honor --random-range-ratio and bimodal prefix-cache flags#9Dao007forever wants to merge 1 commit into
Conversation
… flags Multi-turn random previously ignored --random-range-ratio, --random-cache-hit-fraction, and --random-cache-ratio: the fields were never plumbed into MultiTurnRandomConfig, so every turn was a fixed input_len and the bimodal flags were silently dropped. - range-ratio: per-turn input lengths now sampled uniformly from [len*(1-r), len*(1+r)] (turn 0 from input_len, turns 1+ from per_turn_input_len). - bimodal (per-conversation): a cache_hit_fraction of whole conversations are "warm" and prepend a shared cached base prefix to turn 0 (base[..round(L*cache_ratio)] + unique suffix). Warm slices are leading slices of the same base, so they share server-side prefix-cache hits and ride in history across later turns. Cold conversations are fully unique. --random-prefix-len is ignored when bimodal (the base is the shared prefix), matching single-turn semantics. - Bimodal/range draws use independent RNGs, so the fixed non-bimodal path is byte-for-byte unchanged (existing ignored tests still pass). - config validation: bimodal requires --dataset-name random, is mutually exclusive with --multi-turn-prefix-*, needs both cache flags > 0, both in [0, 1]. - Added test_bimodal_prefix_cache_multi_turn and test_range_ratio_varies_lengths_multi_turn; updated existing config literals; documented in AGENTS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019pRPzcLp8seCPMPTCzZNry
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a32ce32f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let (in_low, in_high) = cfg.range_ratio.input_bounds(real_input_len); | ||
| let (pt_low, pt_high) = cfg.range_ratio.input_bounds(real_per_turn_len); |
There was a problem hiding this comment.
When users pass --multi-turn --dataset-name random --random-range-ratio with a nonzero output ratio (including the bare-float form, which applies to both input and output), this path only samples input bounds; each ConversationTurn is still created with the fixed expected_output_len: output_len, and the request path enforces that fixed length via min_tokens/max_completion_tokens. This makes multi-turn random benchmarks silently use fixed output lengths while single-turn random varies them, so throughput/latency results are not for the requested workload. Please also sample per-turn output lengths from range_ratio.output_bounds.
Useful? React with 👍 / 👎.
| )); | ||
| } | ||
|
|
||
| let use_range = !cfg.range_ratio.is_fixed(); |
There was a problem hiding this comment.
With --multi-turn-prefix-global-ratio or --multi-turn-prefix-conversation-ratio, generate_multi_turn_random returns through generate_prefix_sharing_conversations before this new range-ratio block is reached, and that helper never uses cfg.range_ratio. The CLI still accepts --random-range-ratio in this mode, so prefix-sharing experiments silently remain fixed-length instead of using the requested length jitter; either sample lengths in the prefix-sharing generator or reject the combination.
Useful? React with 👍 / 👎.
1 participant
Multi-turn random previously ignored --random-range-ratio, --random-cache-hit-fraction, and --random-cache-ratio: the fields were never plumbed into MultiTurnRandomConfig, so every turn was a fixed input_len and the bimodal flags were silently dropped.
Claude-Session: https://claude.ai/code/session_019pRPzcLp8seCPMPTCzZNry