Add cold-shape cross-run ablation v2#125
Open
Darkroom4364 wants to merge 1 commit into
Open
Conversation
📝 WalkthroughWalkthroughThis PR implements a second-generation offline cold-shape cross-run learning ablation (v2) to fairly test whether a persistent database compounds learnings across runs. It introduces a new replay pipeline that partitions data into held-out buckets, trains ranking models, evaluates three selector conditions across multiple seeds and iterations, and reports findings showing database seeding does not independently improve over stateless baselines, while cost-model ranking achieves near-optimal performance. ChangesCold-Shape Cross-Run Learning Ablation v2
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/cold_shape_cross_run_ablation_v2.py`:
- Around line 187-191: _split_record_key currently rejects 2-part legacy keys
and returns None; update it to accept keys split into 2 parts by ":" and return
operator="matmul" with bucket and hardware (i.e., for len(parts)==2 return
("matmul", parts[0], parts[1])) while keeping the existing len==3 behavior and
returning None otherwise so legacy `bucket:hardware` rows are included in
heldout selection, training DB construction, kNN/seed scoring, and replay.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 0e3fcd01-b9de-4dae-9702-f0f2e7cadafe
📒 Files selected for processing (8)
docs/paper/evaluation.mddocs/paper/noeris-submission.texdocs/paper/noeris.mddocs/results/README.mddocs/results/cold-shape-cross-run-ablation-v2.jsondocs/results/cold-shape-cross-run-ablation-v2.mdscripts/cold_shape_cross_run_ablation_v2.pytests/test_cold_shape_cross_run_ablation_v2.py
Comment on lines
+187
to
+191
| def _split_record_key(key: str) -> tuple[str, str, str] | None: | ||
| parts = key.split(":", 2) | ||
| if len(parts) != 3: | ||
| return None | ||
| return parts[0], parts[1], parts[2] |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
python - <<'PY'
import json
from pathlib import Path
path = Path(".noeris/cost-model-training.json")
data = json.loads(path.read_text(encoding="utf-8"))
legacy = [key for key in data.get("records", {}) if len(key.split(":", 2)) == 2]
print(f"legacy_two_part_keys={len(legacy)}")
for key in legacy[:20]:
print(key)
PYRepository: 0sec-labs/noeris
Length of output: 83
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show key parsing function + usages
rg -n "def _split_record_key|_split_record_key\(" scripts/cold_shape_cross_run_ablation_v2.py
# Show bucket record collection + training db construction around the lines mentioned
python - <<'PY'
from pathlib import Path
p = Path("scripts/cold_shape_cross_run_ablation_v2.py")
lines = p.read_text(encoding="utf-8").splitlines()
for start,end in [(160,210),(210,260),(300,360),(360,430)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
# Inspect cost_model parsing contract
python - <<'PY'
from pathlib import Path
p = Path("src/research_engine/cost_model.py")
lines = p.read_text(encoding="utf-8").splitlines()
# Print around 240-320 and search for legacy handling
for start,end in [(230,290),(290,360),(360,440)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
# Find any code that constructs legacy two-part keys
rg -n "\"matmul\"|operator.*matmul|bucket:hardware|split\\(\\\":\\\",\\s*2\\)" -S src/research_engine/cost_model.py scripts/cold_shape_cross_run_ablation_v2.pyRepository: 0sec-labs/noeris
Length of output: 18166
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show key parsing function + usages
rg -n "def _split_record_key|_split_record_key\(" scripts/cold_shape_cross_run_ablation_v2.py
# Show bucket record collection + training db construction around the lines mentioned
python - <<'PY'
from pathlib import Path
p = Path("scripts/cold_shape_cross_run_ablation_v2.py")
lines = p.read_text(encoding="utf-8").splitlines()
for start,end in [(160,210),(210,260),(300,360),(360,430)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
# Inspect cost_model parsing contract
python - <<'PY'
from pathlib import Path
p = Path("src/research_engine/cost_model.py")
lines = p.read_text(encoding="utf-8").splitlines()
# Print around 240-440 and search for legacy handling
for start,end in [(240,320),(320,390),(390,470)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
# Find any code that constructs legacy two-part keys
rg -n "operator.*matmul|CostModel\\.train_from_databases|split\\(\\\":\\\",\\s*2\\)|bucket:hardware" -S src/research_engine/cost_model.py scripts/cold_shape_cross_run_ablation_v2.pyRepository: 0sec-labs/noeris
Length of output: 18037
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "def _split_record_key|_split_record_key\(" scripts/cold_shape_cross_run_ablation_v2.py
python - <<'PY'
from pathlib import Path
p = Path("scripts/cold_shape_cross_run_ablation_v2.py")
lines = p.read_text(encoding="utf-8").splitlines()
for start,end in [(170,205),(205,250),(290,335),(335,390)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
python - <<'PY'
from pathlib import Path
p = Path("src/research_engine/cost_model.py")
lines = p.read_text(encoding="utf-8").splitlines()
for start,end in [(240,330),(330,420),(420,510)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
rg -n "operator.*matmul|train_from_databases|split\\(\\\":\\\",\\s*2\\)" -S src/research_engine/cost_model.py scripts/cold_shape_cross_run_ablation_v2.pyRepository: 0sec-labs/noeris
Length of output: 15796
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "def _split_record_key|_split_record_key\(" scripts/cold_shape_cross_run_ablation_v2.py
python - <<'PY'
from pathlib import Path
p = Path("scripts/cold_shape_cross_run_ablation_v2.py")
lines = p.read_text(encoding="utf-8").splitlines()
for start,end in [(170,205),(205,250),(290,335),(335,390)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
python - <<'PY'
from pathlib import Path
p = Path("src/research_engine/cost_model.py")
lines = p.read_text(encoding="utf-8").splitlines()
for start,end in [(240,330),(330,420),(420,510)]:
print("\n# ---",start,"-",end,"---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
rg -n "operator.*matmul|train_from_databases|split\\(\\\":\\\",\\s*2\\)" -S src/research_engine/cost_model.py scripts/cold_shape_cross_run_ablation_v2.pyRepository: 0sec-labs/noeris
Length of output: 15796
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "raw_records|records\s*=|\.noeris|json\.load|json\.loads|heldout_keys|bucket:hardware" scripts/cold_shape_cross_run_ablation_v2.py
python - <<'PY'
from pathlib import Path
p = Path("scripts/cold_shape_cross_run_ablation_v2.py")
lines = p.read_text(encoding="utf-8").splitlines()
for start,end in [(1,140),(140,240),(240,340),(340,440),(440,560),(560,760)]:
print(f"\n# --- {start}-{end} ---")
for i in range(start-1, min(end, len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PYRepository: 0sec-labs/noeris
Length of output: 31633
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "cost-model-training\.json" -S . || true
rg -n "cold_shape_cross_run_ablation_v2|build_report\\(" -S scripts src || true
# find any scripts that generate ".noeris/cost-model-training.json"
fd -e py -e sh -e js -e ts .noeris -S . || true
fd -e py .noeris -S . || true
# Find where ".noeris" directory is populated
rg -n "\\.noeris" -S scripts src README.md .github || trueRepository: 0sec-labs/noeris
Length of output: 8682
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "cost-model-training\.json" -S . || true
rg -n "cost[-_ ]model[-_ ]training" -S . || true
rg -n "\\.noeris" -S scripts src README.md .github || true | head -n 200Repository: 0sec-labs/noeris
Length of output: 12458
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect collect_cost_model_data.py around how record keys are written
rg -n "bucket:hardware|operator:bucket:hardware|split\\(\":\",\\s*2\\)|\"matmul\"|records\\[" scripts/collect_cost_model_data.py scripts/train_cost_model_v2.py scripts/train_cost_model_v2.py tests/test_cold_shape_cross_run_ablation_v2.py src/research_engine/cost_model.py || true
python - <<'PY'
from pathlib import Path
paths = [
Path("scripts/collect_cost_model_data.py"),
Path("scripts/train_cost_model_v2.py"),
Path("tests/test_cold_shape_cross_run_ablation_v2.py"),
]
for p in paths:
if not p.exists():
continue
print("\n====", p, "====")
txt = p.read_text(encoding="utf-8").splitlines()
# print relevant regions around "records" or "key"
for i,line in enumerate(txt, start=1):
if "records" in line and ("output" in line or "json" in line or "=" in line):
pass
# just print first 180 lines for those small files
max_lines = 220
for i in range(1, min(max_lines, len(txt))+1):
print(f"{i:4d}: {txt[i-1]}")
PYRepository: 0sec-labs/noeris
Length of output: 19684
🏁 Script executed:
#!/bin/bash
set -euo pipefail
targets=(
"scripts/collect_cost_model_data.py"
"scripts/train_cost_model_v2.py"
"tests/test_cold_shape_cross_run_ablation_v2.py"
)
for f in "${targets[@]}"; do
echo "== $f =="
rg -n "bucket:hardware|operator:bucket:hardware|records|split\\(\"\":\"\",\\s*2\\)|matmul|cost-model-training\\.json|_split_record_key" "$f" || true
done
python - <<'PY'
from pathlib import Path
import re
def show_section(path, pattern, context=5, max_lines=120):
p=Path(path)
lines=p.read_text(encoding="utf-8").splitlines()
for i,line in enumerate(lines, start=1):
if re.search(pattern,line):
start=max(1,i-context)
end=min(len(lines), i+context)
print(f"\n--- {path}:{start}-{end} (match at {i}) ---")
for j in range(start,end+1):
print(f"{j:4d}: {lines[j-1]}")
if end-start>max_lines:
break
show_section("scripts/collect_cost_model_data.py", r"records", context=8)
show_section("scripts/collect_cost_model_data.py", r"matmul", context=12)
show_section("scripts/train_cost_model_v2.py", r"records", context=8)
show_section("tests/test_cold_shape_cross_run_ablation_v2.py", r"cold_shape_cross_run_ablation_v2|split|legacy|bucket", context=15)
PYRepository: 0sec-labs/noeris
Length of output: 17234
🏁 Script executed:
#!/bin/bash
set -euo pipefail
targets=(
"scripts/collect_cost_model_data.py"
"scripts/train_cost_model_v2.py"
"tests/test_cold_shape_cross_run_ablation_v2.py"
)
for f in "${targets[@]}"; do
echo "== $f =="
rg -n "bucket:hardware|operator:bucket:hardware|records|split\\(\\\":\\\",\\s*2\\)|matmul|cost-model-training\\.json|_split_record_key" "$f" || true
done
python - <<'PY'
from pathlib import Path
import re
def show_section(path, pattern, context=6, max_groups=6):
p=Path(path)
lines=p.read_text(encoding="utf-8").splitlines()
count=0
for i,line in enumerate(lines, start=1):
if re.search(pattern,line):
start=max(1,i-context)
end=min(len(lines), i+context)
print(f"\n--- {path}:{start}-{end} (match at {i}) ---")
for j in range(start,end+1):
print(f"{j:4d}: {lines[j-1]}")
count+=1
if count>=max_groups:
return
show_section("scripts/collect_cost_model_data.py", r"records", context=8)
show_section("scripts/collect_cost_model_data.py", r"matmul", context=12)
show_section("scripts/train_cost_model_v2.py", r"records", context=10)
show_section("tests/test_cold_shape_cross_run_ablation_v2.py", r"legacy|bucket|split|_split_record_key|cost-model-training\\.json", context=15)
PYRepository: 0sec-labs/noeris
Length of output: 15037
🏁 Script executed:
#!/bin/bash
set -euo pipefail
targets=(
"scripts/collect_cost_model_data.py"
"scripts/train_cost_model_v2.py"
"tests/test_cold_shape_cross_run_ablation_v2.py"
)
for f in "${targets[@]}"; do
echo "== $f =="
rg -n "bucket:hardware|operator:bucket:hardware|records|split\\(\\\":\\\",\\s*2\\)|matmul|cost-model-training\\.json|_split_record_key" "$f" || true
done
python - <<'PY'
from pathlib import Path
import re
def show_section(path, pattern, context=6, max_groups=6):
p=Path(path)
lines=p.read_text(encoding="utf-8").splitlines()
count=0
for i,line in enumerate(lines, start=1):
if re.search(pattern,line):
start=max(1,i-context)
end=min(len(lines), i+context)
print(f"\n--- {path}:{start}-{end} (match at {i}) ---")
for j in range(start,end+1):
print(f"{j:4d}: {lines[j-1]}")
count+=1
if count>=max_groups:
return
show_section("scripts/collect_cost_model_data.py", r"records", context=8)
show_section("scripts/collect_cost_model_data.py", r"matmul", context=12)
show_section("scripts/train_cost_model_v2.py", r"records", context=10)
show_section("tests/test_cold_shape_cross_run_ablation_v2.py", r"legacy|bucket|split|_split_record_key|cost-model-training\\.json", context=15)
PYRepository: 0sec-labs/noeris
Length of output: 15037
🏁 Script executed:
python - <<'PY'
import json
from pathlib import Path
p = Path(".noeris/cost-model-training.json")
data = json.loads(p.read_text(encoding="utf-8"))
keys = list(data.get("records", {}).keys())
legacy = [k for k in keys if len(k.split(":", 2)) == 2]
print("legacy_two_part_keys=", len(legacy))
PYRepository: 0sec-labs/noeris
Length of output: 84
🏁 Script executed:
python - <<'PY'
import json
from pathlib import Path
data = json.loads(Path(".noeris/cost-model-training.json").read_text(encoding="utf-8"))
legacy = [k for k in data.get("records", {}).keys() if len(k.split(":", 2)) == 2]
print("legacy_two_part_keys=", len(legacy))
PYRepository: 0sec-labs/noeris
Length of output: 84
🏁 Script executed:
python - <<'PY'
import json
from pathlib import Path
data = json.loads(Path(".noeris/cost-model-training.json").read_text(encoding="utf-8"))
legacy = [k for k in data.get("records", {}).keys() if len(k.split(":", 2)) == 2]
print("legacy_two_part_keys=", len(legacy))
PYRepository: 0sec-labs/noeris
Length of output: 84
Add backward-compatible parsing for legacy bucket:hardware record keys in _split_record_key
src/research_engine/cost_model.py::CostModel.train_from_databases() supports legacy 2-part record keys (bucket:hardware, treated as operator="matmul"), but scripts/cold_shape_cross_run_ablation_v2.py::_split_record_key() only accepts operator:bucket:hardware and returns None for 2-part keys—so those rows would be skipped during heldout selection, training DB construction, kNN/seed scoring, and replay for older databases. The checked-in .noeris/cost-model-training.json currently has legacy_two_part_keys=0, so the default artifact/tests aren’t affected.
Suggested fix
def _split_record_key(key: str) -> tuple[str, str, str] | None:
parts = key.split(":", 2)
- if len(parts) != 3:
- return None
- return parts[0], parts[1], parts[2]
+ if len(parts) == 3:
+ return parts[0], parts[1], parts[2]
+ if len(parts) == 2:
+ bucket, hardware = parts
+ return "matmul", bucket, hardware
+ return None🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/cold_shape_cross_run_ablation_v2.py` around lines 187 - 191,
_split_record_key currently rejects 2-part legacy keys and returns None; update
it to accept keys split into 2 parts by ":" and return operator="matmul" with
bucket and hardware (i.e., for len(parts)==2 return ("matmul", parts[0],
parts[1])) while keeping the existing len==3 behavior and returning None
otherwise so legacy `bucket:hardware` rows are included in heldout selection,
training DB construction, kNN/seed scoring, and replay.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.noeris/cost-model-training.jsonrows.stateless_random,database_seeded, andcost_model_rankingacross 3 seeds and 6 iterations.Result
cost_model_ranking: 0.9971 mean final normalized throughput, 0.29% mean final regret, 100% 90%-best success.database_seeded: 0.9604 mean final normalized throughput, 3.96% mean final regret, 75% 90%-best success.Verification
python3 scripts/cold_shape_cross_run_ablation_v2.pypython3 -m unittest tests.test_cold_shape_cross_run_ablation_v2 tests.test_public_claim_artifactspython3 -m compileall scripts/cold_shape_cross_run_ablation_v2.py tests/test_cold_shape_cross_run_ablation_v2.pypython3 scripts/check_public_claim_artifacts.pygit diff --checkNote:
ruffis not installed in this environment, soruff check ...could not run.Closes #112
Summary by CodeRabbit
Documentation
New Features
Tests