eval hardening: pinned revision enforcement, pair-level dedup, multi-shard sampling - #3
Open
ClarenceDan wants to merge 3 commits into
Open
eval hardening: pinned revision enforcement, pair-level dedup, multi-shard sampling#3ClarenceDan wants to merge 3 commits into
ClarenceDan wants to merge 3 commits into
Conversation
…port 1. eval_torch.py: add verify_commit_hash() using HfApi SHA check to prevent model-swap attacks between validator pin and eval server download. run_bootstrap_test() now accepts list of shard_keys with proportional sequence allocation and cross-shard shuffle. 2. eval_server.py: call verify_commit_hash() in _ensure_king and _load_challenger before model loading. EvalRequest accepts shard_keys list with backward-compat single shard_key fallback. 3. validator.py: replace evaluated_repos set with eval_counts dict keyed by (repo, commit) tuple. TEUTONIC_EVAL_MAX_RETRIES (default 1) controls max evals per commit. TEUTONIC_EVAL_SHARD_COUNT (default 1) selects K shards via blake2b with personalization salts for each shard index.
- require 40-char commit SHAs for model verification - persist eval dedup and validation failures by king/challenger commit pair - refuse insecure startup when king revision is unpinned - dedupe shard keys and reject unpinned revisions at eval server ingress - harden multi-shard sampling edge cases for zero/leftover allocation
- migrate legacy king state without king_revision to pinned HF SHAs - retry revision recovery during liveness checks instead of hard-failing startup - preserve strict pinned-revision enforcement for new evals once migrated
This was referenced May 3, 2026
roykollensvendsen
added a commit
to roykollensvendsen/teutonic
that referenced
this pull request
May 3, 2026
Spec source: signature (State(r2)) + __init__ field defaults + the R2 contract (r2.get(key) returns parsed JSON or None, r2.put(key, data) persists). State persists to keys including king/current.json, state/queue.json, state/seen_hotkeys.json, state/validator_state.json. Outcome: confirms-behavior plus one resolved spec-ambiguity. 7 tests cover empty-storage load, per-field round-trip for king/counter/ last_weight_block/seen, missing-keys tolerance, and the intentional non-persistence of failed_repos / evaluated_repos. Spec-ambiguity resolved: I assumed failed_repos and evaluated_repos would persist alongside `seen` in seen_hotkeys.json. They don't — that key is named for hotkeys (its content), and these two sets hold repo names used only for in-memory dedup during a single validator run. The test now documents this as intentional design. Test design intentionally avoids full-dict equality — assertions are per-field round-trip identity. This keeps tests resilient to upstream PR #3 (unarbos#3) which adds new State fields. Mocking: dict-backed r2_mock fixture stands in for validator.R2. get returns the stored value (or None), put writes it.
roykollensvendsen
added a commit
to roykollensvendsen/teutonic
that referenced
this pull request
May 3, 2026
Spec source: signature (State(r2)) + __init__ field defaults + the R2 contract (r2.get(key) returns parsed JSON or None, r2.put(key, data) persists). State persists to keys including king/current.json, state/queue.json, state/seen_hotkeys.json, state/validator_state.json. Outcome: confirms-behavior plus one resolved spec-ambiguity. 7 tests cover empty-storage load, per-field round-trip for king/counter/ last_weight_block/seen, missing-keys tolerance, and the intentional non-persistence of failed_repos / evaluated_repos. Spec-ambiguity resolved: I assumed failed_repos and evaluated_repos would persist alongside `seen` in seen_hotkeys.json. They don't — that key is named for hotkeys (its content), and these two sets hold repo names used only for in-memory dedup during a single validator run. The test now documents this as intentional design. Test design intentionally avoids full-dict equality — assertions are per-field round-trip identity. This keeps tests resilient to upstream PR #3 (unarbos#3) which adds new State fields. Mocking: dict-backed r2_mock fixture stands in for validator.R2. get returns the stored value (or None), put writes it.
roykollensvendsen
added a commit
to roykollensvendsen/teutonic
that referenced
this pull request
May 5, 2026
Spec source: signature (State(r2)) + __init__ field defaults + the R2 contract (r2.get(key) returns parsed JSON or None, r2.put(key, data) persists). State persists to keys including king/current.json, state/queue.json, state/seen_hotkeys.json, state/validator_state.json. Outcome: confirms-behavior plus one resolved spec-ambiguity. 7 tests cover empty-storage load, per-field round-trip for king/counter/ last_weight_block/seen, missing-keys tolerance, and the intentional non-persistence of failed_repos / evaluated_repos. Spec-ambiguity resolved: I assumed failed_repos and evaluated_repos would persist alongside `seen` in seen_hotkeys.json. They don't — that key is named for hotkeys (its content), and these two sets hold repo names used only for in-memory dedup during a single validator run. The test now documents this as intentional design. Test design intentionally avoids full-dict equality — assertions are per-field round-trip identity. This keeps tests resilient to upstream PR #3 (unarbos#3) which adds new State fields. Mocking: dict-backed r2_mock fixture stands in for validator.R2. get returns the stored value (or None), put writes it.
roykollensvendsen
added a commit
to roykollensvendsen/teutonic
that referenced
this pull request
May 6, 2026
Spec source: signature (State(r2)) + __init__ field defaults + the R2 contract (r2.get(key) returns parsed JSON or None, r2.put(key, data) persists). State persists to keys including king/current.json, state/queue.json, state/seen_hotkeys.json, state/validator_state.json. Outcome: confirms-behavior plus one resolved spec-ambiguity. 7 tests cover empty-storage load, per-field round-trip for king/counter/ last_weight_block/seen, missing-keys tolerance, and the intentional non-persistence of failed_repos / evaluated_repos. Spec-ambiguity resolved: I assumed failed_repos and evaluated_repos would persist alongside `seen` in seen_hotkeys.json. They don't — that key is named for hotkeys (its content), and these two sets hold repo names used only for in-memory dedup during a single validator run. The test now documents this as intentional design. Test design intentionally avoids full-dict equality — assertions are per-field round-trip identity. This keeps tests resilient to upstream PR #3 (unarbos#3) which adds new State fields. Mocking: dict-backed r2_mock fixture stands in for validator.R2. get returns the stored value (or None), put writes it.
roykollensvendsen
added a commit
to roykollensvendsen/teutonic
that referenced
this pull request
May 8, 2026
Spec source: signature (State(r2)) + __init__ field defaults + the R2 contract (r2.get(key) returns parsed JSON or None, r2.put(key, data) persists). State persists to keys including king/current.json, state/queue.json, state/seen_hotkeys.json, state/validator_state.json. Outcome: confirms-behavior plus one resolved spec-ambiguity. 7 tests cover empty-storage load, per-field round-trip for king/counter/ last_weight_block/seen, missing-keys tolerance, and the intentional non-persistence of failed_repos / evaluated_repos. Spec-ambiguity resolved: I assumed failed_repos and evaluated_repos would persist alongside `seen` in seen_hotkeys.json. They don't — that key is named for hotkeys (its content), and these two sets hold repo names used only for in-memory dedup during a single validator run. The test now documents this as intentional design. Test design intentionally avoids full-dict equality — assertions are per-field round-trip identity. This keeps tests resilient to upstream PR #3 (unarbos#3) which adds new State fields. Mocking: dict-backed r2_mock fixture stands in for validator.R2. get returns the stored value (or None), put writes it.
roykollensvendsen
added a commit
to roykollensvendsen/teutonic
that referenced
this pull request
May 8, 2026
Spec source: signature (State(r2)) + __init__ field defaults + the R2 contract (r2.get(key) returns parsed JSON or None, r2.put(key, data) persists). State persists to keys including king/current.json, state/queue.json, state/seen_hotkeys.json, state/validator_state.json. Outcome: confirms-behavior plus one resolved spec-ambiguity. 7 tests cover empty-storage load, per-field round-trip for king/counter/ last_weight_block/seen, missing-keys tolerance, and the intentional non-persistence of failed_repos / evaluated_repos. Spec-ambiguity resolved: I assumed failed_repos and evaluated_repos would persist alongside `seen` in seen_hotkeys.json. They don't — that key is named for hotkeys (its content), and these two sets hold repo names used only for in-memory dedup during a single validator run. The test now documents this as intentional design. Test design intentionally avoids full-dict equality — assertions are per-field round-trip identity. This keeps tests resilient to upstream PR #3 (unarbos#3) which adds new State fields. Mocking: dict-backed r2_mock fixture stands in for validator.R2. get returns the stored value (or None), put writes it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several important improvements to evaluation robustness, security, and flexibility for the model evaluation server and validator. Key changes include strict commit hash verification to prevent model-swap attacks, support for evaluating across multiple data shards, and more reliable state tracking for evaluation attempts and failures.
Security and Robustness Improvements:
Strict commit hash verification: Added
verify_commit_hash(and related logic) to ensure that only pinned, 40-character commit SHAs are accepted for model revisions, preventing model-swap attacks by verifying the actual commit on HuggingFace matches the expected SHA. This is enforced in both the eval server and validator. (eval_torch.py,eval_server.py,validator.py) [1] [2] [3] [4] [5] [6] [7]Input validation: The eval server now rejects evaluation requests that do not provide a valid commit SHA for both king and challenger revisions, or that lack at least one shard key. (
eval_server.py) [1] [2] [3]Evaluation Flexibility and Data Handling:
run_bootstrap_testand the API accept a list of shard keys, distribute the evaluation set proportionally across available shards, and shuffle sequences across shards to ensure fair sampling. (eval_torch.py,eval_server.py) [1] [2] [3] [4]Validator State and Retry Handling:
validator.py) [1] [2] [3]Other Improvements:
eval_server.py,validator.py) [1] [2] [3]These changes significantly improve the security, reliability, and flexibility of the evaluation infrastructure.