feat(auth): replace static hotkey/API-key auth with Bittensor validator whitelisting and 50% consensus (#5)

* feat(auth): replace static hotkey/API-key auth with Bittensor validator whitelisting and 50% consensus

Integrate dynamic validator whitelisting from Bittensor netuid 100 and
consensus-based evaluation triggering, replacing the previous single
AUTHORIZED_HOTKEY + WORKER_API_KEY authentication system.

Authentication now uses a dynamic whitelist of validators fetched every
5 minutes from the Bittensor blockchain via bittensor-rs. Validators
must have validator_permit, be active, and have >=10,000 TAO stake.
POST /submit requests only trigger evaluations when >=50% of whitelisted
validators submit the same archive payload (identified by SHA-256 hash).

New modules:
- src/validator_whitelist.rs: ValidatorWhitelist with parking_lot::RwLock,
  background refresh loop with 3-retry exponential backoff, connection
  resilience (keeps cached whitelist on failure), starts empty and rejects
  requests with 503 until first successful sync
- src/consensus.rs: ConsensusManager using DashMap for lock-free vote
  tracking, PendingConsensus entries with TTL (default 60s), reaper loop
  every 30s, max 100 pending entries cap, duplicate vote detection

Modified modules:
- src/auth.rs: Removed AUTHORIZED_HOTKEY import, api_key field from
  AuthHeaders, X-Api-Key header extraction, InvalidApiKey error variant.
  verify_request() now takes &ValidatorWhitelist instead of API key string.
  Updated all tests accordingly.
- src/config.rs: Removed AUTHORIZED_HOTKEY constant and worker_api_key
  field. Added bittensor_netuid, min_validator_stake_tao,
  validator_refresh_secs, consensus_threshold, consensus_ttl_secs with
  env var support and sensible defaults. Updated banner output.
- src/handlers.rs: Added ValidatorWhitelist and ConsensusManager to
  AppState. submit_batch now: checks whitelist non-empty (503), validates
  against whitelist, computes SHA-256 of archive, records consensus vote,
  returns 202 with pending status or triggers evaluation on consensus.
  Moved active batch check to consensus-reached branch only.
- src/main.rs: Added module declarations, creates ValidatorWhitelist and
  ConsensusManager, spawns background refresh and reaper tasks.
- Cargo.toml: Added bittensor-rs git dependency and mandatory
  [patch.crates-io] for w3f-bls.
- Dockerfile: Added protobuf-compiler, cmake, clang, mold build deps
  for bittensor-rs substrate dependencies. Copies .cargo config.
- AGENTS.md and src/AGENTS.md: Updated data flow, module map, env vars,
  authentication docs to reflect new architecture.

BREAKING CHANGE: WORKER_API_KEY env var and X-Api-Key header no longer required.
All validators on Bittensor netuid 100 with sufficient stake are auto-whitelisted.

* ci: trigger CI run

* fix(security): address auth bypass, input validation, and config issues

- Move nonce consumption AFTER signature verification in verify_request()
  to prevent attackers from burning legitimate nonces via invalid signatures
- Fix TOCTOU race in NonceStore::check_and_insert() using atomic DashMap
  entry API instead of separate contains_key + insert
- Add input length limits for auth headers (hotkey 128B, nonce 256B,
  signature 256B) to prevent memory exhaustion via oversized values
- Add consensus_threshold validation in Config::from_env() — must be
  in range (0.0, 1.0], panics at startup if invalid
- Add saturating conversion for consensus required calculation to prevent
  integer overflow on f64→usize cast
- Add tests for all security fixes

* fix(dead-code): remove orphaned default_concurrent fn and unnecessary allow(dead_code)

* fix: code quality issues in bittensor validator consensus

- Extract magic number 100 to configurable MAX_PENDING_CONSENSUS
- Restore #[allow(dead_code)] on DEFAULT_MAX_OUTPUT_BYTES constant
- Use anyhow::Context instead of map_err(anyhow::anyhow!) in validator_whitelist

* fix(security): address race condition, config panic, SS58 checksum, and container security

- consensus.rs: Fix TOCTOU race condition in record_vote by using
  DashMap entry API (remove_entry) to atomically check votes and remove
  entry while holding the shard lock, preventing concurrent threads from
  inserting votes between drop and remove
- config.rs: Replace assert! with proper Result<Self, String> return
  from Config::from_env() to avoid panicking in production on invalid
  CONSENSUS_THRESHOLD values
- main.rs: Update Config::from_env() call to handle Result with expect
- auth.rs: Add SS58 checksum verification using Blake2b-512 (correct
  Substrate algorithm) in ss58_to_public_key_bytes to reject addresses
  with corrupted checksums; previously only decoded base58 without
  validating the 2-byte checksum suffix
- Dockerfile: Add non-root executor user for container runtime security

* fix(dead-code): remove unused max_output_bytes config field and constant

Remove DEFAULT_MAX_OUTPUT_BYTES constant and max_output_bytes Config field
that were defined and populated from env but never read anywhere outside
config.rs. Both had #[allow(dead_code)] annotations suppressing warnings.

* fix(quality): replace expect/unwrap with proper error handling, extract magic numbers to constants

- main.rs: Replace .expect() on Config::from_env() with match + tracing::error! + process::exit(1)
- validator_whitelist.rs: Extract retry count (3) and backoff base (2) to named constants
- validator_whitelist.rs: Replace unwrap_or_else on Option with if-let pattern
- consensus.rs: Extract reaper interval (30s) to REAPER_INTERVAL_SECS constant

* fix(security): address multiple security vulnerabilities in PR files

- consensus.rs: Remove archive_data storage from PendingConsensus to
  prevent memory exhaustion (up to 50GB with 100 pending × 500MB each).
  Callers now use their own archive bytes since all votes for the same
  hash have identical data.

- handlers.rs: Stream multipart upload with per-chunk size enforcement
  instead of buffering entire archive before checking size limit.
  Sanitize error messages to not leak internal details (file paths,
  extraction errors) to clients; log details server-side instead.

- auth.rs: Add nonce format validation requiring non-empty printable
  ASCII characters (defense-in-depth against log injection and empty
  nonce edge cases).

- main.rs: Replace .unwrap() on TcpListener::bind and axum::serve with
  proper error logging and process::exit per AGENTS.md rules.

- ws.rs: Replace .unwrap() on serde_json::to_string with
  unwrap_or_default() to comply with AGENTS.md no-unwrap rule.

* fix(dead-code): rename misleading underscore-prefixed variable in consensus

* fix(quality): replace unwrap/expect with proper error handling in production code

- main.rs:21: Replace .parse().unwrap() on tracing directive with
  unwrap_or_else fallback to INFO level directive
- main.rs:36: Replace .expect() on workspace dir creation with
  error log + process::exit(1) pattern
- main.rs:110: Replace .expect() on ctrl_c handler with if-let-Err
  that logs and returns gracefully
- executor.rs:189: Replace semaphore.acquire().unwrap() with match
  that handles closed semaphore by creating a failed TaskResult

All changes follow AGENTS.md rule: no .unwrap()/.expect() in
production code paths. Test code is unchanged.

* docs: refresh AGENTS.md

Author: echobt

AGENTS.md

@@ -11,29 +11,51 @@ This is a **single-crate Rust binary** (`term-executor`) built with Axum. There
 ### Data Flow
 
 ```
-Client → POST /submit (multipart archive) → term-executor
-  1. Authenticate via X-Hotkey, X-Nonce, X-Signature, X-Api-Key headers
-  2. Extract uploaded archive (zip/tar.gz) containing tasks/ and agent_code/
-  3. Parse each task: workspace.yaml, prompt.md, tests/
-  4. For each task (concurrently, up to limit):
-     a. git clone the target repository at base_commit
-     b. Run install commands (pip install, etc.)
-     c. Write & execute agent code in the repo
-     d. Write test source files into the repo
-     e. Run test scripts (bash), collect exit codes
-  5. Aggregate results (reward per task, aggregate reward)
-  6. Stream progress via WebSocket (GET /ws?batch_id=...)
-  7. Return results via GET /batch/{id}
+Validator → POST /submit (multipart archive) → term-executor
+  1. Authenticate via X-Hotkey, X-Nonce, X-Signature headers
+  2. Verify hotkey is in the dynamic validator whitelist (Bittensor netuid 100, >10k TAO stake)
+  3. Compute SHA-256 hash of archive bytes
+  4. Record vote in ConsensusManager
+  5. If <50% of whitelisted validators have voted for this hash:
+     → Return 202 Accepted with pending_consensus status
+  6. If ≥50% consensus reached:
+     a. Extract uploaded archive (zip/tar.gz) containing tasks/ and agent_code/
+     b. Parse each task: workspace.yaml, prompt.md, tests/
+     c. For each task (concurrently, up to limit):
+        i.   git clone the target repository at base_commit
+        ii.  Run install commands (pip install, etc.)
+        iii. Write & execute agent code in the repo
+        iv.  Write test source files into the repo
+        v.   Run test scripts (bash), collect exit codes
+     d. Aggregate results (reward per task, aggregate reward)
+     e. Stream progress via WebSocket (GET /ws?batch_id=...)
+     f. Return results via GET /batch/{id}
+```
+
+### Background Tasks
+
+```
+ValidatorWhitelist refresh loop (every 5 minutes):
+  1. Connect to Bittensor subtensor via BittensorClient::with_failover()
+  2. Sync metagraph for netuid 100
+  3. Filter validators: validator_permit && active && stake >= 10,000 TAO
+  4. Atomically replace whitelist with new set of SS58 hotkeys
+  5. On failure: retry up to 3 times with exponential backoff, keep cached whitelist
+
+ConsensusManager reaper loop (every 30 seconds):
+  1. Remove pending consensus entries older than TTL (default 60s)
 ```
 
 ### Module Map
 
 | File | Responsibility |
 |---|---|
-| `src/main.rs` | Entry point — bootstraps config, session manager, executor, Axum server, reaper tasks |
-| `src/config.rs` | `Config` struct loaded from environment variables with defaults; `AUTHORIZED_HOTKEY` constant |
+| `src/main.rs` | Entry point — bootstraps config, session manager, executor, validator whitelist, consensus manager, Axum server, background tasks |
+| `src/config.rs` | `Config` struct loaded from environment variables with defaults; Bittensor and consensus configuration |
 | `src/handlers.rs` | Axum route handlers: `/health`, `/status`, `/metrics`, `/submit`, `/batch/{id}`, `/batch/{id}/tasks`, `/batch/{id}/task/{task_id}`, `/batches` |
-| `src/auth.rs` | Authentication: `extract_auth_headers()`, `verify_request()`, `validate_ss58()`, sr25519 signature verification via `verify_sr25519_signature()`, `NonceStore` for replay protection, `AuthHeaders`/`AuthError` types |
+| `src/auth.rs` | Authentication: `extract_auth_headers()`, `verify_request()` (whitelist-based), `validate_ss58()`, sr25519 signature verification via `verify_sr25519_signature()`, SS58 checksum via `blake2`, `NonceStore` for replay protection, `AuthHeaders`/`AuthError` types |
+| `src/validator_whitelist.rs` | Dynamic validator whitelist — fetches validators from Bittensor netuid 100 every 5 minutes, filters by stake ≥10k TAO, stores SS58 hotkeys in `parking_lot::RwLock<HashSet>` |
+| `src/consensus.rs` | 50% consensus manager — tracks pending votes per archive hash in `DashMap`, triggers evaluation when ≥50% of whitelisted validators submit same payload, TTL reaper for expired entries |
 | `src/executor.rs` | Core evaluation engine — spawns batch tasks that clone repos, run agents, run tests concurrently |
 | `src/session.rs` | `SessionManager` with `DashMap`, `Batch`, `BatchResult`, `TaskResult`, `BatchStatus`, `TaskStatus`, `WsEvent` types |
 | `src/task.rs` | Archive extraction (zip/tar.gz), task directory parsing, agent code loading, language detection |
@@ -43,8 +65,10 @@ Client → POST /submit (multipart archive) → term-executor
 
 ### Key Shared State (via `Arc`)
 
-- `AppState` (in `handlers.rs`) holds `Config`, `SessionManager`, `Metrics`, `Executor`, `NonceStore`, `started_at`
+- `AppState` (in `handlers.rs`) holds `Config`, `SessionManager`, `Metrics`, `Executor`, `NonceStore`, `started_at`, `ValidatorWhitelist`, `ConsensusManager`
 - `SessionManager` uses `DashMap<String, Arc<Batch>>` for lock-free concurrent access
+- `ValidatorWhitelist` uses `parking_lot::RwLock<HashSet<String>>` for concurrent read access with rare writes
+- `ConsensusManager` uses `DashMap<String, PendingConsensus>` for lock-free concurrent vote tracking
 - Per-batch `Semaphore` in `executor.rs` controls concurrent tasks within a batch (configurable, default: 8)
 - `broadcast::Sender<WsEvent>` per batch for WebSocket event streaming
 
@@ -59,7 +83,8 @@ Client → POST /submit (multipart archive) → term-executor
 - **Archive Handling**: `flate2` + `tar` (tar.gz), `zip` 2 (zip)
 - **Error Handling**: `anyhow` 1 + `thiserror` 2
 - **Logging**: `tracing` + `tracing-subscriber` with env-filter
-- **Crypto/Identity**: `sha2`, `hex`, `base64`, `bs58` (SS58 address validation), `schnorrkel` 0.11 (sr25519 signature verification), `rand_core` 0.6, `uuid` v4
+- **Crypto/Identity**: `sha2`, `hex`, `base64`, `bs58` (SS58 address validation), `schnorrkel` 0.11 (sr25519 signature verification), `blake2` 0.10 (SS58 checksum), `rand_core` 0.6, `uuid` v4
+- **Blockchain**: `bittensor-rs` (git dependency) for Bittensor validator whitelisting via subtensor RPC
 - **Time**: `chrono` with serde support
 - **Build Tooling**: `mold` linker via `.cargo/config.toml`, `clang` as linker driver
 - **Container**: Multi-stage Dockerfile — `rust:1.93-slim-bookworm` builder → `debian:bookworm-slim` runtime (includes python3, pip, venv, build-essential, git, curl)
@@ -71,13 +96,13 @@ Client → POST /submit (multipart archive) → term-executor
 
 2. **All clippy warnings are errors.** Run `cargo +nightly clippy --all-targets -- -D warnings` locally. CI runs the same command and will fail on any warning.
 
-3. **Never expose secrets in logs or responses.** The `AUTHORIZED_HOTKEY` in `src/config.rs` is the only authorized SS58 hotkey. Auth failures log only the rejection, never the submitted hotkey value. Follow this pattern for any new secrets.
+3. **Never expose secrets in logs or responses.** Auth failures log only the rejection, never the submitted hotkey value. Follow this pattern for any new secrets.
 
 4. **All process execution MUST have timeouts.** Every call to `run_cmd`/`run_shell` in `src/executor.rs` takes a `Duration` timeout. Never spawn a child process without a timeout — agent code is untrusted and may hang forever.
 
 5. **Output MUST be truncated.** The `truncate_output()` function in `src/executor.rs` caps output at `MAX_OUTPUT` (1MB). Any new command output capture must use this function to prevent memory exhaustion from malicious agent output.
 
-6. **Shared state must use `Arc` + lock-free structures.** `SessionManager` uses `DashMap` (not `Mutex<HashMap>`). Metrics use `AtomicU64`. New shared state should follow these patterns — never use `std::sync::Mutex` for hot-path data.
+6. **Shared state must use `Arc` + lock-free structures.** `SessionManager` uses `DashMap` (not `Mutex<HashMap>`). Metrics use `AtomicU64`. `ValidatorWhitelist` uses `parking_lot::RwLock`. `ConsensusManager` uses `DashMap`. New shared state should follow these patterns — never use `std::sync::Mutex` for hot-path data.
 
 7. **Semaphore must gate task concurrency.** The per-batch `Semaphore` in `executor.rs` limits concurrent tasks within a batch. The `SessionManager::has_active_batch()` check prevents multiple batches from running simultaneously.
 
@@ -162,10 +187,14 @@ Both hooks are activated via `git config core.hooksPath .githooks`.
 | `AGENT_TIMEOUT_SECS` | `600` | Agent execution timeout |
 | `TEST_TIMEOUT_SECS` | `300` | Test suite timeout |
 | `MAX_ARCHIVE_BYTES` | `524288000` | Max uploaded archive size (500MB) |
-| `MAX_OUTPUT_BYTES` | `1048576` | Max captured output per command (1MB) |
 | `WORKSPACE_BASE` | `/tmp/sessions` | Base directory for session workspaces |
-| `WORKER_API_KEY` | *(required)* | API key that whitelisted hotkeys must provide via `X-Api-Key` header |
+| `BITTENSOR_NETUID` | `100` | Bittensor subnet ID for validator lookup |
+| `MIN_VALIDATOR_STAKE_TAO` | `10000` | Minimum TAO stake for validator whitelisting |
+| `VALIDATOR_REFRESH_SECS` | `300` | Interval for refreshing validator whitelist (seconds) |
+| `CONSENSUS_THRESHOLD` | `0.5` | Fraction of validators required for consensus (0.0–1.0) |
+| `CONSENSUS_TTL_SECS` | `60` | TTL for pending consensus entries (seconds) |
+| `MAX_PENDING_CONSENSUS` | `100` | Maximum number of pending consensus entries |
 
 ## Authentication
 
-Authentication requires four HTTP headers: `X-Hotkey` (SS58 address), `X-Nonce` (unique per-request), `X-Signature` (sr25519 hex signature of `hotkey + nonce`), and `X-Api-Key`. The authorized hotkey is hardcoded as `AUTHORIZED_HOTKEY` in `src/config.rs`. The API key is configured via the `WORKER_API_KEY` environment variable (required). Verification steps: hotkey must match `AUTHORIZED_HOTKEY`, API key must match, SS58 format must be valid, nonce must not have been seen before (replay protection via `NonceStore` in `src/auth.rs` with 5-minute TTL), and the sr25519 signature must verify against the hotkey's public key using the Substrate signing context. Only requests passing all checks can submit batches via `POST /submit`. All other endpoints are open.
+Authentication requires three HTTP headers: `X-Hotkey` (SS58 address), `X-Nonce` (unique per-request), and `X-Signature` (sr25519 hex signature of `hotkey + nonce`). The authorized hotkeys are dynamically loaded from the Bittensor blockchain — all validators on netuid 100 with ≥10,000 TAO stake and an active validator permit are whitelisted. The whitelist refreshes every 5 minutes. Verification steps (in order): hotkey must be in the validator whitelist, SS58 format must be valid, sr25519 signature must verify against the hotkey's public key using the Substrate signing context, and finally the nonce must not have been seen before (replay protection via `NonceStore` in `src/auth.rs` with 5-minute TTL — nonce is only consumed after signature passes). Only requests passing all checks can submit batches via `POST /submit`. Evaluations are only triggered when ≥50% of whitelisted validators have submitted the same archive payload (identified by SHA-256 hash). All other endpoints are open.