Fix: Prevent random score drops due to metadata scope bug in validator#26
Fix: Prevent random score drops due to metadata scope bug in validator#26wazzowsky wants to merge 1 commit intoFLock-io:mainfrom
Conversation
## Problem
Validators occasionally assign zero weights to high-performing miners due to a metadata scope bug in the score normalization process. This causes good datasets to randomly "fall off" the leaderboard despite maintaining their quality.
## Root Cause
The validator code has a subtle but critical bug in `neurons/validator.py`:
1. **First loop (lines 385-421)**: Evaluates miners and assigns `metadata` variable for each UID
- Can break early when `should_set_weights()` returns True
- The `metadata` variable gets overwritten each iteration
2. **Second loop (lines 470-491)**: Normalizes scores for ALL UIDs in `uids_to_eval`
- Uses `metadata.id.competition_id` for score computation
- **BUG**: This `metadata` references only the LAST evaluated UID from the first loop
### Example scenario:
- `uids_to_eval = [10, 20, 30, 40, 50]`
- First loop breaks after UID 30 (due to approaching epoch boundary)
- `metadata` now contains UID 30's data
- Second loop processes ALL UIDs [10, 20, 30, 40, 50]
- UIDs 10, 20, 40, 50 all use UID 30's metadata (wrong!)
- This causes `compute_score()` to return 0 when competition IDs don't match
## Solution
This PR implements a minimal fix:
1. Add `metadata_per_uid = {}` dictionary to store metadata for each UID
2. Store metadata for each UID: `metadata_per_uid[uid] = metadata`
3. Skip normalization for UIDs that weren't evaluated
4. Use correct metadata for each UID: `uid_metadata = metadata_per_uid[uid]`
## Changes
- Only 10 lines added to `neurons/validator.py`
- No breaking changes
- Preserves existing logic while fixing the scope issue
## Testing
To verify this fix:
1. Add debug logging to see which metadata is used for each UID
2. Monitor that unevaluated UIDs are properly skipped
3. Confirm no more zero scores for valid datasets
## Impact
This fix ensures:
- Miners with good datasets maintain consistent scores
- No more random weight drops to zero
- Fair evaluation across all validators
WalkthroughThe update introduces a Changes
Sequence Diagram(s)sequenceDiagram
participant Validator
participant MetadataStore
Validator->>MetadataStore: Retrieve metadata for UID
MetadataStore-->>Validator: Return metadata
Validator->>Validator: Store metadata in metadata_per_uid[UID]
loop For each UID in normalization
Validator->>Validator: Check if metadata_per_uid[UID] exists
alt Metadata exists
Validator->>Validator: Normalize score using UID's metadata
else Metadata missing
Validator->>Validator: Skip normalization for UID
end
end
Possibly related PRs
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🔇 Additional comments (5)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
1 participant
Problem
Validators occasionally assign zero weights to high-performing miners due to a metadata scope bug in the score normalization process. This causes good datasets to randomly "fall off" the leaderboard despite maintaining their quality.
Root Cause
The validator code has a subtle but critical bug in
neurons/validator.py:First loop (lines 385-421): Evaluates miners and assigns
metadatavariable for each UIDshould_set_weights()returns Truemetadatavariable gets overwritten each iterationSecond loop (lines 470-491): Normalizes scores for ALL UIDs in
uids_to_evalmetadata.id.competition_idfor score computationmetadatareferences only the LAST evaluated UID from the first loopExample scenario:
uids_to_eval = [10, 20, 30, 40, 50]metadatanow contains UID 30's datacompute_score()to return 0 when competition IDs don't matchSolution
This PR implements a minimal fix:
metadata_per_uid = {}dictionary to store metadata for each UIDmetadata_per_uid[uid] = metadatauid_metadata = metadata_per_uid[uid]Changes
neurons/validator.pyTesting
To verify this fix:
Impact
This fix ensures:
Summary by CodeRabbit