AIP score flakiness #280
Description
Priority Level
Medium
Task Summary
The Attribute Inference Protection score seems to be fluctuating a lot between runs, specifically for or ai_generated_essays and call_transcripts, which are predominantly text columns. The score just randomly jumps between 10/5 or 10/7.5.
https://wandb.ai/nemo-llm-service/matt_faiss_removed_reb/table?nw=nwusermkornfield
https://wandb.ai/nemo-llm-service/matt_faiss_still_there_2/table?nw=nwusermkornfield
I'm suspecting something like columns being designated different types between runs. But should confirm.
Technical Details & Implementation Plan
No response
Dependencies
No response