Question about potential issues in DA-Code gold datasets

[LearningKeqi]

Dear Yiming and Jianwen,

Thank you for releasing DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models and the accompanying datasets. This work is very helpful and intriguing. I’m currently evaluating models using the dataset downloaded from your project’s Hugging Face page, and I may have found some inconsistencies in the gold files.

data-wrangling-007: The task description asks to standardize the “value configuration,” mapping all DOHC variations to “DOHC.” However, the gold output still contains variations such as “DOHC with VIS” and “DOHC with VGT.”

data-wrangling-001: The task specifies removing records where total_gross == 0, but in the gold file the first row’s total_gross is 0.

Besides the above two cases, I’ve noticed similar "Error Gold" cases elsewhere as well. Could you please confirm whether the gold files currently on Hugging Face are correct? If there has been an update or if corrected gold outputs are available, I’d be grateful for a pointer. If I’ve misunderstood the intended rules, any clarification would also be very helpful.

Many thanks for your time and for the excellent work.

[LearningKeqi]

PS. I also checked the 'gold data files' in the google drive, they share the exactly same questions ~

[ayanami2003]

Thanks for your interest in our work and for taking the time to review the gold outputs.

You're right — the version on Hugging Face hasn’t been updated for a while. We’ve been maintaining the latest version on Google Drive instead:
https://drive.google.com/file/d/1WxcrijbCgdHzFSSSt2HVlkJqQrBWQ2IL/view?usp=sharing

We’ll also update the Hugging Face files soon to keep them in sync.

We’ve checked the two cases you mentioned:

• data-wrangling-007: The “DOHC” normalization wasn’t applied correctly in some entries.
• data-wrangling-001: There was a record with total_gross == 0 that should have been removed.

Both issues have been fixed in the latest Google Drive version.

If you spot other tasks where the gold outputs look incorrect, we’d appreciate it if you could share a list so we can review them closely.

[LearningKeqi]

Thanks Jianwen for your prompt reply!

I just made a mistake, the "data-wrangling-007" I mentioned before should actually be "data-wrangling-003", which asks to standarize variations of DOHC to DOHC. While "data-wrangling-007" asks to normalize ‘Gear Box’ and 'Drive Type' Column.

I can see that in the latest gold data you provided, 003's 'Value configuration' column still has variations of 'DOHC' like 'DOHC with VIS' , while 007 and 001 looks good now.

Thanks again for your reply. I would be very grateful if you could also check other gold files~

[ayanami2003]

Thanks for pointing this out. We are still checking the data, and will let you know promptly if there are any major updates.