Skip to content

Fix Hugging Face dataset paths and worker startup diagnostics#235

Draft
TianhuaTao wants to merge 2 commits into
mainfrom
tianhua/dev
Draft

Fix Hugging Face dataset paths and worker startup diagnostics#235
TianhuaTao wants to merge 2 commits into
mainfrom
tianhua/dev

Conversation

@TianhuaTao

Copy link
Copy Markdown

Summary

  • update task DataSource paths to fully qualified Hugging Face dataset repo IDs and add a fallback for silent_tqdm
  • give trust_remote_code workers isolated Hugging Face modules caches to avoid concurrent dynamic-code cache writes
  • add optional PyTorch worker thread limits and more detailed worker/vLLM startup and cleanup logging

Testing

  • Not run; PR opened from the existing remote branch without a local checkout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant