Skip to content

Conversation

@nielsrolf
Copy link
Collaborator

This PR contains many changes:

  • all jobs are what used to be CustomJobs: they are always expected to mount the source files that will be run on the worker. This makes the development flow much better: change a file locally, run a job, the job gets a new hash/id and runs on a worker almost immediately if a running worker is idle. This also allows people to copy the existing jobs, modify them, and keep them in a separate repo - no more docker image updates are required to change somehting in the finetuning jobs
  • mmlu-pro jobs are added
  • a Job (singular) class has been added, allowing job.restart(), job.cancel(), job.download(target_dir), job.runs instead of more verbose v1 syntax

Breaking changes:

  • ow.deploy has moved to ow.api.deploy, same for multideploy
  • built in jobs are only added to the ow. namespace when they are imported. For example: to make finetuning available, you now need to run import openweights.jobs.unsloth. This works because the imports run @register('fine_tuning'). More info is in the readme
  • if you run the same jobs that you used to run, they will now have different hashes

You don't need to migrate: if you don't, your jobs will use the v1 worker images. If you update the client, then new jobs will use the v2 worker images. So hopefully v1 and v2 can coexist.

@nielsrolf nielsrolf merged commit 5e88e29 into main Feb 19, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants