V2 only custom jobs #16
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains many changes:
CustomJobs: they are always expected to mount the source files that will be run on the worker. This makes the development flow much better: change a file locally, run a job, the job gets a new hash/id and runs on a worker almost immediately if a running worker is idle. This also allows people to copy the existing jobs, modify them, and keep them in a separate repo - no more docker image updates are required to change somehting in the finetuning jobsJob(singular) class has been added, allowingjob.restart(),job.cancel(),job.download(target_dir),job.runsinstead of more verbose v1 syntaxBreaking changes:
ow.deployhas moved toow.api.deploy, same for multideployow.namespace when they are imported. For example: to make finetuning available, you now need to runimport openweights.jobs.unsloth. This works because the imports run@register('fine_tuning'). More info is in the readmeYou don't need to migrate: if you don't, your jobs will use the v1 worker images. If you update the client, then new jobs will use the v2 worker images. So hopefully v1 and v2 can coexist.