Repository navigation
possible race condition #24
Description
Activity
Thanks for identifying this issue, it can indeed occur when concurrently pulling the same model on the same node.
Our initial design uses
volumeName/mountIDas the inflightKey is to prevent frequent pull requests from same PVC (in controller mode) or pod (in inline/dynamic mode).If we improve it by using
referenceas the inflightKey, two cases need consideration:-
With dragonfly hardlink: upgrade singleinflight to a key-based mutex so
puller.Pullruns sequentially, allowing each volumeName/mountID to create its own model directory. -
Without dragonfly hardlink: hard-link files from the previous model directory to the new one.
This adds a bit of complexity to the logic, but it's still achievable to handle scenario where the same node might concurrently pull the same model.
-
Hey @aftersnow , I would like to work on this issue can you please assign this issue to me ? Thanks!
@rishi-jat Cool! assigned. :)
Hi @rishi-jat, are you still working on this issue?
There is now an active PR for the same duplicate-pull problem: #34. It implements digest-based cross-volume pull deduplication. Please let us know if you still plan to continue with this issue, or whether we should use #34 as the current implementation path.
@imeoer @aftersnow , I have open the PR to fix the issue PTAL when you get a chance. thanks
Now inflightKey is volumeName/mountID, suppose:
(1) Pod A request Model X, inflightKey = "pull-volumeA/mountA"
(2) Pod B request Model X, inflightKey = "pull-volumeB/mountB"
which means same model X can be dowloaded concurrently, causes possible race condition.