Repository navigation
Deduplicate concurrent pulls for identical model references across pods with node-level singleflight #35
Description
Activity
Thanks for raising this. There is already an active PR that addresses the main duplicate-pull problem: #34. It implements digest-based cross-volume pull deduplication so concurrent requests for the same model content on the same node can reuse an existing pull via hardlinks instead of downloading again.
Could you please check whether #34 covers the issue you reported here? There is also an older related issue, #24, which describes the same
volumeName/mountIDinflight-key limitation.Reacted by AngrySeanThanks for raising this. There is already an active PR that addresses the main duplicate-pull problem: #34. It implements digest-based cross-volume pull deduplication so concurrent requests for the same model content on the same node can reuse an existing pull via hardlinks instead of downloading again.
Could you please check whether #34 covers the issue you reported here? There is also an older related issue, #24, which describes the same
volumeName/mountIDinflight-key limitation.Thanks! I have checked #34, this should work, do we need to take second level concurrent control that is a global semaphore to prevent too many model pulls into consideration?
Thanks for checking. I agree that #34 covers the duplicate-pull case for the same model content.
The global semaphore is a separate concern.
pull_config.concurrencycurrently controls concurrency inside a single pull, not the number of concurrent model pulls on a node. I think we should track the node-level pull semaphore as a follow-up item, either in this issue or a separate PR.@rishi-jat, since you were assigned to the older related issue #24, would you be interested in taking this follow-up work?
Yes, I’d be interested in taking the follow-up work for node-level pull concurrency control.
From the discussion, #34 addresses deduplication for identical model content, but it does not limit the number of concurrent model pulls running on a node. A semaphore-based concurrency layer for node-level pull scheduling makes sense to avoid excessive network and disk IO under high parallel workloads.
I’ll investigate the current pull execution flow and propose an implementation approach for node-level pull concurrency limiting as a follow-up to #34.
Reacted by AngrySean and imeoerI recommend addressing this issue at the layer level rather than the image level.
We can maintain a map like
layer digest -> []pathto enable layer-level cache reuse:- Insert
layer digest -> []pathinAfterPullLayerhook. - Inspect image layers on new pulling, and try create hardlink by
layer digest -> []path. - If hardlinking fails in step 2, fall back to modctl pull.
This map key / values need also be cleaned up during volume cleanup, when daemonsets are upgraded / restarted, this map can be rebuilt on startup.
For layer-level pull singleinflight, maybe we can keep only one
backend.Newinstance and do singleinflight in modctl. cc @chlinsFor layer-level flow control of network and disk I/O, we can implement global rate limiting using
BeforePullLayer(we can get layer size in the hook func).Reacted by Rishi Jat- Insert
@imeoer @xujihui1985 - #39 PTAL. thanks!
The current pull deduplication key is based on volumeName and mountID, not on model reference.
For static inline volumes, nodePublishVolumeStaticInlineVolume calls worker.PullModel(..., volumeName, "", reference, modelDir, ...), and Worker.pullModel uses:
inflightKey := fmt.Sprintf("pull-%s/%s", volumeName, mountID)As a result, if two pods on the same node publish inline volumes with the same model reference but different volume names, they will not share the same in-flight pull.
That means on a single node:
two pods requesting the same model reference can still trigger duplicate pulls if they use different volume identities
there should be 2 level pull concurrency control, singleflight by model reference, semaphore for global download parallelism.