Time-box runs, add rolling top-up windows, and skip provably empty checks - #8
Open
ryanwjackson wants to merge 3 commits into
Open
ryanwjackson wants to merge 3 commits into
ryanwjackson wants to merge 3 commits into
Conversation
…ecks
Three changes to how a copy execution spends its window:
- The job's timeoutMinutes now reaches the container as
COPY_TIMEOUT_MINUTES, and rclone gets --max-duration with a margin
(a quarter of the timeout, at most 15 minutes) and --cutoff-mode soft,
so in-flight uploads finish instead of being killed mid-chunk by the
platform's replicaTimeout. rclone's exit code 10 (window closed with
work remaining) is recorded as transfer_window_exhausted and reported
as success: the schedule is the resume mechanism, and a non-zero exit
would trigger an immediate unscheduled retry via replicaRetryLimit.
- A new optional source.topUpMaxAge job field ("48h") selects only
recently modified files and adds --no-traverse, so a frequent schedule
stops re-listing the whole destination tree once an initial full copy
exists. It cannot be combined with includePaths or modifiedOnOrAfter,
and the fixed modifiedOnOrAfter cutoff deliberately does not get
--no-traverse, which is a net loss on wide selections.
- Before a whole-tree copy, one listing probes whether the destination
is empty; if it provably is, --no-check-dest --retries 1 skips
per-file existence checks for that run. Any content or any probe
error selects the normal comparing path.
Every run now also sends Microsoft's documented ISV user-agent so
SharePoint throttles the copy less aggressively.
Older deployments lack the new variables: pull treats them as optional
and apply backfills them. infra/main.json regenerated (bicep 0.46.1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtQWZLnq1WDQ79k8Bn5xbo
Reject zero and leading-zero top-up magnitudes to match copyctl.py, require COPY_TIMEOUT_MINUTES to be at least 5 so --max-duration can never compute to rclone's disabled value, and make the top-up test exercise the probe-skip guard by faking an empty destination. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018jcdrAtd4YsuMCsDWjuFSA
The shell rejects leading zeros in SOURCE_TOP_UP_MAX_AGE, so copyctl now does too instead of deploying a value every run would then refuse. Job files written before topUpMaxAge existed keep validating, the same way the template tolerates old parameter files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCsfMHP5zKkxTiQf9rFpyn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large trees make every run pay a full destination re-listing, and the platform timeout can kill rclone mid-upload. This changes how each execution spends its window.
What changed
Time-boxing —
timeoutMinutesnow reaches the container asCOPY_TIMEOUT_MINUTES, and rclone runs with--max-duration(timeout minus a margin: a quarter of the timeout, capped at 15 minutes) and--cutoff-mode soft, so transfers in flight finish cleanly instead of being killed byreplicaTimeout. rclone exit code 10 — the window closed with work remaining — is logged astransfer_window_exhaustedand reported as success: the schedule is the resume mechanism, and a non-zero exit would fire an immediate unscheduled retry throughreplicaRetryLimit: 1and still mark the execution failed.Rolling top-up window — new optional
source.topUpMaxAgefield (e.g."48h"): copy only files modified within the window, with--no-traverseso the destination tree is never listed. On a converged large tree this turns a full-walk run into a source-walk-plus-candidate-checks run. Mutually exclusive withincludePathsandmodifiedOnOrAfter; the fixed cutoff deliberately does not get--no-traverse, which backfires on wide selections. Size the window to at least twice the schedule interval and keep a periodic job without it as the full reconcile pass (see theexample.jsonccomment).Empty-destination fast path — one
lsfprobe before a whole-tree copy; if the destination is provably empty,--no-check-dest --retries 1skips per-file existence checks for that run. Any listed content or any probe error selects the normal comparing path, so partially populated destinations are never at risk of duplicates.Throttling — every run sends Microsoft's documented ISV user-agent (
ISV|rclone.org|rclone/<version>) so SharePoint throttles the service less aggressively.Compatibility
pulltreats the new fields as optional;applybackfills them.topUpMaxAgeexisted keep deploying (safe access in the template); the portal form emits the field as empty.infra/main.jsonregenerated with bicep 0.46.1 (generator stamps in the diff are from the version bump).Testing
copyctl.py validate,tests/config_test.py,tests/uidefinition_test.py,tests/template_test.py,tests/transfer_test.sh, and shellcheck all pass locally; bicep builds clean. Not covered: a real Azure execution — the probe, time-box, and exit-10 paths are exercised against the fake rclone only.🤖 Generated with Claude Code
https://claude.ai/code/session_01QtQWZLnq1WDQ79k8Bn5xbo