Repository navigation
Conversation
|
Claude finished @jcastro-dotcms's task in 3m 16s —— View job Code ReviewI reviewed the diff against The PR is careful and consistent: everything is gated behind Existing
Resolved
Checks performed: flag-off behavior is preserved (every change is behind an · branch |
|
Pull Request Unsafe to Rollback!!!
|
9690354 to
5ffcec2
Compare
2da5728 to
4701296
Compare
4701296 to
fc1d9d5
Compare
5ffcec2 to
00cb9e1
Compare
|
Pull Request Unsafe to Rollback!!!
|
00cb9e1 to
a7e7291
Compare
fc1d9d5 to
a0e9eab
Compare
|
test |
|
(Disregard the previous comment — it was posted in error while verifying tooling access.) |
|
Pull Request Unsafe to Rollback!!!
Note: this is narrow in practice — |
a0e9eab to
2a60b57
Compare
a7e7291 to
187a09b
Compare
2a60b57 to
6964d89
Compare
187a09b to
9422974
Compare
6964d89 to
788f4d0
Compare
9422974 to
0b1ed27
Compare
dotCMS-Machine-User
left a comment
There was a problem hiding this comment.
✅ dotbot review: all reviewer models (meta/muse-spark-1.3, ~z-ai/glm-latest) agree — patch is correct.
approved automatically by dotbot
…ough S3 Sixth slice of the S3 asset storage work. With FEATURE_FLAG_S3_ASSET_STORAGE on, completed temporary uploads and their custom metadata are stored in S3 with an access and expiry receipt, WebDAV temporary files publish payloads and path records with conditional writes, and BinaryCleanupJob expires both. Mixed-case temporary names are kept, and flag-off WebDAV and temporary uploads match main.
…from the storage provider
… WebDAV listing cleanup robust These fixes apply only with FEATURE_FLAG_S3_ASSET_STORAGE on; flag-off behavior is unchanged. WebDAV COPY of a CMS file no longer passes the caller's auto-publish flag on the S3 path. The copy is saved as an unpublished working version, as on main, so it needs only edit permission. The stream closing that the S3 path added is kept. Focal points that the image filter stores under a temp_<content inode> id stay in the local temporary directory, where BinaryCleanupJob removes them as before. Those ids never get an upload receipt, so S3 records written for them were never cleaned up. The unreachable flag check in the legacy metadata loop is removed. A WebDAV folder listing builds each temporary file resource from the entry the listing already read, instead of looking it up again, so a temporary file deleted during the listing no longer fails it. If S3 cannot be read, the failure is logged and only the temporary children are left out, so the CMS files and folders still list. WebdavTemporaryStorage.children reads only the records that decide each direct child: the child's own record, and descendants only for a folder with no live record of its own. Records below a live folder are no longer fetched on every listing. Expired temporary uploads have their S3 objects removed at the TTL, but their local copies are left to the existing CLEANUP_TMP_FILES_OLDER_THAN_HOURS age rule, so a check-in that resolved the file just before expiry keeps its source. Temporary-upload cleanup handles each upload separately. A failure is logged, that upload keeps its receipt and payload for the next run, the rest are still cleaned, and one combined failure is thrown so the job still reports it. Adds WebdavTemporaryStorageTest (in-memory remote) for the listing fixes, and unit tests for the focal-point routing and the per-upload cleanup. DotWebdavHelperTest now expects an unpublished copy and no longer reads the s3.cms.enabled system property. The storage doc describes the new behavior, the remaining listing cost next to the tombstone note, and the accepted metadata race.
0b1ed27 to
e7d3e70
Compare
788f4d0 to
4f3af4b
Compare
| public File createTempFile(String path) throws IOException{ | ||
| File file = new File(getTempDir().getPath() + path); | ||
| File file = new File(getTempDir().getPath() + temporaryPath(path)); | ||
| if (com.dotcms.storage.AssetStorageFeature.isEnabled()) return file; |
There was a problem hiding this comment.
Reject new WebDAV temp uploads whose name lacks a temp-resource component in path validation
Current code:
public File createTempFile(String path) throws IOException{
File file = new File(getTempDir().getPath() + temporaryPath(path));
if (com.dotcms.storage.AssetStorageFeature.isEnabled()) return file;Problem: With the flag on, createTempFile no longer creates parent directories, and callers that pass plain file paths rely on writeCompletedTempFile (which does Files.createDirectories). Verify all callers pass paths whose parents get created, or store() fails with NoSuchFileException. See FileMetadataAPIImpl and TempFileAPI for the analogous directory-creation ordering concerns.
| final Entry entry = read(key); | ||
| if (entry != null && !entry.expired()) return entry; | ||
| // Parents can be implicit, including after a concurrent child write and folder deletion. | ||
| final var children = children(file); |
There was a problem hiding this comment.
WebdavTemporaryStorage.stat(folder) throws DotRuntimeException on a transient S3 read error
Current code:
} catch (IOException | DotDataException failure) {
throw new DotRuntimeException("Unable to inspect WebDAV staging file", failure);
}Problem: TempFolderResourceImpl.getModifiedDate() calls stat(folder), so a transient S3 outage now causes an unchecked exception instead of the previous graceful null/folder-time fallback. A read-only PROPFIND on a WebDAV client fails hard. Consider catching DotRuntimeException in getModifiedDate() or swallowing transient errors in stat.
| * @param dotDavHelper the WebDAV helper | ||
| * @param hostAPI the site API | ||
| * @return the resource, or {@code null} if the URL names nothing | ||
| */ |
There was a problem hiding this comment.
case-insensitive temp-resource lookup may miss entries created with case-mapped path
Current code:
if (!com.dotcms.storage.AssetStorageFeature.isEnabled() || !dotDavHelper.isTempResource(url)) url = url.toLowerCase();Problem: The temporaryPath helper in DotWebdavHelper lowercases only components before the first temp-resource component (e.g. (.DS_Store) or .AppleDouble style entries). But WebdavTemporaryStorage.key() records the temp component with original case, while loadTempFile builds the key with temporaryPath which stops lowercasing at the temp boundary — good. Verify recordPath round-trip matches for entries whose CMS prefix components differ in case between write and read (e.g. .../HOST/ vs .../host/); the CMS prefix is lowercased on read but recorded as-created on write from a different code path (BasicFolderResourceImpl.createNew uses stripMapping(originalPath) without lowercasing). If cases diverge, stat(tempFile) returns null and the resource appears missing.
| throw new DotRuntimeException("Invalid file upload"); | ||
| } | ||
| createTempPermissionFile(tempFolder, allowList); | ||
| if (AssetStorageFeature.isEnabled()) { |
There was a problem hiding this comment.
mark temp file managed only after completeTempFile succeeds, not during create
Current code:
if (AssetStorageFeature.isEnabled()) {
try {
Files.writeString(java.nio.file.Path.of(com.dotmarketing.util.ConfigUtils.getAssetTempPath(),
tempFileId, TemporaryAssetStorage.MANAGED_MARKER), "");Problem: The .s3-upload marker is written at creation, before any receipt exists. If the upload never completes (client abort, or store() fails on an S3 outage), the local bytes are fenced off: getTempFile returns empty because isManagedLocally is true but receipt() is absent, and there is no legacy fallback. Until BinaryCleanupJob ages the file out, a valid local copy is unreadable even though the flag-off path would serve it. Consider writing the marker only after store() verifies the S3 copy (store() already writes it), or removing the marker in store()'s failure path.
|
... |
|
Pull Request Unsafe to Rollback!!!
|
|
dotbot code review:
Prior case-sensitivity note is speculative verification without a proven divergent write/read path in the current patch, so it does not meet flag-worthy evidence standards. No new P0/P1 defects proven in this review run. Tip: comment with "/dotbot address comments" to attempt automated fixes for unresolved review threads. reviewed by dotbot · meta/muse-spark-1.3 · medium |
| } | ||
| createTempPermissionFile(tempFolder, allowList); | ||
| if (AssetStorageFeature.isEnabled()) { | ||
| try { |
There was a problem hiding this comment.
🟡 [P2] TempFileAPI.java:139 managed marker written before upload completes, fencing servable local bytes
Current code:
if (AssetStorageFeature.isEnabled()) {
try {
Files.writeString(java.nio.file.Path.of(com.dotmarketing.util.ConfigUtils.getAssetTempPath(),
tempFileId, TemporaryAssetStorage.MANAGED_MARKER), "");Problem: Marker is written at create time, before any receipt exists. If the upload aborts or store() fails on an S3 outage, getTempFile sees isManagedLocally true with no receipt and returns empty with no legacy fallback, fencing a valid local copy until cleanup ages it out.
Fix:
if (AssetStorageFeature.isEnabled()) {
// marker is written by TemporaryAssetStorage.store() once the S3 copy is verified
}(store() already writes owner.resolve(MANAGED_MARKER); writing it only there, or deleting it in store()'s failure path, closes the window.)
| // Parents can be implicit, including after a concurrent child write and folder deletion. | ||
| final var children = children(file); | ||
| return children.isEmpty() ? null : new Entry(key, true, 0, | ||
| children.stream().mapToLong(Entry::modified).max().orElseThrow(), ""); |
There was a problem hiding this comment.
🟡 [P2] WebdavTemporaryStorage.java:150 stat() throws unchecked on transient S3 error in PROPFIND path
Current code:
} catch (IOException | DotDataException failure) {
throw new DotRuntimeException("Unable to inspect WebDAV staging file", failure);
}Problem: TempFolderResourceImpl.getModifiedDate() (TempFolderResourceImpl.java:173) calls stat(folder) unguarded, so a transient S3 read error now fails a read-only PROPFIND that previously never threw.
Fix:
} catch (IOException | DotDataException failure) {
throw new DotRuntimeException("Unable to inspect WebDAV staging file", failure);
}(Either catch DotRuntimeException in getModifiedDate() and return null, or degrade stat() errors to a logged null for this listing-only caller. This is consistent with the deliberate "fail loud on S3 error" choice elsewhere, so confirm intent before changing.)
|
dotbot code review:
The change is consistently gated behind the default-off FEATURE_FLAG_S3_ASSET_STORAGE, preserving legacy behavior when the flag is off. S3 paths are validated against traversal/symlinks, writes use conditional reservations with rollback, and cleanup is per-item with retry semantics. The prior case-sensitivity concern is resolved: both the write path (BasicFolderResourceImpl.createNew -> createTempFile -> temporaryPath) and the read path (loadTempFile -> temporaryPath) normalize the CMS prefix identically, so write and read keys match. The remaining points are opt-in-path resilience trade-offs, not blocking correctness bugs. Tip: comment with "/dotbot address comments" to attempt automated fixes for unresolved review threads. reviewed by dotbot · ~z-ai/glm-latest · medium |
Refs #37868
Proposed Changes
TempFileAPI, image editor output) are stored in thetemporary-assetsgroup with an immutable receipt recording who may use the upload and when it expires. Access and expiry are checked before any bytes download, so another node can serve the upload. A payload is published only after its stream closes. Custom metadata on a temporary upload, such as a focal point, is stored with it in S3. Focal points the image filter writes for a content image (atemp_id with no receipt) stay on the local path thatBinaryCleanupJobalready ages out.webdav-temporarygroup with conditional writes, so a stale writer cannot publish after a deletion; cleanup tombstones expired records. A folder listing reads only the records of its direct children and builds each resource from the record it read, so a file deleted during the listing is skipped, and an S3 failure leaves out only the temporary children instead of failing the CMS listing.BinaryCleanupJobalso expires S3 temporary uploads and WebDAV files. At expiry it removes the S3 objects and leaves the local directory to the existingCLEANUP_TMP_FILES_OLDER_THAN_HOURSrule, and it handles each upload separately so one bad receipt cannot stop the pass.FocalPointAPITestnow completes its temporary uploads the way the upload API and image editor do; with the flag on, an unfinished upload is intentionally not a usable temp resource.DotWebdavHelperTestretries its eviction asserts briefly, because eviction deliberately declines while a background reindex holds a cache lease.DotWebdavHelperTestis now registered inMainSuite2a. It was never in a suite, so CI never ran it.MainSuite2awas the fastest suite in six recent PR runs (about 22 to 35 minutes against 37 to 47 for the others), following the registration guide indocs/testing/INTEGRATION_TESTS.md. Its header comment says to avoid adding tests to it; that comment looks out of date against the timings, so please say if you would rather it went elsewhere.Behavior with the flag off
Unchanged from main; WebDAV and temporary uploads use the local filesystem as before.
Review fixes
The last commit on this branch (
fix(storage): keep WebDAV copies unpublished and ...) addresses a full review of this PR. All of it is flag-on only:WebdavTemporaryStorageTestcovers the listing changes against an in-memory remote that models ETags, so they run in CI.Checklist
WebdavTemporaryStorageTest. Integration on this branch before the review fixes, flag off: 124 run, 0 failures, including the existingTempFileAPITest,TempFileResourceTest,BinaryCleanupJobTest,BinaryExporterServletTestandDotWebdavHelperTest; flag on: 85 run, 0 failures. After the review fixes,DotWebdavHelperTest,TempFileAPITest,BinaryCleanupJobTest,BinaryExporterServletTestandFocalPointAPITestpassed flag off and on in the run on the top of the stack (feat(storage): serve renditions, compiled CSS and templates through S3 asset storage #37776).