feat(storage): add feature-gated S3 storage layer for shared asset storage

docs/README.md

@@ -77,6 +77,7 @@ Descriptions say **when to load** the doc, not what it contains.
 | [E2E_TESTS.md](testing/E2E_TESTS.md) | Working on the Playwright suite in `core-web/apps/dotcms-ui-e2e` |
 | [API_TESTING.md](testing/API_TESTING.md) | Writing API tests — Postman (`dotcms-postman/`) and Karate (`test-karate/`) |
 | [PERFORMANCE_TESTS.md](testing/PERFORMANCE_TESTS.md) | Load testing with JMeter (`test-jmeter/`) or the Kubernetes analytics suite |
+| [BINARY_S3_STORAGE.md](testing/BINARY_S3_STORAGE.md) | Working on S3 asset storage (`FEATURE_FLAG_S3_ASSET_STORAGE`): what changes with the flag on and off, and how to run its MinIO and integration checks |
 | [test-cases/README.md](test-cases/README.md) | Looking for the manual/feature test-suite index |
 
 ## CLI (`tools/dotcms-cli`)

docs/testing/BINARY_S3_STORAGE.md

@@ -0,0 +1,129 @@
+# S3 asset storage
+
+S3 asset storage lets dotCMS keep binary assets and their metadata durably in S3, with the
+local asset directory acting as a cache. It is opt-in and off by default. With the flag off,
+the filesystem/NFS behavior is unchanged.
+
+This page describes what is in the code today. It grows as the remaining parts land.
+
+## Behavior with the flag off
+
+With the flag off, the storage chain, its providers, metadata and static push publishing run
+the same code paths as before. Two details are worth knowing:
+
+- This feature packages the AWS STS module (`aws-java-sdk-sts`). Before, the web identity step
+  of the SDK's default credential chain always failed for lack of that module, and the chain
+  moved on to the profile files and then the container or EC2 instance role. With the flag off,
+  every place where dotCMS falls back to the default chain (static push publishing, endpoint
+  validation and the S3 metadata provider) uses `NoWebIdentityCredentialsProviderChain`, which
+  is that chain without the web identity step, so a pod with a web identity token keeps using
+  the identity it used before. Two differences remain because the module is on the classpath:
+  a profile in the AWS config files that assumes a role (`role_arn`) now resolves instead of
+  failing over to the instance role, and an OSGi plugin that builds its own
+  `DefaultAWSCredentialsProviderChain` now gets the web identity step.
+- The flag is logged at INFO only when it is on. With it off, the first read logs at debug.
+
+## Feature flag
+
+Set `FEATURE_FLAG_S3_ASSET_STORAGE=true` (Docker: `DOT_FEATURE_FLAG_S3_ASSET_STORAGE=true`)
+in the environment or `dotmarketing-config.properties`, then restart every node. The default
+is `false`.
+
+The flag is read once per process (`AssetStorageFeature.isEnabled()`) and then kept, because
+several storage objects capture the mode when they are built. A value set only in the system
+table may not be seen at first read, and a runtime change is ignored until restart. In-memory
+overrides through `Config.setProperty` do re-read it, which is how tests switch modes. Tests
+that mock `Config` statically must call `AssetStorageFeature.reset()` themselves.
+
+## Storage layer behavior with the flag on
+
+The storage chain (`ChainableStoragePersistenceAPI`) and its providers change as follows:
+
+- Writes publish to every durable provider before the new local copy becomes visible, and a
+  failed remote write keeps the previous local contents.
+- Reads restore a missing local copy from S3. A miss is not cached as a 404, because another
+  node can upload the same key at any time. Object reads restore under the same per-key lock
+  as writes and deletes, so a restore cannot overwrite a newer write on the same chain.
+- A zero-length or truncated local metadata file is a read failure, not a miss. When S3 holds a
+  readable copy, the chain replaces the local file with it under the per-key lock. Otherwise the
+  read fails and the local file is kept, so the metadata is never treated as absent and
+  regenerated, which would drop custom attributes such as a focal point.
+- Deletes keep the local copy until all durable providers accept the deletion, so an
+  in-flight restore on the same chain cannot bring the object back.
+- Storage and database query failures propagate instead of being reported as a missing
+  object.
+- Providers can list objects under a prefix, verify that a durable copy has the same
+  contents (`hasDurableCopy`), and copy an existing file or object to durable storage without
+  overwriting a conflicting one (`backfillFile`, `backfillObject`).
+- S3 existence checks (`existsGroup`, `existsObject`, `hasDurableCopy`) list at most one key
+  per lookup. Only listings that need every key (`listObjectPaths`, `listObjectSnapshots`,
+  `deleteGroup` and static push) follow every page.
+
+Two building blocks are included for the slices that follow. `S3ContentAddressedStorage`
+stores one immutable copy of each set of bytes at
+`asset-blobs/sha256/<chars-1-2>/<chars-3-4>/<chars-5-6>/<chars-7-8>/<sha256>`.
+`SharedExtractedMetadata` caches byte-derived Tika extraction at
+`extracted-metadata/<source-sha256>/<configuration-hash>.json`. Nothing calls either one yet.
+
+## Credentials and region
+
+With the flag on, leave both `storage.file-metadata.s3.access-key` and
+`storage.file-metadata.s3.secret-access-key` unset to use the AWS SDK default credential
+provider chain. The configured bucket region and endpoint are kept in that mode. A partial
+key pair fails initialization. The AWS STS module is packaged so role-based providers,
+including web identity, work with the flag on.
+
+Without a custom endpoint, a configured region selects the regional AWS endpoint. With no
+region, the client uses the global endpoint and the SDK looks up the bucket's region on first
+use. Requests are signed with SigV4, the SDK's default S3 signer. A custom endpoint
+(`storage.file-metadata.s3.endpoint`) must be an absolute HTTP(S) URL and requires
+`storage.file-metadata.s3.bucket-region` for request signing. Invalid endpoint configuration
+fails instead of silently selecting AWS. With the flag off, configuration behaves as before.
+
+## Sharing one bucket across installations
+
+With the flag on, `storage.file-metadata.s3.namespace` (Docker:
+`DOT_STORAGE_FILE_METADATA_S3_NAMESPACE`) separates the keys of installations that share a
+bucket. Owned keys become `asset-namespaces/<namespace>/<group>/<existing-key>`. Two prefixes
+stay at the bucket root and are shared by every namespace: content-addressed blobs
+(`asset-blobs/`), so identical bytes are stored once, and shared extracted metadata
+(`extracted-metadata/`), so identical bytes are extracted once. Use the same
+value for all nodes in one cluster and distinct values for independent installations. Values
+are 1 to 64 letters, digits, underscores or hyphens, starting with a letter or digit. Empty is
+the default and keeps the existing layout.
+
+Set the namespace before the installation first stores assets. Changing it later does not
+migrate existing records. It separates application keys only; it does not restrict what the
+bucket credentials can access.
+
+## Static push publishing
+
+Enabling the flag also changes static push publishing to S3 (`AWSS3Storage`). Clients sign
+with SigV4 and list every page of objects instead of only the first 1,000. Without a custom
+endpoint, a configured region selects that region's endpoint; with no region, the client uses
+the global endpoint and the SDK looks up the bucket's region, so a region is not required.
+Endpoints that use the default credential chain are not pinned to a region either. Their
+configured endpoint and region are still not passed to the client on that path, as before, so
+an S3-compatible custom endpoint needs a key and secret.
+
+## Running the checks
+
+The S3 checks use the real filesystem provider, chain and AWS adapter against a disposable
+MinIO bucket. `BinaryS3StorageTest` is skipped unless `s3.test.endpoint` is set; the other
+tests always run. The credentials below are disposable local test values.
+
+```sh
+docker run -d --rm --name binary-s3-test \
+  -p 127.0.0.1:19002:9000 \
+  -e MINIO_ROOT_USER=binary-storage-test \
+  -e MINIO_ROOT_PASSWORD=binary-storage-test \
+  minio/minio:latest server /data
+
+./mvnw test -pl :dotcms-core -Dmaven.build.cache.enabled=false \
+  -Dtest=AssetStorageFeatureTest,AssetStorageFeatureLatchTest,S3StorageConfigurationTest,NoWebIdentityCredentialsProviderChainTest,BinaryS3StorageTest \
+  -Ds3.test.endpoint=http://127.0.0.1:19002
+
+docker stop binary-s3-test
+```
+
+CI does not yet provide the MinIO service, so `BinaryS3StorageTest` does not run there.

dotCMS/pom.xml

@@ -763,6 +763,10 @@
             <groupId>com.amazonaws</groupId>
             <artifactId>aws-java-sdk-s3</artifactId>
         </dependency>
+        <dependency>
+            <groupId>com.amazonaws</groupId>
+            <artifactId>aws-java-sdk-sts</artifactId>
+        </dependency>
         <dependency>
             <groupId>software.amazon.awssdk</groupId>
             <artifactId>sqs</artifactId>

dotCMS/src/enterprise/java/com/dotcms/enterprise/publishing/staticpublishing/AWSS3Publisher.java

@@ -20,6 +20,7 @@
 import com.dotcms.publisher.environment.business.EnvironmentAPI;
 import com.dotcms.publisher.pusher.PushUtils;
 import com.dotcms.publisher.util.PusheableAsset;
+import com.dotcms.storage.NoWebIdentityCredentialsProviderChain;
 import com.dotcms.publishing.*;
 import com.dotcms.system.event.local.business.LocalSystemEventsAPI;
 import com.dotcms.system.event.local.type.pushpublish.EndpointFailureDetail;
@@ -507,6 +508,8 @@ private String updateStatusFailedToSend(PublishAuditHistory currentStatusHistory
     /**
 	 * Creates and returns an AWSS3EndPointPublisher.
 	 * Uses the default AWS credentials provider chain if the tokenProp and/or secretProp are not set.
+	 * With S3 asset storage off, that chain leaves out the web identity step, as it did before the
+	 * STS module was packaged (see {@link NoWebIdentityCredentialsProviderChain#defaultChain()}).
 	 * Uses the provided tokenProp and secretProp if they are both set.
 	 *
 	 * @param tokenProp - the AWS credentials token key
@@ -519,7 +522,7 @@ private AWSS3EndPointPublisher getAWSS3EndPointPublisher(final String tokenProp,
             final String secretProp, final String endPoint, final String region) {
 		AWSS3EndPointPublisher endPointPublisher;
 		if (!UtilMethods.isSet(tokenProp) || !UtilMethods.isSet(secretProp)) {
-			DefaultAWSCredentialsProviderChain creds = new DefaultAWSCredentialsProviderChain();
+			DefaultAWSCredentialsProviderChain creds = NoWebIdentityCredentialsProviderChain.defaultChain();
 			endPointPublisher = new AWSS3EndPointPublisher(creds);
 		} else {
 			AWSS3Configuration awss3Configuration =