Skip to content

HDDS-15515. Support object Content-Type end-to-end in S3 Gateway#10472

Merged
Gargi-jais11 merged 2 commits into
apache:masterfrom
rich7420:HDDS-15515
Jun 24, 2026
Merged

HDDS-15515. Support object Content-Type end-to-end in S3 Gateway#10472
Gargi-jais11 merged 2 commits into
apache:masterfrom
rich7420:HDDS-15515

Conversation

@rich7420

@rich7420 rich7420 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

The S3 Gateway never persisted an object's Content-Type. The request
Content-Type was only forwarded to HeaderPreprocessor for SigV4 signing
(X-Ozone-Original-Content-Type) and was never stored in the OM key metadata;
GetObject did not set it from the object and HeadObject hard-coded
binary/octet-stream. Because the value is never stored, CopyObject /
UploadPartCopy cannot preserve or replace it, so copied (and plain) objects
consistently default to octet-stream.

This is an end-to-end gap, not a copy-only issue. This PR adds object
Content-Type support across the read/write/copy paths, aligning with AWS S3:

  • PutObject / CreateMultipartUpload — store the request Content-Type in
    the OM key metadata, mirroring the existing ETag-in-metadata convention.
  • GetObject / HeadObject — return the stored Content-Type, defaulting to
    binary/octet-stream when absent (the response-content-type query
    parameter still overrides).
  • CopyObject metadata directive — COPY (default) preserves the source
    object's Content-Type (it rides along in the copied source metadata);
    REPLACE applies the Content-Type from the copy request.
  • A user-supplied x-amz-meta-content-type is remapped to a dedicated key so
    it does not collide with the object's stored Content-Type. The reserved-key
    remap/rebuild that previously existed only for ETag is now factored into a
    shared helper and reused for both.

Before vs after (response):

Path Before After
PUT/MPU Content-Type not persisted stored in key metadata
GET only set from request header / response-content-type falls back to stored Content-Type
HEAD hard-coded binary/octet-stream stored Content-Type (or default)
COPY (COPY) dest defaults to octet-stream keeps source Content-Type
COPY (REPLACE) dest defaults to octet-stream uses request Content-Type

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15515

How was this patch tested?

https://github.com/rich7420/ozone/actions/runs/27210927693

The S3 Gateway never persisted an object's Content-Type: it was only forwarded
to HeaderPreprocessor for SigV4 signing, GET did not set it from the object and
HEAD hard-coded binary/octet-stream, so CopyObject could not preserve/replace it.

Store the request Content-Type in the OM key metadata (mirroring the existing
ETag-in-metadata convention), return it on GET/HEAD (default binary/octet-stream),
and propagate it on CopyObject (COPY keeps the source value, REPLACE uses the
request value). A user-supplied x-amz-meta-content-type is remapped to avoid
colliding with the stored Content-Type. Fixes s3-tests
test_object_copy_{retaining,replacing}_metadata and copy_verify_contenttype.
@rich7420

Copy link
Copy Markdown
Contributor Author

Cc @peterxcli

@Gargi-jais11 Gargi-jais11 requested a review from peterxcli June 10, 2026 06:36
@Gargi-jais11 Gargi-jais11 self-requested a review June 18, 2026 03:56
@Gargi-jais11

Copy link
Copy Markdown
Contributor

Thanks @rich7420 for finding out the issue. I got similar root cause for test case failure in MINT test. Didn't know about your patch, I have marked my jira as duplicate https://issues.apache.org/jira/browse/HDDS-15591. Will review the patch.

Test failure : Mint aws-sdk-go-v2 / testBasicObjectOperations

@rich7420

Copy link
Copy Markdown
Contributor Author

no problem, thanks @Gargi-jais11 to mention that

@Gargi-jais11

Copy link
Copy Markdown
Contributor

Add a testcase to TestObjectHead for x-amz-meta-content-type-
One end-to-end PUT + HEAD test coverage for write remap, read rebuild, and separation of system vs user metadata:
To test user Content Type metadata does not collide with object ContentType.

…date reserved-key remap; add MPU/HEAD tests

- GET no longer reflects the request Content-Type header; it is based on the
  stored object Content-Type with response-content-type still overriding.
- Consolidate the ETag/Content-Type reserved-key remap+rebuild into a single
  RESERVED_METADATA_KEYS table and drop the duplicate CONTENT_TYPE constant.
- Add tests: HEAD x-amz-meta-content-type round-trip and CreateMultipartUpload
  Content-Type storage.
@rich7420

Copy link
Copy Markdown
Contributor Author

@Gargi-jais11 thanks for your comments!

@chungen0126 chungen0126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@Gargi-jais11 Gargi-jais11 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rich7420 for updating the PR. LGTM! , +1.

@Gargi-jais11 Gargi-jais11 merged commit 182aaa7 into apache:master Jun 24, 2026
47 checks passed
@rich7420

Copy link
Copy Markdown
Contributor Author

@Gargi-jais11 and @chungen0126
thanks for the review!

@peterxcli peterxcli added the s3 S3 Gateway label Jun 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

s3 S3 Gateway

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants