Skip to content

Conversation

@GarrettBeatty
Copy link
Contributor

@GarrettBeatty GarrettBeatty commented Nov 4, 2025

This change cherry picks commits related to s3 transfer manager durability that we want to release. All of the commits (except for number 2 and 4) have been approved already.

At a high level the changes are

  1. Increase default part size from 5MB to 8MB
  2. Ensure all fields are copied from high level transfer utility requests to low level s3 requests
    2a. This also includes create new AbortMultipartsuploads api in order to ensure we can capture expectedbucketowner, requestpayer fields.
  3. Multi part upload additional validations

Description

  1. Increase multipart upload default part size to 8MB
  2. 7c56fce <----- manual commit which is a subset of hhttps://github.com/Added PutObjectResponse to TransferUtilityUploadResponse mapping #4045 and just includes the changes related to updating generator to give s3 unit test project access to the s3 project. I basically just did a cherry pick of this PR and then did git reset HEAD~ and then re-added only the files i needed from this PR.
  3. Add missing fields to Transfer Utility request objects
  4. df0a432 <--------- manual commit which removes unseeded tests from previous cherry pick to resolve merge conflict
  5. Fix issue with HeadersCollection in ResponseMapperTests
  6. Add additional validation to UploadPartRequests in Transfer Utility
  7. create TransferUtilityAbortMultiPartUploadRequest and new api

Motivation and Context

This is to ensure we have proper validations in multi part upload and have the recommended part size.

Testing

  1. Dry Run fdfb7a7a-b653-4634-95d4-813ae9984a8c - pass
  2. I also did manual testing using s3 encryption client and confirmed there werent any issues.
    2a. I tried multi part upload and regular upload for the encryption client.
    2b. I tried scenario not specifying part size
    2c. I also tried scenario specifying 10MB part size.
    https://gist.github.com/GarrettBeatty/05dc6ecd47e063f1ac2e458ac6ed2753 test program

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
    There is one runtime behavior change we are making which is increasing the default part size from 5MB to 8MB. We have decided this is okay because the benefit for customers is it will reduce the number of API calls and 8MB is optimal part size as per s3

Checklist

  • My code follows the code style of this project
  • My change requires a change to the documentation
  • I have updated the documentation accordingly
  • I have read the README document
  • I have added tests to cover my changes
  • All new and existing tests passed

License

  • I confirm that this pull request can be released under the Apache 2 license

@GarrettBeatty GarrettBeatty changed the title Feature/uploaddownload2 Transfer Manager Durability and Part Size Updates Nov 4, 2025
@GarrettBeatty GarrettBeatty force-pushed the feature/uploaddownload2 branch from 60dcd2c to a254743 Compare November 5, 2025 17:52
@GarrettBeatty GarrettBeatty requested a review from normj November 5, 2025 18:15
@GarrettBeatty GarrettBeatty changed the title Transfer Manager Durability and Part Size Updates Transfer MPU Validations and Part Size Updates Nov 5, 2025
@GarrettBeatty GarrettBeatty changed the title Transfer MPU Validations and Part Size Updates Transfer MPU Validations, Part Size Updates, and passing all request fields to low level request Nov 5, 2025
@GarrettBeatty GarrettBeatty marked this pull request as ready for review November 5, 2025 21:24
@GarrettBeatty GarrettBeatty merged commit a9ea719 into development Nov 6, 2025
15 checks passed
@GarrettBeatty GarrettBeatty deleted the feature/uploaddownload2 branch November 6, 2025 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants