AWS non-retryable streaming request error

[gauthampkrishnan]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I am trying to upload a file using aws putObject but with body as createReadStream and using retryMode as standard but on retry it throw error non-retryable streaming request error.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/package-name@version, ...

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

18.20.4

Reproduction Steps

try to upload a file using body as stream and try to replicate a failure while uploading file

Observed Behavior

non-retryable streaming request error

Expected Behavior

I think it shouldnt throw error and it should retry

Possible Solution

No response

Additional Information/Context

No response

[kuhe]

That is normal. The request reads the stream but cannot rewind it if it fails.

[gauthampkrishnan]

So what is the solution to do retries while using streams as body ? are there any solution for this scenario ?

[gauthampkrishnan]

Would using Upload() method from lib-storage helps in this case ?https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-lib-storage/ as i am guessing this would buffer the parts and buffered parts would be retried depending on the client retry config?

[kuhe]

You can buffer the stream or

let attempts = 3;
let stream = ... ;
while (--attempts) {
  try {
    await s3.putObject({ Body: stream });
  break; 
  } catch (e) {
    stream = ... ; // rewind/reacquire
  }
}

[aBurmeseDev]

@gauthampkrishnan - Can you share the code you are working on so that we can better assist you? Here's docs on SDK retry strategies and behavior for your reference: https://github.com/aws/aws-sdk-js-v3/blob/main/supplemental-docs/CLIENTS.md#retry-strategy-retrystrategy-retrymode-maxattempts

[gauthampkrishnan]

@aBurmeseDev the code is simple, example

const stream = createReadStream(filePath)
try {
    await s3.putObject({ Body: stream });
  } catch (e) {
   console.log(e)
  }

s3 uses nodehttphandler with retry settings

and i am using the nodeHttpHandler by passing in maxRetries so it retries for buffer, but for stream it will throw error but based on above conversation looks like that is expected and need to implement a custom function with error handler inorder to retry for body which uses stream. I think the Upload() function from library lib-storage does the same under the hood it reads stream and creates buffered parts and I also believe the buffered parts would be retried on fail so I guess I will use same and maybe this issue can be closed.

[aBurmeseDev]

In definition, Upload() allows for efficient uploading of buffers, blobs, or streams, using a configurable amount of concurrency to perform multipart uploads where possible. This enables uploading large files or streams of unknown size due to the use of multipart uploads under the hood. Here's some code examples for Upload() if you might find it helpful: https://github.com/aws/aws-sdk-js-v3/tree/main/lib/lib-storage/example-code

Let us know if you need further support.

[gauthampkrishnan]

Ya so my doubt is if I pass in a retry config with maxRetries to s3 client and try to use the Upload() https://github.com/aws/aws-sdk-js-v3/tree/main/lib/lib-storage/example-code and as its using multipart file upload under the hood will it do the retry of parts if it fails in between? for example lets say i am using stream and upload() function so upload() will divide the data into parts and uploads to bucket lets say one of the part failed in that case would it retry uploading that part?