S3: put_object doesn't yield to tokio runtime

[rtdavis22]

Hi, I'm calling put_object with a ByteStream that reads from a file. The file is very large so it takes several seconds for the upload to complete. It appears that during the upload, execution is never yielded back to the Tokio runtime. Is there any way that I can yield to the Tokio runtime as the file is uploading?

[jdisanti]

That's odd. Can you provide more details about your Tokio runtime? Are you using the single threaded runtime? What symptoms are you observing that are leading you to conclude that the upload is blocking?

[konnextv]

@rtdavis22 Do you by chance know of any changes in aws-sdk-s3 that mitigate this issue? I think I just encountered this by chance and can't really believe that this has been a known issue for 2 years.

[Velfi]

I wouldn't call this a "known issue" because there is no minimal reproducible example and it's not clear if this is an issue with tokio, the SDK, or the user's code. If you're having an issue, we'd be happy to help so long as we can reproduce it.

[konnextv]

Thanks for your feedback @Velfi. From reading the tokio issue that is linked above I can imagine this will be a hard one to reproduce.
I will dig into this tomorrow, but the outline of what I am doing in a loop (throttled by tokio::time::sleep and with a oneshot channel that is monitored using the tokio::select! macro) is the following:

• iterate over matching files in a directory, then for each file:
• do ByteStream::from_path(&path).await
• create a PubObjectFluentBuilder supplying the bytestream (with .unwrap())
• send the request with request.send().await

Here is why I think what I witnessed matches what the tokio issue is about. The loop was not exited when the oneshot channel was fired which can be explained by blocking.
NOTE: I am kind of new to Rust. While writing this I realize that I am using potentially blocking filesystem functions inside the spawned future - from what I learned so far this is not a good practice but the filesystem should not block indefinitely (correct me if I am wrong). If I am wrong on this, this is my error - I do not want to turn this discussion into a lesson on blocking functions but maybe this will help someone running into this in the future.

[Velfi]

I'd be curious to know

• How many files are you dealing with?
• What is the median size of files + are there any outliers?
• How are you limiting concurrency?
• Where is this code running? What are the specs of the runner.

[konnextv]

• How many files are you dealing with?
  1-2 per iteration (apart from initial exports when there could be a few hundred files)
• What is the median size of files + are there any outliers?
  ~1.5MB per file with few exceptions where files are smaller
• How are you limiting concurrency?
  There is one async fn that iterates over the files. That is run in a loop with a sleep/cancel after. I was using rt-multi-thread but
  switched to rt after realizing that I do not need multithreading.
• Where is this code running? What are the specs of the runner?
  An on-premise Windows 10 workstation. CPUs are Intel i5/i7 with around 12 threads.