Stalled-stream protection

[Velfi]

What is stalled stream protection?

When making HTTP calls, it's possible for a connection to 'stall out' and emit no more data due to server-side issues. In the event this happens, it's desirable for the stream to error out as quickly as possible. While timeouts can protect you from this issue, they aren't adaptive to the amount of data being sent and so must be configured specifically for each use case. When enabled, stalled-stream protection will ensure that bad streams error out quickly, regardless of the amount of data being
downloaded.

Protection is enabled by default for all clients but can be configured or disabled.

Configuring stalled stream protection

By default, stalled stream protection checks stream throughput once per second. If throughput is below 1B/s for longer than 5 seconds (the default grace period), then the stream will error out.

You can configure this grace period by modifying the SDK config:

use aws_smithy_runtime_api::client::stalled_stream_protection::StalledStreamProtectionConfig;
use std::time::Duration;

let config = aws_config::from_env()
    .stalled_stream_protection(StalledStreamProtectionConfig::enabled().build())
    .load()
    .await;

Or by modifying the service config:

// The config struct is re-exported in the `config` module of all SDK crates.
use aws_sdk_s3::config::StalledStreamProtectionConfig;
use aws_sdk_s3::Config;
use std::time::Duration;

let config = Config::builder()
    .stalled_stream_protection(
        StalledStreamProtectionConfig::enabled()
            .grace_period(Duration::from_secs(1))
            .build()
    )
    .build();

Disabling stalled stream protection

Just as in the previous examples, stalled stream can be disabled at the SDK config and service config levels:

use aws_smithy_runtime_api::client::stalled_stream_protection::StalledStreamProtectionConfig;

let config = aws_config::from_env()
    // Calling `.build()` is unnecessary when disabling stalled stream protection with `StalledStreamProtectionConfig::disabled()`.
    .stalled_stream_protection(StalledStreamProtectionConfig::disabled())
    .load()
    .await;

Additionally, it's possible to disable stalled stream protection for uploads or downloads individually:

use aws_smithy_runtime_api::client::stalled_stream_protection::StalledStreamProtectionConfig;

let config = aws_config::from_env()
    .stalled_stream_protection(
        // When creating a `StalledStreamProtectionConfig` with `enabled()`, protection is enabled for both uploads and downloads.
        StalledStreamProtectionConfig::enabled()
            .download_enabled(false) // disables download protection
            .upload_enabled(false)      // disables upload protection
            .build()
    )
    .load()
    .await;

Configuring the "grace period"

By default, once a stream's transfer rate dips below the allowed limit for five seconds, it's cancelled and an error is returned. If this five-second grace period isn't a good fit for your workflow, you can change it or disable it:

use aws_smithy_types::stalled_stream_protection::StalledStreamProtectionConfig;

let config = aws_config::from_env()
    .stalled_stream_protection(
        StalledStreamProtectionConfig::enabled()
            .grace_period(Duration::from_secs(1)) // Passing a zero-duration will disable the grace period.
            .build()
    )
    .load()
    .await;

[MathieuDuponchelle]

@Velfi I might be missing something here, but wasn't it a bit rash to make this an opt-out behavior? Some services like AWS transcribe will send out sparse data over websocket by design, suddenly starting to error out after 5 seconds with no data is a bit of a rug pull :)

[Velfi]

When we make decisions, we go through lots of discussion and design review. While we try very hard to make the right decisions for all of our users, sometimes our decisions degrade the experience of other users. I apologize for the trouble that this feature caused you; It wasn't our intention to "rug pull" you.

Some services like AWS transcribe will send out sparse data over websocket by design

Despite it being a near-requirement for this job, we still struggle to know the behavior of every operation for all 400+ services. Furthermore, because this behavior isn't modeled, it's hard for my team to "see". You can expect that we will continue to make mistakes but you can also expect that we'll do our best to address them in a timely manner.

[MathieuDuponchelle]

Hey, I understand, no worries.

we still struggle to know the behavior of every operation for all 400+ services

Right, that's why I would recommend a more cautious approach of disabling the mechanism by default, and only opting into it on a per-service basis. Sending sparse data over a websocket seems like a pretty routine thing to do after all :)