Retries to AWS Bedrock result in 403 signature verification errors

[kevindew]

We've been receiving some cryptic AWS signature verification errors in our codebase when using this gem to communicate invoke models via AWS Bedrock. Example:

{url: "https://bedrock-runtime.eu-west-1.amazonaws.com/model/eu.anthropic.claude-sonnet-4-20250514-v1%3A0/invoke", status: 403, body: {message: "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/model/eu.anthropic.claude-sonnet-4-20250514-v1%253A0/invoke\n\naccept:application/json\nanthropic-version:2023-06-01\ncontent-type:application/json\nhost:bedrock-runtime.eu-west-1.amazonaws.com\nx-amz-content-sha256:696e618325a4088b4f978c36564fe48358c6dcc639d85fa7141fcf0da7ccd46b\nx-amz-date:20250731T102639Z\nx-amz-security-token:<big security token>\nx-stainless-arch:arm64\nx-stainless-lang:ruby\nx-stainless-os:Linux\nx-stainless-package-version:1.3.0\nx-stainless-retry-count:1\nx-stainless-runtime:ruby\nx-stainless-runtime-version:3.4.4\nx-stainless-timeout:600.0\n\naccept;anthropic-version;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-stainless-arch;x-stainless-lang;x-stainless-os;x-stainless-package-version;x-stainless-retry-count;x-stainless-runtime;x-stainless-runtime-version;x-stainless-timeout\n696e618325a4088b4f978c36564fe48358c6dcc639d85fa7141fcf0da7ccd46b'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20250731T102639Z\n20250731/eu-west-1/bedrock/aws4_request\n7e4fb0efce54f63dc761fe80623f13981cab4a02ac2f613ec3a96b3f2a130727'\n"}} (Anthropic::Errors::PermissionDeniedError)

We learnt from AWS support that the times when we received this error resulted in ServiceUnavailableException on their end. We've since dug in further and understand that if AWS Bedrock returns a 503 this gem will try to retry and then the subsequent request will result in a 403.

As this is terribly hard to replicate - requiring AWS Bedrock to return an error - I mocked up a crude monkey patch script to demonstrate the issue for a reproduction:

require "bundler/inline"

gemfile do
  source 'https://rubygems.org'

  gem "anthropic"
  gem "aws-sdk-bedrockruntime"
end

require "anthropic"

class Anthropic::Internal::Transport::PooledNetRequester
  alias_method :original_execute, :execute

  def execute(request)
    status, response, body = original_execute(request)

    if status == 200
      # fake a service unavailable response
      return [503, response, body]
    else
      return status, response, body
    end
  end
end


anthropic = Anthropic::BedrockClient.new

message = anthropic.messages.create(
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: "Hello, Claude"
    }
  ],
  model: "eu.anthropic.claude-sonnet-4-20250514-v1:0"
)

puts(message)

Executing this seems to reliably demonstrate the 403 error

[dtmeadows]

Thanks @kevindew! We'll take a deeper look into this and see how we can fix this!

[ms-jpq]

This is caused by us not re-signing the request during retries, will have a fix out soon!

and thank you so much for making it a reproducible issue @kevindew ❤️‍🩹!

[ms-jpq]

df9f4e1

will close this when released

[kevindew]

Thanks for the fix @ms-jpq

I think this may have caused a regression though I've only spent a couple of minutes looking at it.

Version 1.4.0

➜  ruby cat anthropic-bedrock.rb
require "bundler/inline"

gemfile do
  source 'https://rubygems.org'

  gem "anthropic", "1.4.0"
  gem "aws-sdk-bedrockruntime"
end

require "anthropic"


anthropic = Anthropic::BedrockClient.new

message = anthropic.messages.create(
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: "Hello, Claude"
    }
  ],
  model: "eu.anthropic.claude-sonnet-4-20250514-v1:0"
)

puts(message)
➜  ruby ruby anthropic-bedrock.rb
{:id=>"msg_bdrk_015hqkHBc7zzGRUXdwcbZUUe", :content=>[{:text=>"Hello! It's nice to meet you. How are you doing today? Is there anything I can help you with or would you like to chat about?", :type=>:text}], :model=>:"claude-sonnet-4-20250514", :role=>:assistant, :stop_reason=>:end_turn, :stop_sequence=>nil, :type=>:message, :usage=>{:cache_creation_input_tokens=>0, :cache_read_input_tokens=>0, :input_tokens=>10, :output_tokens=>34}}

Version 1.4.1

➜  ruby cat anthropic-bedrock.rb
require "bundler/inline"

gemfile do
  source 'https://rubygems.org'

  gem "anthropic", "1.4.1"
  gem "aws-sdk-bedrockruntime"
end

require "anthropic"

anthropic = Anthropic::BedrockClient.new

message = anthropic.messages.create(
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: "Hello, Claude"
    }
  ],
  model: "eu.anthropic.claude-sonnet-4-20250514-v1:0"
)

puts(message)
➜  ruby ruby anthropic-bedrock.rb
{:Output=>{:__type=>"com.amazon.coral.service#UnknownOperationException"}, :Version=>"1.0"}

There is potentially a clue in the application I'm working on's test suite which is failing for this update - where it looks like rather than hitting the AWS invoke model endpoint ("/https:\/\/bedrock-runtime\..*\.amazonaws\.com\/model\/.*anthropic\.claude.*?\/invoke/") this is hitting Claude's first party API of /messages ( Real HTTP connections are disabled. Unregistered request: POST https://bedrock-runtime.my-region.amazonaws.com/v1/messages )

[ms-jpq]

ahhhh @kevindew, you are absolutely right, 😞. I will have a fix out shortly

fixing commit: b2996b9

[ms-jpq]

ahh, forgot to ping @kevindew, the fix has been released a couple of hours ago

[joshmlewis]

This still seems to be happening in 1.9. Has anyone found a fix? It seems to be on retries.

[dtmeadows]

Hey @joshmlewis any more details you can share on what you're seeing?

[davidgisbey]

Hi @dtmeadows,

I work with Kev. We saw this issue again this morning when there were problems with Bedrock on AWS' end.

The initial script Kev provided still seems to reliably reproduce the issue:

require "bundler/inline"

gemfile do
  source 'https://rubygems.org'

  gem "anthropic"
  gem "aws-sdk-bedrockruntime"
end

require "anthropic"

class Anthropic::Internal::Transport::PooledNetRequester
  alias_method :original_execute, :execute

  def execute(request)
    status, response, body = original_execute(request)

    if status == 200
      # fake a service unavailable response
      return [503, response, body]
    else
      return status, response, body
    end
  end
end


anthropic = Anthropic::BedrockClient.new

message = anthropic.messages.create(
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: "Hello, Claude"
    }
  ],
  model: "eu.anthropic.claude-sonnet-4-20250514-v1:0"
)

puts(message)

[ms-jpq]

hi @davidgisbey :

ahhhh :(, will take a look at fixing the root cause.

it is a pretty annoying interaction with our retry & lazy uploads logic.

[ms-jpq]

should be closed by: #131

[kevindew]

Great, thank you @ms-jpq. I've confirmed that the script above can no replicate the issue so 🤞