Support IAM role / default credential chain for S3 auth

[ynotoast]

What

Make s3_access_key and s3_secret_key optional. When unset, boto3 falls through to its standard credential chain — env vars, ~/.aws/credentials, EC2/ECS instance metadata (IAM role).

Why

All four boto3 call sites currently pass credentials explicitly, which bypasses boto3's credential chain entirely. There is no way to authenticate via an EC2 instance profile or ECS task role without creating long-lived IAM user keys — against AWS best practice for hosted deployments.

Changes

• src/config/s3.py, src/config/__init__.py — str with min_length → Optional[str] = None
• Extract S3Config.make_client() to centralise boto3 client construction and credential-injection logic
• Four call sites updated to settings.s3.make_client() — direct boto3 imports removed
• Test fixtures updated to patch make_client() directly

Existing behaviour with explicit keys (Garage, MinIO, etc.) is fully unchanged.

[usnavy13]

Thanks for adding support for IAM role / default credential-chain auth. The direction makes sense, but I think there are two issues to address before merging.

1. Existing startup/health S3 validation still requires a broad IAM permission

The PR removes explicit static credentials when S3_ACCESS_KEY / S3_SECRET_KEY are unset, which lets boto3 use IAM roles or the default credential chain. However, the app still validates S3 by calling list_buckets() in ConfigValidator._validate_s3_connection(), and the health check also calls list_buckets().

That means an otherwise valid least-privilege IAM role can still fail startup or health checks if it only has access to the configured app bucket. For example, a role with permissions for code-interpreter-files can successfully HeadBucket, PutObject, and GetObject, but may not have s3:ListAllMyBuckets.

This issue already exists in the current code, but this PR makes it more important because IAM role support is the goal. I’d suggest validating the configured bucket directly with head_bucket(Bucket=settings.s3_bucket) instead of listing all buckets. Health checks should probably do the same.

Relevant locations:

• src/utils/config_validator.py
• src/services/health.py

2. Partial static credential config is silently ignored

S3Config.make_client() only passes credentials to boto3 when both access_key and secret_key are truthy:

if self.access_key and self.secret_key:
    kwargs["aws_access_key_id"] = self.access_key
    kwargs["aws_secret_access_key"] = self.secret_key

That means if an operator accidentally sets only S3_ACCESS_KEY or only S3_SECRET_KEY, the app silently ignores the partial static config and falls back to ambient AWS credentials / IAM role credentials. That could make the app run under a different identity than intended.

This behavior is introduced by this PR. I’d suggest making it a config error when exactly one of S3_ACCESS_KEY / S3_SECRET_KEY is set. Either both should be provided for static S3-compatible auth, or both should be omitted to use boto3’s default credential chain.

Thanks for submitting this. If you can address those issues and verify it working on your end with tests, that would be helpful since we don’t yet have AWS S3 deployed for testing.

[ynotoast]

Thanks @usnavy13 for the quick review!

I have implemented the suggested changes

1. Followed your suggestion to use head_bucket(Bucket=settings.s3_bucket) to fix the broad IAM permissions bug in health.py and config_validator.py. I considered removing the bucket existence check from the validator entirely, however this would mean that misconfiguration would not be caught until the first health check.

2. easy fix for the second point - implemented as suggested too.

I am planning on using my fork for testing in the wild while time zones are out of sync. I will provide relevant findings here :)

[ynotoast]

Production test results - IAM role auth working in the wild

Deployed ynotoast:fix/iam-role-s3-auth to a production EC2 instance (ARM64/Graviton). Config: S3_ACCESS_KEY and S3_SECRET_KEY both unset, EC2 instance IAM role granted s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on the configured bucket only — no s3:ListAllMyBuckets.

All three S3 paths confirmed working:

1. Code execution - Python and Bash executing correctly, 70-200ms latency
2. File upload + read - uploaded a .docx, model read it from /mnt/data/ and summarised contents correctly
3. File generation + download - model wrote an output file to /mnt/data/, download link appeared in UI, file served back correctly

IAM credential chain: zero S3/credential errors across all operations. boto3 picked up the EC2 IAM role silently with no static keys configured. The head_bucket fix is confirmed correct — startup and health checks pass cleanly with the least-privilege role.

Happy to provide logs if useful for the merge decision.

[usnavy13]

Thanks for the update. The two blocking issues from my previous review look addressed now