You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dcb755b
Browse filesBrowse the repository at this point in the historyBrowse files
Adds an S3-compatible wire protocol via the s3s crate, mounted as
fallback_service alongside the existing /v1/* REST routes. SigV4 maps
onto the existing HMAC-derived bucket token scheme so AWS SDKs work with
(accessKeyId: bucket, secretAccessKey: deriveToken(root, bucket)).
Multipart uploads are exposed only on the S3 surface; storage gains
init/write_part/complete/abort/list with crash-safe rename semantics and
GC for orphaned upload dirs. Adds deriveS3Credentials to the TS SDK and
refreshes DESIGN.md plus a new ARCHITECTURE.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: DESIGN.md
+44-2Lines changed: 44 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,10 +24,27 @@ Clean JSON API, Beyond bearer-token auth, no XML, no Sig V4. This is what the Ty
24
24
25
25
### S3-compatible
26
26
27
-
Full S3 wire protocol via [`s3s`](https://github.com/s3s-project/s3s) — a hyper middleware that handles S3 routing, request parsing, multipart lifecycle, and error serialization. We implement one `S3` trait backed by the filesystem.
27
+
Full S3 wire protocol via [`s3s`](https://github.com/Nugine/s3s) — a tower service that handles S3 routing, request parsing, multipart lifecycle, and error serialization. We implement one `S3` trait backed by the same filesystem.
28
28
29
29
S3 "buckets" map directly to Beyond buckets on the filesystem. `CreateBucket` is a `mkdir` + xattr init. `ListBuckets` reads top-level directories.
30
30
31
+
**Auth — SigV4 mapped to our HMAC scheme.** AWS SDKs always sign with `(access_key_id, secret_access_key)`. We answer s3s's "what's the secret for this access_key?" question by reusing the existing token derivation:
So an S3 client using `accessKeyId: "images"` and `secretAccessKey: <hex>` is byte-for-byte equivalent to a REST client sending `Authorization: Bearer <hex>`. Authorization (which buckets a key may touch) is enforced by `S3Access`: `access_key == bucket || access_key == "root"`. The TS SDK exports `deriveS3Credentials(rootToken, bucket)` for symmetry with `deriveToken`.
39
+
40
+
**Path-style only.**`https://{endpoint}/{bucket}/{key…}`. Virtual-hosted-style is not supported (it would require a wildcard cert per project, and AWS clients support `forcePathStyle: true`).
41
+
42
+
**Anonymous reads.** Public objects can be GET/HEAD'd without credentials, mirroring the REST surface. Anonymous writes are denied at `S3Access`.
43
+
44
+
**What we support:**`ListBuckets`, `CreateBucket`, `DeleteBucket`, `HeadBucket`; `PutObject`, `GetObject`, `HeadObject`, `DeleteObject`, `CopyObject`, `ListObjectsV2`; the full multipart five (`CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`, `ListMultipartUploads`, `ListParts`). Within those: `Range`, `If-Match`, `If-None-Match`, `If-Modified-Since`, `If-Unmodified-Since`, `Content-Type`, `x-amz-meta-*`, and `x-amz-acl: public-read | private` (mapped to the same `user.access` xattr the REST surface writes).
@@ -128,7 +145,24 @@ Supported via standard HTTP conditional headers on PUT:
128
145
129
146
### Multipart uploads (S3-compat only)
130
147
131
-
Multipart is not exposed in the native API — streaming handles large uploads directly. The S3-compatible layer supports it for clients that require it. Parts written to `.multipart/{upload_id}/{part_number}`; assembled on `CompleteMultipartUpload` via sequential concatenation → `fsync` → `rename()` → fjall INSERT.
148
+
Multipart is not exposed in the native API — streaming handles large uploads directly. The S3-compatible layer supports it for clients that require it.
{data_dir}/.multipart/{upload_id}/{part_n} raw part bytes; quoted-MD5 etag in `user.etag` xattr
155
+
```
156
+
157
+
`CompleteMultipartUpload`: read the meta sidecar → concatenate parts in caller-supplied order to `.tmp/{uuid}` while computing `MD5({part_md5}|{part_md5}|…)` → fsync → set xattrs from meta → atomic `rename()` to `{bucket}/{key}` → fjall INSERT → `rm -rf .multipart/{upload_id}`. Final etag is `"{md5_of_part_md5s}-{N}"` (AWS convention). `AbortMultipartUpload` is `rm -rf .multipart/{upload_id}` (idempotent).
158
+
159
+
**Crash recovery:**
160
+
161
+
- Crash mid-`UploadPart` → tmp file in `.tmp/`, GC'd by `gc_temp_files`.
162
+
- Crash after `complete` rename, before `.multipart/{id}` cleanup → orphan dir, GC'd by `gc_multipart_uploads`.
163
+
- Crash before rename → object never appears; abort or GC sweeps it.
164
+
165
+
The `.multipart/` directory is dot-prefixed so `list_buckets` skips it automatically.
0 commit comments