Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,8 +337,9 @@ Legend:
| CopyObject | Yes | Supports copy and metadata replacement. |
| Content-MD5 validation | Yes | Invalid and mismatched digests return S3-style errors. |
| Multipart upload | Yes | Create/upload/list/complete/abort local multipart flows. |
| Presigned URL validation | Yes | Covered for local SigV4 GET. |
| AWS SigV4 header auth | Partial | Relaxed mode is default; strict mode validates local credentials. |
| Presigned URL validation | Yes | Strict mode validates SigV4 presigned GET and PUT, including expiry. |
| AWS SigV4 header auth | Partial | Relaxed mode is default; strict mode validates local credentials, a 15-minute clock skew window, and `aws-chunked` chunk and trailer signatures. SigV2 and SigV4a are not implemented. |
| CORS | Yes | PutBucketCors/GetBucketCors/DeleteBucketCors, preflight, and response headers. |
| ACLs, bucket policy, IAM | Partial | Bucket policy and bucket/object ACL metadata endpoints are supported locally, including versionId-aware object ACL metadata; IAM enforcement is not implemented. |
| Versioning | Partial | Bucket versioning, generated and `null` version IDs, version-aware get/delete/copy-source, delete markers, multipart-complete version IDs/ETags, and local ListObjectVersions with key/version markers are supported. |
| Lifecycle | Partial | Bucket lifecycle metadata endpoints are supported; Enabled expiration rules for current objects are applied locally on S3 reads/lists. |
Expand All @@ -349,6 +350,8 @@ Legend:
| S3 Select | Partial | Minimal SelectObjectContent supports `SELECT * FROM S3Object` for CSV and JSON Lines with eventstream responses; filtering and projections are not implemented. |
| Inventory / analytics | Partial | Bucket inventory and analytics configuration metadata endpoints are supported locally; CSV-format enabled inventory configs generate deterministic local reports under bucket storage. |

The standalone S3 image also ships a web console and a server-wide CORS default; see [`services/s3/README.md`](services/s3/README.md).

### GCS-Compatible JSON API

| Feature | Status | Notes |
Expand Down
1 change: 1 addition & 0 deletions services/s3/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,5 @@ md-5 = "0.10"
sha2 = "0.10"
hmac = "0.12"
hex = "0.4"
include_dir = "0.7"
tokio = { version = "1", features = ["rt", "rt-multi-thread", "macros", "net", "io-util", "signal", "time", "sync"] }
15 changes: 10 additions & 5 deletions services/s3/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,15 @@
# Multi-arch (the build stage runs natively and cross-compiles, because rustc
# crashes under amd64 emulation on arm64 hosts):
# docker buildx build --platform linux/amd64,linux/arm64 -f services/s3/Dockerfile .
# Run:
# docker run --rm -p 9000:9000 -v devcloud-s3-data:/data devcloud-s3
# Run (S3 API on 9000, web console on 9001; see services/s3/README.md):
# docker run --rm -p 127.0.0.1:9000:9000 -p 127.0.0.1:9001:9001 \
# -v devcloud-s3-data:/data devcloud-s3
#
# Auth defaults to relaxed. Strict auth: pass DEVCLOUD_S3_AUTH_MODE=strict together with
# DEVCLOUD_S3_ACCESS_KEY_ID / DEVCLOUD_S3_SECRET_ACCESS_KEY / DEVCLOUD_S3_REGION.
# Auth defaults to relaxed, which also leaves the console open. Signature
# verification: pass DEVCLOUD_S3_AUTH_MODE=strict together with
# DEVCLOUD_S3_ACCESS_KEY_ID / DEVCLOUD_S3_SECRET_ACCESS_KEY (and optionally
# DEVCLOUD_S3_REGION). CORS: PutBucketCors per bucket, or
# DEVCLOUD_S3_CORS_ALLOW_ORIGINS as a default for every bucket.
# Create buckets at startup with a comma-separated list (existing buckets are kept):
# docker run --rm -p 9000:9000 -e DEVCLOUD_S3_BUCKETS=uploads,backups devcloud-s3

Expand Down Expand Up @@ -42,7 +46,8 @@ RUN useradd --system --uid 10001 --no-create-home devcloud \
COPY --from=build /devcloud-s3 /usr/local/bin/devcloud-s3
USER devcloud
ENV DEVCLOUD_S3_ADDR=0.0.0.0:9000 \
DEVCLOUD_S3_CONSOLE_ADDR=0.0.0.0:9001 \
DEVCLOUD_S3_STORAGE=/data
VOLUME /data
EXPOSE 9000
EXPOSE 9000 9001
ENTRYPOINT ["devcloud-s3"]
99 changes: 99 additions & 0 deletions services/s3/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# devcloud-s3

Standalone S3-compatible endpoint with a web console. The image runs without the
orchestrator, dashboard, or event relay, and needs no runtime JavaScript tools.

```sh
docker build -f services/s3/Dockerfile -t devcloud-s3 .
docker run --rm \
-p 127.0.0.1:9000:9000 -p 127.0.0.1:9001:9001 \
-v devcloud-s3-data:/data devcloud-s3
# S3 API: http://127.0.0.1:9000 Web console: http://127.0.0.1:9001
```

The image runs as uid **10001**. For a bind mount, make the directory writable by
that uid first. Requests are path-style only and capped at 128 MiB.

| Environment variable | Image default | Meaning |
| --- | --- | --- |
| `DEVCLOUD_S3_ADDR` | `0.0.0.0:9000` | S3 API listen address |
| `DEVCLOUD_S3_STORAGE` | `/data` | Bucket storage root (required) |
| `DEVCLOUD_S3_AUTH_MODE` | `relaxed` | `relaxed` or `strict`; any other value refuses to start |
| `DEVCLOUD_S3_ACCESS_KEY_ID` | empty | Access key, required in strict mode |
| `DEVCLOUD_S3_SECRET_ACCESS_KEY` | empty | Secret key, required in strict mode |
| `DEVCLOUD_S3_REGION` | `us-east-1` | Region accepted in the SigV4 credential scope |
| `DEVCLOUD_S3_BUCKETS` | empty | Comma-separated buckets to create at startup; existing buckets are kept |
| `DEVCLOUD_S3_CORS_ALLOW_ORIGINS` | empty | Comma-separated origins (`*` and one-wildcard patterns allowed) for buckets without their own CORS configuration |
| `DEVCLOUD_S3_CONSOLE_ADDR` | `0.0.0.0:9001` | Console listen address; empty disables the console |
| `DEVCLOUD_S3_CONSOLE_ALLOWED_HOSTS` | empty | Extra comma-separated console Host names; `*` disables the check |

## Signature verification

Relaxed mode accepts every request. To verify SigV4:

```sh
docker run --rm -p 127.0.0.1:9000:9000 -p 127.0.0.1:9001:9001 \
-e DEVCLOUD_S3_AUTH_MODE=strict \
-e DEVCLOUD_S3_ACCESS_KEY_ID=devkey \
-e DEVCLOUD_S3_SECRET_ACCESS_KEY=devsecret \
devcloud-s3
```

Strict mode checks header-signed requests, presigned URLs (GET and PUT, including
expiry), the payload hash, and `aws-chunked` uploads
(`STREAMING-AWS4-HMAC-SHA256-PAYLOAD[-TRAILER]`, whose chunks and trailer are each
verified, and `STREAMING-UNSIGNED-PAYLOAD-TRAILER`). It also requires:

- `x-amz-date` within 15 minutes of the server clock (`RequestTimeTooSkewed`)
- `host` and every `x-amz-*` header to be signed, except `x-amz-user-agent` and
`x-amz-content-sha256` (whose value is signed as the payload hash)
- the credential scope to match the configured access key, region, and request date

Trailer checksums such as `x-amz-checksum-crc32` are accepted but not validated.
SigV2 and SigV4a are not implemented.

## CORS

Without configuration no cross-origin access is granted. Configure a bucket with
the standard API:

```sh
aws --endpoint-url http://127.0.0.1:9000 s3api put-bucket-cors --bucket demo \
--cors-configuration '{"CORSRules":[{"AllowedOrigins":["http://localhost:5173"],
"AllowedMethods":["GET","PUT"],"AllowedHeaders":["*"],"ExposeHeaders":["ETag"]}]}'
```

Or set a default for every bucket without its own rules, which also covers
`ListBuckets` and creating buckets:

```sh
docker run ... -e DEVCLOUD_S3_CORS_ALLOW_ORIGINS=http://localhost:5173 devcloud-s3
```

Preflight `OPTIONS` needs no signature, since browsers never sign it; the actual
request is still verified in strict mode. Avoid `*` with relaxed auth on a machine
that browses the web: any site could then read and change your local buckets.

## Web console

The console lists, creates, and deletes buckets, and browses, uploads, downloads,
and deletes objects. Mutations go through the S3 provider protocol, so they behave
like API calls.

- **Relaxed mode:** no login. Publish port 9001 on `127.0.0.1` as shown above.
- **Strict mode:** HTTP Basic, with the access key as the user name and the secret
key as the password. Basic credentials travel in clear text, so use TLS in front
of it outside a local machine.
- **Other hostnames:** names containing a dot are refused to prevent DNS rebinding.
To reach the console as `s3.dev.example.com`, set
`DEVCLOUD_S3_CONSOLE_ALLOWED_HOSTS=s3.dev.example.com`.

The UI source is `web/s3-console`. `npm run build` there writes
`services/s3/assets/console`, which is committed and embedded into the binary.

## End-to-end tests

`npm run test:e2e` in `web/s3-console` builds this image and tests it through real
sockets, a real browser, and the AWS SDK for JavaScript. It needs Docker and
Playwright's Chromium (`npx playwright install chromium`), and writes under
`web/s3-console/dist/e2e`.
1 change: 1 addition & 0 deletions services/s3/assets/console/assets/index-Bxmjs9G2.css

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading