Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
dd088f8
Add conditional batch removal engine
k-morozov Oct 9, 2026
08edf18
Add S3 conditional batch delete seam
k-morozov Oct 9, 2026
952ae3c
Batch conditional CAS blob redelete
k-morozov Oct 9, 2026
30942fd
Validate conditional S3 batch responses
k-morozov Oct 9, 2026
f75841b
Record conditional GC delete batches
k-morozov Oct 9, 2026
f8f6349
Retry conditional S3 batch after credential refresh
k-morozov Oct 9, 2026
2b2b04c
Probe conditional batch delete capability
k-morozov Oct 12, 2026
e1deef0
Document conditional CAS delete batching
k-morozov Oct 12, 2026
c3ca3b3
Test CAS conditional delete fallback on RustFS
k-morozov Oct 12, 2026
6065d60
Document conditional delete validation boundary
k-morozov Oct 12, 2026
eef7b65
Harden conditional batch delete failures
k-morozov Oct 12, 2026
887e9e2
Keep per-blob redelete state in one struct
k-morozov Oct 12, 2026
598fb7b
Extract conditional delete chunk preparation
k-morozov Oct 12, 2026
af4d3d6
Add `Gc::conditionalDeleteChunkKeys`
k-morozov Oct 12, 2026
a26ea76
Share the I/O job runner between redelete waves
k-morozov Oct 12, 2026
8b0a9b8
Separate redelete result recording from `.meta` cleanup
k-morozov Oct 12, 2026
ad2f86b
Merge antalya-26.8 into cas_batch_deletes
k-morozov Nov 17, 2026
d05e9bd
Fix S3 attempt helper spelling after the 26.8 port
k-morozov Nov 18, 2026
9c2ebb7
Clarify CAS GC conditional-delete job bookkeeping
k-morozov Nov 18, 2026
dadc16a
Report CAS GC failures after each completed I/O wave
k-morozov Nov 18, 2026
de0efdf
Separate CAS GC read and delete job metrics
k-morozov Nov 18, 2026
d45f61e
Use death tests for CAS batch-delete logical errors
k-morozov Nov 18, 2026
64813a7
Verify mixed outcomes in the CAS conditional batch probe
k-morozov Nov 18, 2026
2cd032f
Extract the CAS conditional batch delete probe
k-morozov Nov 18, 2026
0e79378
Restore the required CAS probe structure
k-morozov Nov 18, 2026
d34bfad
Verify CAS conditional batch deletion on RustFS
k-morozov Nov 18, 2026
443df08
Separate conditional and ordinary S3 batch delete support
k-morozov Nov 18, 2026
ca2203c
Document independent S3 batch delete capabilities and RustFS coverage
k-morozov Nov 18, 2026
978fd42
Verify failure handling and cleanup of the CAS conditional batch probe
k-morozov Nov 18, 2026
807b77b
Test conditional batch retries, GC recovery and S3 response validation
k-morozov Nov 18, 2026
487730e
Check each invalid conditional delete against its own diagnostic
k-morozov Nov 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 55 additions & 5 deletions docs/en/antalya/cas/architecture/garbage-collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ follower or a deferred round execution returns before that commit.
| 8 | `fold_ref_intake` | fold | `GET` every new ref-log record and every referenced manifest, extracting blob source edges |
| 9 | `fold_reduce` | fold | Merge prior edges, new deltas and the parent's condemned rows: spare, condemn, graduate or redelete each candidate; compute `suppress_destructive` |
| 10 | `fold_seal_write` | fold | Write the new fold seal once, write-once deterministic, adopting a byte-identical replay instead of rewriting it |
| 11 | `pending_deletes` | fold | The single blob-body delete site: exact-token delete of entries a *previous* round marked `delete_pending`, up to the redelete budget, plus the outcome-log writes |
| 11 | `pending_deletes` | fold | The single blob-body delete site: `HEAD` each entry a *previous* round marked `delete_pending`, then exact-token conditional delete chunks where the mounted backend proved that capability, plus the outcome-log writes |
| 12 | `meta_pool_wait` | fold | Drain the bounded pool of async `.meta` condemn-marker writes queued during the fold |
| 13 | `round_commit` | fold | Retention-prune old generations, then publish the single `gc/state` `CAS` that adopts the whole round |
| 14 | `handoff_reclaim` | post-`CAS` | Reclaim a generation a ref moved off during this round, which the ordinary retention prune already skipped and will not revisit |
Expand Down Expand Up @@ -415,20 +415,70 @@ deletes for entries a *previous* round published as `delete_pending`, processing
`cas_gc_round_redelete_budget` entries per round (the excess is carried unchanged), and writes the
forensic outcome logs.

In the GC code and metrics, a **chunk** is the group of keys handled by one delete job. A chunk can
contain a single key. A **batch** is one storage request that carries multiple keys, such as S3
`DeleteObjects`. The mount probe checks whether the backend supports conditional batches, and GC
uses one-key chunks when that capability is unavailable. Those jobs can still run concurrently
according to `cas_gc_io_concurrency`. **Bulk delete** names the overall feature, including the
sequence of requests needed to process many keys.

- **Runs on:** fold path only
- **Reads:** one `HEAD` per `redelete` entry — the persisted condemned token cannot itself be a
precondition, so the round observes the blob first, which also settles the absent case without
spending a conditional delete
- **Writes / deletes:** a `DELETE` conditional on the observed etag for each `redelete` entry whose
live etag matches the condemned token; one write-once outcome log per shard with settled entries
- **Writes / deletes:** conditional delete chunks for entries whose live etag matches the condemned
token. A chunk is bounded by `cas_gc_bulk_delete_chunk_keys`, the backend conditional-batch limit,
and 1000 keys. An unprobed, unsupported, or `skip_access_check` mount uses one-key chunks. `.meta`
cleanup remains one per settled blob; it is not part of the conditional delete request. One
write-once outcome log is written per shard with settled entries
- **Safety:** an entry is deletable only because a previously committed fold seal published it
`delete_pending` — durable state from an earlier commit, safe at any leader staleness. Exact token
means a stale leader cannot delete a fresh incarnation; `NotFound` and `TokenMismatch` are
tolerated. This round's own commit outcome does not affect the delete's safety.
- **Fails the round if:** a backend delete marker appears in a response — object versioning on a
mis-provisioned pool (`LOGICAL_ERROR`)
- **Observability:** phase row `pending_deletes`; metrics `deleted`, `absent`, `redeleted`,
`graduated`, `replaced`, `spared`, `outcome_logs_written`, `outcome_rows_dropped`
`graduated`, `replaced`, `spared`, `outcome_logs_written`, `outcome_rows_dropped`, `delete_chunks`. `delete_chunks`
counts logical conditional-delete chunks, including sequential execution. `read_blob_jobs_scheduled`
and `delete_chunk_jobs_scheduled` count successfully enqueued read jobs and delete chunk jobs,
respectively. Jobs that later fail still count; enqueue failures do not. Both counters are zero
on the sequential path, including a wave with only one job. `DiskS3DeleteObjects` counts physical
S3 delete attempts, including request retries

GC reports read-job failures after all submitted blob-state reads finish, before starting the
conditional-delete wave. It reports delete-job failures after all submitted delete chunks finish.
Warnings distinguish read jobs, delete chunk jobs, and failures to enqueue either kind of job.
A failed delete job can have an unknown or partial storage outcome; its warning does not imply that
all objects in the chunk remain present.

Read failures do not prevent deletion of successfully read siblings. After both waves, GC records
known outcomes and schedules eligible `.meta` cleanup, then rethrows the original job or enqueue
exception with the lowest input blob index. An unsuccessful round leaves the previously published
pending entries available to the next round.

The `read_blob_jobs_failed` and `delete_chunk_jobs_failed` metrics count jobs that finish with an
exception, including inline execution. A failed delete chunk contributes one job failure, regardless
of its number of blobs. The `read_blob_job_schedule_failures` and `delete_chunk_job_schedule_failures`
metrics each count enqueue refusals, not the number of unsubmitted jobs or blobs. Retries do not
create extra logical jobs; `Gone` and `Mismatch` are normal outcomes, not job failures. These four
metrics replace the former `jobs_failed` aggregate and are emitted even when the phase throws.
Because inline jobs are included only in the execution-failure counters, the scheduling counters
cannot be used as their denominators for an error rate.

The `RoundReport::redelete_failed` field and `CASGCRetiredRedeleteFailed` event retain their existing
units: each failed blob from an executed job counts once, while an enqueue failure counts once per
wave regardless of the number of unsubmitted blobs reported in its warning.

Ordinary S3 batch deletion and conditional batch deletion have independent capability flags. Only
the latter requires the store to enforce a separate `ETag` for each key in `DeleteObjects`. Both
paths use `objects_chunk_size_to_delete` as a size limit. A request-level `NotImplemented` from
the optional batch check leaves GC using one-key chunks. A request-level `InvalidRequest` or
`InvalidArgument` propagates and fails the mount. After a successful probe, a batch request error
follows the normal request retry policy and does not change the pinned capability.

The integration test on RustFS 1.0.6 mounts two disks with GC chunk limits of one and two. It
checks the number of storage requests in each pending-delete phase and lists the bucket to verify
that the blob bodies have disappeared.

Under `suppress_destructive` the `redelete` set is empty by construction, so nothing is deleted, and
there is no graduation either (would-be graduates are carried unchanged); sparing and superseding
Expand Down Expand Up @@ -849,7 +899,7 @@ pool sizes and the chunk size reject `0`:
| `cas_manifest_sweep_delete_budget_keys` | 100 | orphan-manifest sweep `DELETE` budget per round (phases 9, 18) |
| `cas_gc_round_sweep_namespace_budget` | 20 | namespaces whose protection view the sweep may build per page (phase 9) |
| `cas_gc_round_sweep_recovery_op_budget` | 5000 | committed-tail ref-log reads the sweep's recovery walk may spend (phase 9) |
| `cas_gc_bulk_delete_chunk_keys` | 1000 | keys per batch `DELETE` request for write-once families (phases 15, 17); `1` to `1000` |
| `cas_gc_bulk_delete_chunk_keys` | 1000 | keys per batch `DELETE` request for blob redelete and write-once families (phases 11, 15, 17); `1` to `1000` |
| `cas_gc_meta_pool_size` | 16 | bounded pool for condemn-marker writes (phase 12) |
| `cas_gc_io_concurrency` | 16 | bounded pool for the fold's read-ahead of checkpoints, ref logs, manifest bodies and zero-candidate `HEAD`s (phases 8, 9), the graduation `.meta` reads (phase 9), the orphan-sweep planning reads (phase 9), the rebuild read-ahead and the `pending_deletes` `HEAD` + conditional `DELETE` fan-out (phase 11); other GC requests run on the round thread; `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead |

Expand Down
2 changes: 1 addition & 1 deletion docs/en/antalya/cas/bucket-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ establish acceptance of Google's multipart, native-copy, and exact-delete wire b
| AWS S3 | ✓ | Native `ETag`-based conditional dialect for mutable objects and exact deletion; blob publication is unconditional |
| Google Cloud Storage | implementation complete; release gate pending | Generation-token dialect for mutable objects/native-token `HEAD`/exact deletion, opted into via `http_client = gcs_hmac` or `gcp_oauth`; blob publication uses ordinary copy/multipart. Real credentialed GCS groups have not run yet |
| Azure Blob Storage | probably | Azure's REST API documents the equivalent conditional headers, but ClickHouse's Azure object-storage backend does not yet wire up a `CAS` conditional dialect the way the S3 and GCS paths do — untested, not validated by the capability probe |
| Other S3-compatible stores | only with enforced conditional operations | The capability probe is the actual gate: a store that silently ignores `If-None-Match`/`If-Match` (accepting and applying the write regardless) fails the probe and is refused. `RustFS` passes the full battery and is used as the project's test backend; `Garage` was evaluated and rejected because it silently ignores conditional operations |
| Other S3-compatible stores | only with enforced conditional operations | A store that silently ignores required `If-None-Match` or `If-Match` conditions fails the mount probe. RustFS 1.0.6 passes both the required single-key checks and the optional per-object `ETag` check for `DeleteObjects`. Garage was evaluated and rejected because it silently ignores conditional operations. |

The full mechanics of the two dialects — how the backend detects which one a given endpoint speaks,
what the capability probe actually checks, and how exact-token deletes map onto each provider's
Expand Down
2 changes: 1 addition & 1 deletion docs/en/antalya/cas/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ entirely before release. Treat this table as a snapshot of the current build, no
| `cas_part_folder_cache_max_entry_bytes` | 16 MiB | Oversized part-folder views bypass retention above this size |
| `cas_manifest_decode_cache_bytes` | 128 MiB | Manifest decode cache byte budget (`0` disables) |
| `cas_gc_meta_pool_size` | `16` | Bounded pool size for GC per-hash freshness-meta writes |
| `cas_gc_io_concurrency` | `16` | Bounded pool size for GC object-storage requests that run in parallel: the fold's read-ahead (checkpoints, ref logs, manifests, zero-candidate HEADs), the orphan-manifest sweep planning reads, the `SYSTEM CAS GC REBUILD` read-ahead, the `pending_deletes` blob `HEAD` + conditional `DELETE` fan-out, and the namespace janitor's batch deletes of dead ref logs and snapshots. Not covered: meta writes (`cas_gc_meta_pool_size`) and all other GC requests, which run on the round thread. `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead |
| `cas_gc_io_concurrency` | `16` | Bounded pool size for GC object-storage requests that run in parallel: the fold's read-ahead (checkpoints, ref logs, manifests, zero-candidate HEADs), the orphan-manifest sweep planning reads, the `SYSTEM CAS GC REBUILD` read-ahead, the `pending_deletes` blob `HEAD` + conditional `DELETE` chunks, and the namespace janitor's batch deletes of dead ref logs and snapshots. S3 multi-key conditional chunks are enabled only after the writable mount proves their per-object ETag semantics; unsupported storage and `skip_access_check` use one-key conditional deletes. Not covered: meta writes (`cas_gc_meta_pool_size`) and all other GC requests, which run on the round thread. `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead |
| `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove, conditional write, the mount-time bucket-versioning probe), at least 1. Together with the connect cap it forms the attempt envelope (`cas_attempt_timeout_ms + 2 × cap`; the cap is `cas_attempt_timeout_ms` itself when the disk's `connect_timeout_ms` is `0`, else `min(connect_timeout_ms, cas_attempt_timeout_ms)`) that the lease arithmetic reserves: one TCP connect and one TLS handshake under the cap each, send/receive bounded per socket operation by `cas_attempt_timeout_ms`. With background renewal the cadence check requires `cas_mount_renew_period_ms + 2 × envelope + cas_lease_safety_margin_ms < cas_mount_lease_ttl_ms`, which puts an effective ceiling on the frozen connect cap: under the defaults (TTL 40000, period 10000, margin 2000) the envelope must stay under 14000, so with the default attempt timeout a disk `connect_timeout_ms` of `0` or of 4500 ms or more refuses to open writable — lower the connect timeout or raise the TTL if you hit this. Blob body publication does not use this budget: its object-creating request is sent once under the disk's `request_timeout_ms`, with the connect timeout under the same cap |
| `cas_request_window_ms` | `120000` | How long a failing request keeps retrying before it gives up, measured from the start of the call. It applies to control-plane requests and to the blob publication loop. At the limit the request gives up with a retryable `NETWORK_ERROR`. A writable disk refuses to open unless the window is larger than twice the attempt envelope (a write reserves its attempt and the read that settles it); a read-only disk, unless it is larger than the envelope. On a writable disk every attempt also needs room left in the mount lease, so while renewals fail a request stops when the lease runs out, even inside a longer window. The ref-table recovery retry budget is derived as twice this value: 240 s by default, so a recovery attempt that spends one whole window inside a single write still has room for a second try. `90000` gives a 90 s window and a 180 s recovery budget |
| `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: the attempt envelope + `cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, and `cas_mount_renew_period_ms` + 2 × envelope + `cas_lease_safety_margin_ms` too, or the disk refuses to open writable |
Expand Down
21 changes: 19 additions & 2 deletions docs/en/operations/system-tables/cas_gc_log.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ The phases, in execution order:
| `fold_ref_intake` | Read and fold every new ref log and the manifest bodies its edges name. | one `GET` per new log, one `GET` per manifest edge |
| `fold_reduce` | The per-shard in-degree merge: condemn, spare, graduate. | prior-run streaming `GET`s, one `HEAD` per zero-transition candidate, run `PUT`s |
| `fold_seal_write` | Publish the new fold seal. | one `PUT` |
| `pending_deletes` | The single content-delete site: exact-token deletes of previously published `delete_pending` entries, plus the outcome logs. `phase_metrics` carries `jobs_scheduled` (entries sent to the GC I/O pool; `0` for no candidates, sequential execution, singleton batches, or refusal before the first submission) and `jobs_failed`. | one `HEAD` and at most one `DELETE` per entry, one outcome-log `PUT` per shard with rows |
| `pending_deletes` | The single content-delete site: exact-token deletes of previously published `delete_pending` entries, plus the outcome logs. `phase_metrics` carries `read_blob_jobs_scheduled` and `delete_chunk_jobs_scheduled` (read jobs and delete chunk jobs successfully enqueued in the parallel pool), separate execution and enqueue failure counters described below, and `delete_chunks` (logical conditional-delete chunks, including sequential execution). Both scheduling counters are `0` on the sequential path, including waves with at most one job. Unsupported or unprobed storage uses one-key chunks. | one `HEAD` per entry, conditional delete chunks, one outcome-log `PUT` per shard with rows |
| `meta_pool_wait` | Drain the round's per-hash freshness-meta writes. | none on this thread — see the caveat below |
| `round_commit` | The generation-retention prune and the round's single `gc/state` compare-and-swap. | prune `LIST`s and deletes, one compare-and-swap |
| `handoff_reclaim` | Wholesale-reclaim generations a moved run ref stranded below the retention cursor. | prefix `LIST`s and deletes |
Expand Down Expand Up @@ -124,6 +124,12 @@ WHERE round_id = '...' AND event_type = 'Phase'
ORDER BY event_time_microseconds;
```

For `pending_deletes`, the former `jobs_scheduled` key is now `read_blob_jobs_scheduled`, and
`delete_batches` is now `delete_chunks`. The former `jobs_failed` aggregate is replaced by separate
read/delete execution and enqueue failure counters below. Queries over older rows must use the
keys written by that server version. Other phases, including
`meta_pool_wait`, retain their existing keys.

Two caveats when reading these rows:

- Work scheduled onto the GC meta pool runs on other threads, so the `meta_pool_wait` row's
Expand All @@ -132,7 +138,18 @@ Two caveats when reading these rows:
- Requests issued by GC I/O pool workers — the `pending_deletes` fan-out and the fold read-ahead of
`fold_ref_intake` and `fold_reduce` — also run on other threads and are missing from those phase
rows' `ProfileEvents`. They still count in `system.events`. On `pending_deletes`, read
`phase_metrics` `jobs_scheduled` / `jobs_failed` next to `redeleted`.
`phase_metrics` `read_blob_jobs_scheduled` / `delete_chunk_jobs_scheduled` next to `redeleted`.
The two scheduling counters include enqueued jobs that fail during execution; they exclude jobs
whose enqueue failed and jobs executed inline. `read_blob_jobs_failed` and `delete_chunk_jobs_failed`
count jobs that finish with an exception, including inline jobs. A failed multi-blob chunk counts
once. These execution-failure counters can therefore be nonzero when the scheduling counters are
zero; the latter are not error-rate denominators. `read_blob_job_schedule_failures` and
`delete_chunk_job_schedule_failures` count enqueue refusals once each, regardless of the number of
unsubmitted jobs or blobs. Retries do not multiply logical job counts. All four failure counters
are emitted, including zeros and phase exits with an exception. Read errors are reported after
the read wave completes, and delete errors after the delete wave completes. The `Finish` row's
`entries_redelete_failed` and `CASGCRetiredRedeleteFailed` retain the legacy units: failed blobs
from executed jobs plus one per enqueue refusal.
- Phase durations do not sum to the round's `duration_ms`. The round also performs untimed
bookkeeping between phases, and the `Finish` row's `duration_ms` remains the authority on total
round time.
Expand Down
Loading
Loading