Skip to content

Batch conditional CAS GC blob deletes on S3 - #2534

Draft
k-morozov wants to merge 31 commits into
antalya-26.8from
cas_batch_deletes
Draft

k-morozov wants to merge 31 commits into
antalya-26.8from
cas_batch_deletes

Conversation

@k-morozov

@k-morozov k-morozov commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

CAS garbage collection used one S3 DeleteObject request per retired blob body. Group these deletes into bounded DeleteObjects requests and send the observed ETag for each key. The mount probe enables batching after checking that matching objects are removed, a stale token leaves the replacement intact, and an already absent key is reported as Gone. Without batch support, GC uses one-key chunks with the existing I/O concurrency. Freshness .meta cleanup remains per object.

The local RustFS 1.0.6 integration test exercised the real mount probe and compared two isolated CAS disks. Both reclaimed six blob bodies, confirmed by listing RustFS directly. The disk with a chunk limit of two made three DeleteObjects requests, while the one-key control made six delete requests. Live AWS S3 validation is separate.

Upstream PR #70786 shows how ClickHouse previously detected ordinary DeleteObjects support by attempting a batch and interpreting the response.

Closes: #2480
Related: #2351
Related: ClickHouse#70786

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Reduce S3 delete requests during CAS garbage collection by batching retired blob body deletes with a separate ETag condition for each object on supported storage.

Documentation entry for user-facing changes

CAS garbage collection: pending deletes

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

@k-morozov k-morozov linked an issue Oct 9, 2026 that may be closed by this pull request
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [487730e]

@k-morozov

Copy link
Copy Markdown
Collaborator Author

rustfs/rustfs#8518

@filimonov

filimonov commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

I think we will need to add the feature test on disk start (add to other checks, but set the flag / show info instead of stop)

@k-morozov k-morozov mentioned this pull request Oct 28, 2026
1 of 74 tasks
@k-morozov

Copy link
Copy Markdown
Collaborator Author

#2876

@k-morozov
k-morozov changed the base branch from antalya-26.6 to antalya-26.8 November 17, 2026 16:58
k-morozov added a commit that referenced this pull request Nov 17, 2026
Resolve the CAS GC conflicts while preserving the antalya-26.8 marker-round semantics.

Related: #2534
k-morozov added a commit that referenced this pull request Nov 17, 2026
Bring the pull request up to the current release-branch tip after retargeting.

Related: #2534
k-morozov and others added 16 commits November 17, 2026 18:01
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Replace the parallel `io_results`, `io_errors` and `delete_submitted`
vectors in `Gc::redeleteBlobs` with one `RedeleteBlobState` per entry.
The observation result (`live_token`) is now separate from the delete
result (`del`), so `performRedeleteIo` becomes `observeRetiredBlob`.

Rename `applyRedeleteOutcome` to `recordRedeleteSuccess` and
`reportRedeleteFailure` to `recordRedeleteFailure`: the delete has
already run when they are called, and `Outcome` already names the
persisted `GcOutcomes` log. Rename the test hook and test names to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Move the selection of token-matched entries and their split into
conditional delete chunks out of `Gc::redeleteBlobs` into
`Gc::prepareConditionalDeleteChunks`. The second wave now only submits
the prepared chunks, so `matched`, `delete_begin` and `delete_indices`
are gone.

The function and its value types are public so that the new
`CASGCConditionalDeleteChunks` tests can call it without a pool or a
backend request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Move the conditional delete chunk size calculation out of
`Gc::redeleteBlobs` into a method next to `Gc::bulkDeleteChunkKeys`.
No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Add `Gc::runRedeleteIoJobs` and use it for both the observation wave
and the conditional delete wave in `Gc::redeleteBlobs`. It runs jobs
inline when `concurrency <= 1` or there is at most one job, and on the
GC I/O pool otherwise. It returns per-job errors and the enqueue
refusal.

`jobs_scheduled` still counts only observation jobs submitted to the
pool. The delete wave now also runs inline at `concurrency <= 1`, and
both waves use one injected enqueue refusal message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Split the tail of `Gc::redeleteBlobs`:
- `Gc::recordRedeleteResults` records failures and successes and returns
  the error with the lowest entry index;
- `Gc::scheduleBlobMetaDeletes` schedules `.meta` deletion for `Removed`
  and `Gone` blobs;
- `Gc::redeleteBlobs` rethrows the error after both steps.

`Gc::recordRedeleteSuccess` no longer schedules `.meta` deletion, so a
later batch `.meta` delete changes only `Gc::scheduleBlobMetaDeletes`.
If recording throws, no `.meta` deletion is scheduled for the round; the
next round records the blobs as `Absent` and schedules it then.

Rename `RedeleteBlobState::del` to `removal` (it can come from the
observation wave, not only from a delete) and the wave results to
`observe_jobs` and `delete_jobs`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
k-morozov and others added 10 commits November 17, 2026 22:01
Preserve conditional batch deletes alongside the new GC outcome accounting and request-window policy. Bind conditional batch retries through `CasRequests::bindPolicy` to match the updated retry API.

Related: #2534
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
The conditional batch-delete path still calls `setClickhouseAttemptNumber`,
which was renamed to `setClickHouseAttemptNumber` in 26.8. Use the current
helper so `S3ObjectStorage` compiles against its declaration.

Related: #2534
CI: https://altinity-build-artifacts.s3.amazonaws.com/PRs/2534/ad2f86b097dcd86fd01a8cffeccb0f4e7ce52cfa/build_amd_debug/build_clickhouse/build_clickhouse.log

Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Name blob-state reads, chunk jobs, removal results and scheduling failures
explicitly so per-blob state is distinct from per-chunk work. Build each
`ConditionalDeleteChunk` locally and track an unfinished chunk with
`std::optional` before moving it into the result.

Add `const` to immutable value parameters and update the affected test
helpers and references.

Validation: `git diff --check` and static reference checks.
Build and tests were not run for this change.

Related: #2534

Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Report blob-read failures before starting eligible deletes, and report
chunk failures after all submitted deletes finish. Distinguish read,
delete, and enqueue failures while preserving per-blob accounting and
the original exception with the lowest input blob index.

Cover mixed wave failures, warning order, and refused-enqueue tails in
sequential and parallel execution.

Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
The pending-deletes `jobs_failed` aggregate mixed failed blobs with
enqueue refusals, so a failed multi-blob chunk looked like multiple job
failures. Count failed read jobs, failed delete chunk jobs, and enqueue
refusals for each wave separately. Include inline execution failures;
keep legacy report and profile-event accounting unchanged.

Use `read_blob_jobs_scheduled` and `delete_chunk_jobs_scheduled` for
successful pool submissions, and `delete_chunks` for logical chunks
including inline execution. Update tests, metric consumers, and docs.

Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
`LOGICAL_ERROR` aborts in debug and sanitizer builds before
`EXPECT_THROW` can handle it, terminating the unit-test process.
Use `EXPECT_DEATH` for oversized conditional batches and duplicate
S3 response keys in those builds; retain exception checks in release.
Construct the scripted HTTP server inside the death-test child.

CI log:
https://altinity-build-artifacts.s3.amazonaws.com/PRs/2534/d05e9bdde03c9e602fe485f8c33e46a63d287854/unit_tests_asan_ubsan/job.log

Related: #2534

Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Exercise two matching tokens, a stale token, and an already removed key
in one request. Require `Gone` for the absent key, check that both
matching objects are gone before cleanup, and verify that the stale
request left the replacement intact. Cover false success and an
incorrect outcome for the absent key with regression tests.

Share the request implementation through a private
`removeManyByConditional` overload with an explicit limit, and avoid
copying response keys during S3 validation.

Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Move the optional batch check into `probeConditionalBatchDelete`, where
it owns its temporary objects and checks their state before cleanup.
Pass the key and token from the required single-object check to exercise
the already absent case in the same batch.

Define chunk, batch, and bulk delete in the GC documentation. Explain
that one-key chunks still use the configured I/O concurrency.

Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Keep the seven required checks and their outer cleanup flow in
`runCapabilityProbe`. Call the extracted `probeConditionalBatchDelete`
from the same point in that flow so the refactor remains scoped to the
optional batch check.

Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Run identical drop workloads on two isolated CAS disks with chunk limits
of one and two. Let the mount probe enable conditional batching, verify
logical chunk counts and S3 delete requests per round, and list RustFS to
confirm that both workloads reclaim all blob bodies.

Verified locally with RustFS 1.0.6: six bodies required six requests with
limit one and three requests with limit two.

Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
@k-morozov k-morozov changed the title CAS gc with conditional batch deletes Batch conditional CAS GC blob deletes on S3 Nov 18, 2026
k-morozov and others added 5 commits November 18, 2026 18:15
Use the configured key limit after the CAS mount probe confirms
conditional batching. Keep the ordinary DeleteObjects capability
independent of conditional request results in both directions.

Classify only an explicit request-level NotImplemented response as
unsupported for conditional deletes. Preserve InvalidRequest and
InvalidArgument as S3 errors. Cover capability independence and a
subsequent multi-key request after a runtime refusal.

Verified with 71 focused unit tests and the RustFS 1.0.6 CAS integration
test.

Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Exercise a lost response after partial deletion and a concurrent blob replacement. Check that retries retain the original tokens and that an exhausted attempt leaves the published pending entries available to the next GC round.

Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
The Google Test simple regex matcher rejects alternation. Check the
duplicate key, wrong key, and foreign backend separately, including the
keys and backend IDs in the precondition diagnostics. Enable abort on
logical errors in the death-test child so the matcher also runs in release
builds, route its fatal diagnostic to stderr, and keep checking the exception in the release parent.

https://github.com/Altinity/ClickHouse/actions/runs/44406205665/job/135198595553

Co-authored-by: Codex (GPT-6) <noreply@openai.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CAS: batches for pending_deletes

3 participants