Repository navigation
Conversation
Collaborator
Author
Member
|
I think we will need to add the feature test on disk start (add to other checks, but set the flag / show info instead of stop) |
Collaborator
Author
k-morozov
added a commit
that referenced
this pull request
Nov 17, 2026
Resolve the CAS GC conflicts while preserving the antalya-26.8 marker-round semantics. Related: #2534
k-morozov
added a commit
that referenced
this pull request
Nov 17, 2026
Bring the pull request up to the current release-branch tip after retargeting. Related: #2534
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Replace the parallel `io_results`, `io_errors` and `delete_submitted` vectors in `Gc::redeleteBlobs` with one `RedeleteBlobState` per entry. The observation result (`live_token`) is now separate from the delete result (`del`), so `performRedeleteIo` becomes `observeRetiredBlob`. Rename `applyRedeleteOutcome` to `recordRedeleteSuccess` and `reportRedeleteFailure` to `recordRedeleteFailure`: the delete has already run when they are called, and `Outcome` already names the persisted `GcOutcomes` log. Rename the test hook and test names to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Move the selection of token-matched entries and their split into conditional delete chunks out of `Gc::redeleteBlobs` into `Gc::prepareConditionalDeleteChunks`. The second wave now only submits the prepared chunks, so `matched`, `delete_begin` and `delete_indices` are gone. The function and its value types are public so that the new `CASGCConditionalDeleteChunks` tests can call it without a pool or a backend request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Move the conditional delete chunk size calculation out of `Gc::redeleteBlobs` into a method next to `Gc::bulkDeleteChunkKeys`. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Add `Gc::runRedeleteIoJobs` and use it for both the observation wave and the conditional delete wave in `Gc::redeleteBlobs`. It runs jobs inline when `concurrency <= 1` or there is at most one job, and on the GC I/O pool otherwise. It returns per-job errors and the enqueue refusal. `jobs_scheduled` still counts only observation jobs submitted to the pool. The delete wave now also runs inline at `concurrency <= 1`, and both waves use one injected enqueue refusal message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Split the tail of `Gc::redeleteBlobs`: - `Gc::recordRedeleteResults` records failures and successes and returns the error with the lowest entry index; - `Gc::scheduleBlobMetaDeletes` schedules `.meta` deletion for `Removed` and `Gone` blobs; - `Gc::redeleteBlobs` rethrows the error after both steps. `Gc::recordRedeleteSuccess` no longer schedules `.meta` deletion, so a later batch `.meta` delete changes only `Gc::scheduleBlobMetaDeletes`. If recording throws, no `.meta` deletion is scheduled for the round; the next round records the blobs as `Absent` and schedules it then. Rename `RedeleteBlobState::del` to `removal` (it can come from the observation wave, not only from a delete) and the wave results to `observe_jobs` and `delete_jobs`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
k-morozov
force-pushed
the
cas_batch_deletes
branch
from
November 17, 2026 17:03
897bafe to
8b0a9b8
Compare
Preserve conditional batch deletes alongside the new GC outcome accounting and request-window policy. Bind conditional batch retries through `CasRequests::bindPolicy` to match the updated retry API. Related: #2534 Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
The conditional batch-delete path still calls `setClickhouseAttemptNumber`, which was renamed to `setClickHouseAttemptNumber` in 26.8. Use the current helper so `S3ObjectStorage` compiles against its declaration. Related: #2534 CI: https://altinity-build-artifacts.s3.amazonaws.com/PRs/2534/ad2f86b097dcd86fd01a8cffeccb0f4e7ce52cfa/build_amd_debug/build_clickhouse/build_clickhouse.log Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Name blob-state reads, chunk jobs, removal results and scheduling failures explicitly so per-blob state is distinct from per-chunk work. Build each `ConditionalDeleteChunk` locally and track an unfinished chunk with `std::optional` before moving it into the result. Add `const` to immutable value parameters and update the affected test helpers and references. Validation: `git diff --check` and static reference checks. Build and tests were not run for this change. Related: #2534 Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Report blob-read failures before starting eligible deletes, and report chunk failures after all submitted deletes finish. Distinguish read, delete, and enqueue failures while preserving per-blob accounting and the original exception with the lowest input blob index. Cover mixed wave failures, warning order, and refused-enqueue tails in sequential and parallel execution. Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
The pending-deletes `jobs_failed` aggregate mixed failed blobs with enqueue refusals, so a failed multi-blob chunk looked like multiple job failures. Count failed read jobs, failed delete chunk jobs, and enqueue refusals for each wave separately. Include inline execution failures; keep legacy report and profile-event accounting unchanged. Use `read_blob_jobs_scheduled` and `delete_chunk_jobs_scheduled` for successful pool submissions, and `delete_chunks` for logical chunks including inline execution. Update tests, metric consumers, and docs. Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
`LOGICAL_ERROR` aborts in debug and sanitizer builds before `EXPECT_THROW` can handle it, terminating the unit-test process. Use `EXPECT_DEATH` for oversized conditional batches and duplicate S3 response keys in those builds; retain exception checks in release. Construct the scripted HTTP server inside the death-test child. CI log: https://altinity-build-artifacts.s3.amazonaws.com/PRs/2534/d05e9bdde03c9e602fe485f8c33e46a63d287854/unit_tests_asan_ubsan/job.log Related: #2534 Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Exercise two matching tokens, a stale token, and an already removed key in one request. Require `Gone` for the absent key, check that both matching objects are gone before cleanup, and verify that the stale request left the replacement intact. Cover false success and an incorrect outcome for the absent key with regression tests. Share the request implementation through a private `removeManyByConditional` overload with an explicit limit, and avoid copying response keys during S3 validation. Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Move the optional batch check into `probeConditionalBatchDelete`, where it owns its temporary objects and checks their state before cleanup. Pass the key and token from the required single-object check to exercise the already absent case in the same batch. Define chunk, batch, and bulk delete in the GC documentation. Explain that one-key chunks still use the configured I/O concurrency. Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Keep the seven required checks and their outer cleanup flow in `runCapabilityProbe`. Call the extracted `probeConditionalBatchDelete` from the same point in that flow so the refactor remains scoped to the optional batch check. Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Run identical drop workloads on two isolated CAS disks with chunk limits of one and two. Let the mount probe enable conditional batching, verify logical chunk counts and S3 delete requests per round, and list RustFS to confirm that both workloads reclaim all blob bodies. Verified locally with RustFS 1.0.6: six bodies required six requests with limit one and three requests with limit two. Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Use the configured key limit after the CAS mount probe confirms conditional batching. Keep the ordinary DeleteObjects capability independent of conditional request results in both directions. Classify only an explicit request-level NotImplemented response as unsupported for conditional deletes. Preserve InvalidRequest and InvalidArgument as S3 errors. Cover capability independence and a subsequent multi-key request after a runtime refusal. Verified with 71 focused unit tests and the RustFS 1.0.6 CAS integration test. Co-authored-by: Codex (GPT-6.2 Sol) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
Exercise a lost response after partial deletion and a concurrent blob replacement. Check that retries retain the original tokens and that an exhausted attempt leaves the published pending entries available to the next GC round. Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
The Google Test simple regex matcher rejects alternation. Check the duplicate key, wrong key, and foreign backend separately, including the keys and backend IDs in the precondition diagnostics. Enable abort on logical errors in the death-test child so the matcher also runs in release builds, route its fatal diagnostic to stderr, and keep checking the exception in the release parent. https://github.com/Altinity/ClickHouse/actions/runs/44406205665/job/135198595553 Co-authored-by: Codex (GPT-6) <noreply@openai.com> Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CAS garbage collection used one S3
DeleteObjectrequest per retired blob body. Group these deletes into boundedDeleteObjectsrequests and send the observedETagfor each key. The mount probe enables batching after checking that matching objects are removed, a stale token leaves the replacement intact, and an already absent key is reported asGone. Without batch support, GC uses one-key chunks with the existing I/O concurrency. Freshness.metacleanup remains per object.The local RustFS
1.0.6integration test exercised the real mount probe and compared two isolated CAS disks. Both reclaimed six blob bodies, confirmed by listing RustFS directly. The disk with a chunk limit of two made threeDeleteObjectsrequests, while the one-key control made six delete requests. Live AWS S3 validation is separate.Upstream PR #70786 shows how ClickHouse previously detected ordinary
DeleteObjectssupport by attempting a batch and interpreting the response.Closes: #2480
Related: #2351
Related: ClickHouse#70786
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Reduce S3 delete requests during CAS garbage collection by batching retired blob body deletes with a separate
ETagcondition for each object on supported storage.Documentation entry for user-facing changes
CAS garbage collection: pending deletes
CI/CD Options
Exclude tests:
Regression jobs to run: