diff --git a/docs/en/antalya/cas/architecture/garbage-collection.md b/docs/en/antalya/cas/architecture/garbage-collection.md index d429280da6fd1..5ed69c17049e2 100644 --- a/docs/en/antalya/cas/architecture/garbage-collection.md +++ b/docs/en/antalya/cas/architecture/garbage-collection.md @@ -52,7 +52,7 @@ follower or a deferred round execution returns before that commit. | 8 | `fold_ref_intake` | fold | `GET` every new ref-log record and every referenced manifest, extracting blob source edges | | 9 | `fold_reduce` | fold | Merge prior edges, new deltas and the parent's condemned rows: spare, condemn, graduate or redelete each candidate; compute `suppress_destructive` | | 10 | `fold_seal_write` | fold | Write the new fold seal once, write-once deterministic, adopting a byte-identical replay instead of rewriting it | -| 11 | `pending_deletes` | fold | The single blob-body delete site: exact-token delete of entries a *previous* round marked `delete_pending`, up to the redelete budget, plus the outcome-log writes | +| 11 | `pending_deletes` | fold | The single blob-body delete site: `HEAD` each entry a *previous* round marked `delete_pending`, then exact-token conditional delete chunks where the mounted backend proved that capability, plus the outcome-log writes | | 12 | `meta_pool_wait` | fold | Drain the bounded pool of async `.meta` condemn-marker writes queued during the fold | | 13 | `round_commit` | fold | Retention-prune old generations, then publish the single `gc/state` `CAS` that adopts the whole round | | 14 | `handoff_reclaim` | post-`CAS` | Reclaim a generation a ref moved off during this round, which the ordinary retention prune already skipped and will not revisit | @@ -415,12 +415,22 @@ deletes for entries a *previous* round published as `delete_pending`, processing `cas_gc_round_redelete_budget` entries per round (the excess is carried unchanged), and writes the forensic outcome logs. +In the GC code and metrics, a **chunk** is the group of keys handled by one delete job. A chunk can +contain a single key. A **batch** is one storage request that carries multiple keys, such as S3 +`DeleteObjects`. The mount probe checks whether the backend supports conditional batches, and GC +uses one-key chunks when that capability is unavailable. Those jobs can still run concurrently +according to `cas_gc_io_concurrency`. **Bulk delete** names the overall feature, including the +sequence of requests needed to process many keys. + - **Runs on:** fold path only - **Reads:** one `HEAD` per `redelete` entry — the persisted condemned token cannot itself be a precondition, so the round observes the blob first, which also settles the absent case without spending a conditional delete -- **Writes / deletes:** a `DELETE` conditional on the observed etag for each `redelete` entry whose - live etag matches the condemned token; one write-once outcome log per shard with settled entries +- **Writes / deletes:** conditional delete chunks for entries whose live etag matches the condemned + token. A chunk is bounded by `cas_gc_bulk_delete_chunk_keys`, the backend conditional-batch limit, + and 1000 keys. An unprobed, unsupported, or `skip_access_check` mount uses one-key chunks. `.meta` + cleanup remains one per settled blob; it is not part of the conditional delete request. One + write-once outcome log is written per shard with settled entries - **Safety:** an entry is deletable only because a previously committed fold seal published it `delete_pending` — durable state from an earlier commit, safe at any leader staleness. Exact token means a stale leader cannot delete a fresh incarnation; `NotFound` and `TokenMismatch` are @@ -428,7 +438,47 @@ forensic outcome logs. - **Fails the round if:** a backend delete marker appears in a response — object versioning on a mis-provisioned pool (`LOGICAL_ERROR`) - **Observability:** phase row `pending_deletes`; metrics `deleted`, `absent`, `redeleted`, - `graduated`, `replaced`, `spared`, `outcome_logs_written`, `outcome_rows_dropped` + `graduated`, `replaced`, `spared`, `outcome_logs_written`, `outcome_rows_dropped`, `delete_chunks`. `delete_chunks` + counts logical conditional-delete chunks, including sequential execution. `read_blob_jobs_scheduled` + and `delete_chunk_jobs_scheduled` count successfully enqueued read jobs and delete chunk jobs, + respectively. Jobs that later fail still count; enqueue failures do not. Both counters are zero + on the sequential path, including a wave with only one job. `DiskS3DeleteObjects` counts physical + S3 delete attempts, including request retries + +GC reports read-job failures after all submitted blob-state reads finish, before starting the +conditional-delete wave. It reports delete-job failures after all submitted delete chunks finish. +Warnings distinguish read jobs, delete chunk jobs, and failures to enqueue either kind of job. +A failed delete job can have an unknown or partial storage outcome; its warning does not imply that +all objects in the chunk remain present. + +Read failures do not prevent deletion of successfully read siblings. After both waves, GC records +known outcomes and schedules eligible `.meta` cleanup, then rethrows the original job or enqueue +exception with the lowest input blob index. An unsuccessful round leaves the previously published +pending entries available to the next round. + +The `read_blob_jobs_failed` and `delete_chunk_jobs_failed` metrics count jobs that finish with an +exception, including inline execution. A failed delete chunk contributes one job failure, regardless +of its number of blobs. The `read_blob_job_schedule_failures` and `delete_chunk_job_schedule_failures` +metrics each count enqueue refusals, not the number of unsubmitted jobs or blobs. Retries do not +create extra logical jobs; `Gone` and `Mismatch` are normal outcomes, not job failures. These four +metrics replace the former `jobs_failed` aggregate and are emitted even when the phase throws. +Because inline jobs are included only in the execution-failure counters, the scheduling counters +cannot be used as their denominators for an error rate. + +The `RoundReport::redelete_failed` field and `CASGCRetiredRedeleteFailed` event retain their existing +units: each failed blob from an executed job counts once, while an enqueue failure counts once per +wave regardless of the number of unsubmitted blobs reported in its warning. + +Ordinary S3 batch deletion and conditional batch deletion have independent capability flags. Only +the latter requires the store to enforce a separate `ETag` for each key in `DeleteObjects`. Both +paths use `objects_chunk_size_to_delete` as a size limit. A request-level `NotImplemented` from +the optional batch check leaves GC using one-key chunks. A request-level `InvalidRequest` or +`InvalidArgument` propagates and fails the mount. After a successful probe, a batch request error +follows the normal request retry policy and does not change the pinned capability. + +The integration test on RustFS 1.0.6 mounts two disks with GC chunk limits of one and two. It +checks the number of storage requests in each pending-delete phase and lists the bucket to verify +that the blob bodies have disappeared. Under `suppress_destructive` the `redelete` set is empty by construction, so nothing is deleted, and there is no graduation either (would-be graduates are carried unchanged); sparing and superseding @@ -849,7 +899,7 @@ pool sizes and the chunk size reject `0`: | `cas_manifest_sweep_delete_budget_keys` | 100 | orphan-manifest sweep `DELETE` budget per round (phases 9, 18) | | `cas_gc_round_sweep_namespace_budget` | 20 | namespaces whose protection view the sweep may build per page (phase 9) | | `cas_gc_round_sweep_recovery_op_budget` | 5000 | committed-tail ref-log reads the sweep's recovery walk may spend (phase 9) | -| `cas_gc_bulk_delete_chunk_keys` | 1000 | keys per batch `DELETE` request for write-once families (phases 15, 17); `1` to `1000` | +| `cas_gc_bulk_delete_chunk_keys` | 1000 | keys per batch `DELETE` request for blob redelete and write-once families (phases 11, 15, 17); `1` to `1000` | | `cas_gc_meta_pool_size` | 16 | bounded pool for condemn-marker writes (phase 12) | | `cas_gc_io_concurrency` | 16 | bounded pool for the fold's read-ahead of checkpoints, ref logs, manifest bodies and zero-candidate `HEAD`s (phases 8, 9), the graduation `.meta` reads (phase 9), the orphan-sweep planning reads (phase 9), the rebuild read-ahead and the `pending_deletes` `HEAD` + conditional `DELETE` fan-out (phase 11); other GC requests run on the round thread; `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead | diff --git a/docs/en/antalya/cas/bucket-requirements.md b/docs/en/antalya/cas/bucket-requirements.md index 0ea10e8370d64..99ef36a0510c7 100644 --- a/docs/en/antalya/cas/bucket-requirements.md +++ b/docs/en/antalya/cas/bucket-requirements.md @@ -142,7 +142,7 @@ establish acceptance of Google's multipart, native-copy, and exact-delete wire b | AWS S3 | ✓ | Native `ETag`-based conditional dialect for mutable objects and exact deletion; blob publication is unconditional | | Google Cloud Storage | implementation complete; release gate pending | Generation-token dialect for mutable objects/native-token `HEAD`/exact deletion, opted into via `http_client = gcs_hmac` or `gcp_oauth`; blob publication uses ordinary copy/multipart. Real credentialed GCS groups have not run yet | | Azure Blob Storage | probably | Azure's REST API documents the equivalent conditional headers, but ClickHouse's Azure object-storage backend does not yet wire up a `CAS` conditional dialect the way the S3 and GCS paths do — untested, not validated by the capability probe | -| Other S3-compatible stores | only with enforced conditional operations | The capability probe is the actual gate: a store that silently ignores `If-None-Match`/`If-Match` (accepting and applying the write regardless) fails the probe and is refused. `RustFS` passes the full battery and is used as the project's test backend; `Garage` was evaluated and rejected because it silently ignores conditional operations | +| Other S3-compatible stores | only with enforced conditional operations | A store that silently ignores required `If-None-Match` or `If-Match` conditions fails the mount probe. RustFS 1.0.6 passes both the required single-key checks and the optional per-object `ETag` check for `DeleteObjects`. Garage was evaluated and rejected because it silently ignores conditional operations. | The full mechanics of the two dialects — how the backend detects which one a given endpoint speaks, what the capability probe actually checks, and how exact-token deletes map onto each provider's diff --git a/docs/en/antalya/cas/configuration.md b/docs/en/antalya/cas/configuration.md index 326777aa1f40a..0a7dd46b47e46 100644 --- a/docs/en/antalya/cas/configuration.md +++ b/docs/en/antalya/cas/configuration.md @@ -107,7 +107,7 @@ entirely before release. Treat this table as a snapshot of the current build, no | `cas_part_folder_cache_max_entry_bytes` | 16 MiB | Oversized part-folder views bypass retention above this size | | `cas_manifest_decode_cache_bytes` | 128 MiB | Manifest decode cache byte budget (`0` disables) | | `cas_gc_meta_pool_size` | `16` | Bounded pool size for GC per-hash freshness-meta writes | -| `cas_gc_io_concurrency` | `16` | Bounded pool size for GC object-storage requests that run in parallel: the fold's read-ahead (checkpoints, ref logs, manifests, zero-candidate HEADs), the orphan-manifest sweep planning reads, the `SYSTEM CAS GC REBUILD` read-ahead, the `pending_deletes` blob `HEAD` + conditional `DELETE` fan-out, and the namespace janitor's batch deletes of dead ref logs and snapshots. Not covered: meta writes (`cas_gc_meta_pool_size`) and all other GC requests, which run on the round thread. `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead | +| `cas_gc_io_concurrency` | `16` | Bounded pool size for GC object-storage requests that run in parallel: the fold's read-ahead (checkpoints, ref logs, manifests, zero-candidate HEADs), the orphan-manifest sweep planning reads, the `SYSTEM CAS GC REBUILD` read-ahead, the `pending_deletes` blob `HEAD` + conditional `DELETE` chunks, and the namespace janitor's batch deletes of dead ref logs and snapshots. S3 multi-key conditional chunks are enabled only after the writable mount proves their per-object ETag semantics; unsupported storage and `skip_access_check` use one-key conditional deletes. Not covered: meta writes (`cas_gc_meta_pool_size`) and all other GC requests, which run on the round thread. `1` runs the covered requests sequentially. `cas_gc_read_concurrency` is rejected without an alias; use `cas_gc_io_concurrency` instead | | `cas_attempt_timeout_ms` | `5000` | Budget for one HTTP attempt of a writable Native mount's control-plane requests (read, head, list, remove, conditional write, the mount-time bucket-versioning probe), at least 1. Together with the connect cap it forms the attempt envelope (`cas_attempt_timeout_ms + 2 × cap`; the cap is `cas_attempt_timeout_ms` itself when the disk's `connect_timeout_ms` is `0`, else `min(connect_timeout_ms, cas_attempt_timeout_ms)`) that the lease arithmetic reserves: one TCP connect and one TLS handshake under the cap each, send/receive bounded per socket operation by `cas_attempt_timeout_ms`. With background renewal the cadence check requires `cas_mount_renew_period_ms + 2 × envelope + cas_lease_safety_margin_ms < cas_mount_lease_ttl_ms`, which puts an effective ceiling on the frozen connect cap: under the defaults (TTL 40000, period 10000, margin 2000) the envelope must stay under 14000, so with the default attempt timeout a disk `connect_timeout_ms` of `0` or of 4500 ms or more refuses to open writable — lower the connect timeout or raise the TTL if you hit this. Blob body publication does not use this budget: its object-creating request is sent once under the disk's `request_timeout_ms`, with the connect timeout under the same cap | | `cas_request_window_ms` | `120000` | How long a failing request keeps retrying before it gives up, measured from the start of the call. It applies to control-plane requests and to the blob publication loop. At the limit the request gives up with a retryable `NETWORK_ERROR`. A writable disk refuses to open unless the window is larger than twice the attempt envelope (a write reserves its attempt and the read that settles it); a read-only disk, unless it is larger than the envelope. On a writable disk every attempt also needs room left in the mount lease, so while renewals fail a request stops when the lease runs out, even inside a longer window. The ref-table recovery retry budget is derived as twice this value: 240 s by default, so a recovery attempt that spends one whole window inside a single write still has room for a second try. `90000` gives a 90 s window and a 180 s recovery budget | | `cas_lease_safety_margin_ms` | `2000` | Startup-only margin validated against the mount lease TTL: the attempt envelope + `cas_lease_safety_margin_ms` must be strictly less than the mount lease TTL, and `cas_mount_renew_period_ms` + 2 × envelope + `cas_lease_safety_margin_ms` too, or the disk refuses to open writable | diff --git a/docs/en/operations/system-tables/cas_gc_log.md b/docs/en/operations/system-tables/cas_gc_log.md index 9123f2d92ff3b..dad4001a4672a 100644 --- a/docs/en/operations/system-tables/cas_gc_log.md +++ b/docs/en/operations/system-tables/cas_gc_log.md @@ -81,7 +81,7 @@ The phases, in execution order: | `fold_ref_intake` | Read and fold every new ref log and the manifest bodies its edges name. | one `GET` per new log, one `GET` per manifest edge | | `fold_reduce` | The per-shard in-degree merge: condemn, spare, graduate. | prior-run streaming `GET`s, one `HEAD` per zero-transition candidate, run `PUT`s | | `fold_seal_write` | Publish the new fold seal. | one `PUT` | -| `pending_deletes` | The single content-delete site: exact-token deletes of previously published `delete_pending` entries, plus the outcome logs. `phase_metrics` carries `jobs_scheduled` (entries sent to the GC I/O pool; `0` for no candidates, sequential execution, singleton batches, or refusal before the first submission) and `jobs_failed`. | one `HEAD` and at most one `DELETE` per entry, one outcome-log `PUT` per shard with rows | +| `pending_deletes` | The single content-delete site: exact-token deletes of previously published `delete_pending` entries, plus the outcome logs. `phase_metrics` carries `read_blob_jobs_scheduled` and `delete_chunk_jobs_scheduled` (read jobs and delete chunk jobs successfully enqueued in the parallel pool), separate execution and enqueue failure counters described below, and `delete_chunks` (logical conditional-delete chunks, including sequential execution). Both scheduling counters are `0` on the sequential path, including waves with at most one job. Unsupported or unprobed storage uses one-key chunks. | one `HEAD` per entry, conditional delete chunks, one outcome-log `PUT` per shard with rows | | `meta_pool_wait` | Drain the round's per-hash freshness-meta writes. | none on this thread — see the caveat below | | `round_commit` | The generation-retention prune and the round's single `gc/state` compare-and-swap. | prune `LIST`s and deletes, one compare-and-swap | | `handoff_reclaim` | Wholesale-reclaim generations a moved run ref stranded below the retention cursor. | prefix `LIST`s and deletes | @@ -124,6 +124,12 @@ WHERE round_id = '...' AND event_type = 'Phase' ORDER BY event_time_microseconds; ``` +For `pending_deletes`, the former `jobs_scheduled` key is now `read_blob_jobs_scheduled`, and +`delete_batches` is now `delete_chunks`. The former `jobs_failed` aggregate is replaced by separate +read/delete execution and enqueue failure counters below. Queries over older rows must use the +keys written by that server version. Other phases, including +`meta_pool_wait`, retain their existing keys. + Two caveats when reading these rows: - Work scheduled onto the GC meta pool runs on other threads, so the `meta_pool_wait` row's @@ -132,7 +138,18 @@ Two caveats when reading these rows: - Requests issued by GC I/O pool workers — the `pending_deletes` fan-out and the fold read-ahead of `fold_ref_intake` and `fold_reduce` — also run on other threads and are missing from those phase rows' `ProfileEvents`. They still count in `system.events`. On `pending_deletes`, read - `phase_metrics` `jobs_scheduled` / `jobs_failed` next to `redeleted`. + `phase_metrics` `read_blob_jobs_scheduled` / `delete_chunk_jobs_scheduled` next to `redeleted`. + The two scheduling counters include enqueued jobs that fail during execution; they exclude jobs + whose enqueue failed and jobs executed inline. `read_blob_jobs_failed` and `delete_chunk_jobs_failed` + count jobs that finish with an exception, including inline jobs. A failed multi-blob chunk counts + once. These execution-failure counters can therefore be nonzero when the scheduling counters are + zero; the latter are not error-rate denominators. `read_blob_job_schedule_failures` and + `delete_chunk_job_schedule_failures` count enqueue refusals once each, regardless of the number of + unsubmitted jobs or blobs. Retries do not multiply logical job counts. All four failure counters + are emitted, including zeros and phase exits with an exception. Read errors are reported after + the read wave completes, and delete errors after the delete wave completes. The `Finish` row's + `entries_redelete_failed` and `CASGCRetiredRedeleteFailed` retain the legacy units: failed blobs + from executed jobs plus one per enqueue refusal. - Phase durations do not sum to the round's `duration_ms`. The round also performs untimed bookkeeping between phases, and the `Finish` row's `duration_ms` remains the authority on total round time. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h index eb502f9724c21..2735bb55eb14a 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasBackend.h @@ -20,6 +20,11 @@ #include #include +namespace DB::ErrorCodes +{ + extern const int NOT_IMPLEMENTED; +} + namespace DB::Cas { @@ -169,6 +174,11 @@ class Backend /// `DeleteMarker` is a removal that did NOT reclaim: a versioned bucket archived a noncurrent /// version instead. Distinct from `Removed` because reclaiming the storage is the point. enum class RawRemoval : uint8_t { Removed, Gone, Mismatch, DeleteMarker }; + struct RawConditionalRemove + { + String key; + String expected_token; + }; /// Reads the whole object, or nullopt when the key is absent. virtual std::optional read (const String & key, TransportAccess &) = 0; @@ -201,10 +211,23 @@ class Backend /// absence is success. virtual void removeManyWriteOnce(const std::vector & keys, TransportAccess &) = 0; + virtual std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) + { + if (removals.size() > 1) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Conditional batch removal is not implemented by this backend"); + if (removals.empty()) + return {}; + return {remove(removals.front().key, removals.front().expected_token, access)}; + } + /// The most keys one `removeManyWriteOnce` sends to the storage as one request; at least 1. A decorator /// must forward it: the default of 1 would turn every batch below it into one-key requests. virtual size_t bulkDeleteKeyLimit() const { return 1; } + virtual size_t conditionalBatchDeleteKeyLimit() const { return 1; } + virtual void pinConditionalBatchDeleteSupport(bool) {} + /// Authoritative, cache-bypassing probe of one key -- see `ProbeOutcome`. DEFAULT (used by every /// backend without sharper raw-error evidence, e.g. `InMemoryBackend`): derived from `head`/`read` /// alone, so it can only distinguish `Present` from `KeyAbsent`, and ANY exception from either diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp index b6bf981a40f1a..d996077166732 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.cpp @@ -326,6 +326,20 @@ void InMemoryBackend::removeManyWriteOnce(const std::vector & keys } } +std::vector InMemoryBackend::removeManyByConditional( + const std::vector & removals, TransportAccess &) +{ + for (const RawConditionalRemove & removal : removals) + checkExpectedValue(removal.key, removal.expected_token); + + std::lock_guard lock(mutex_); + std::vector results; + results.reserve(removals.size()); + for (const RawConditionalRemove & removal : removals) + results.push_back(applyDelete(removal.key, removal.expected_token)); + return results; +} + void InMemoryBackend::failNextBulkRemoveWith(std::exception_ptr error) { std::lock_guard lock(mutex_); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h index 9e3100efdf3e9..bd82f639145e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInMemoryBackend.h @@ -51,6 +51,8 @@ class InMemoryBackend : public Backend /// `hold_deletes_` exactly as `remove` does: a held delete is queued and lands only on /// `landPendingDelete`. void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override; + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override; /// The next `removeManyWriteOnce` throws `error` instead of deleting anything; one-shot, like /// `failNextWriteWith`. void failNextBulkRemoveWith(std::exception_ptr error); @@ -62,6 +64,8 @@ class InMemoryBackend : public Backend /// The emulation deletes keys one by one under its lock, so the CAS maximum is the only bound. size_t bulkDeleteKeyLimit() const override { return kBulkDeleteMaxKeys; } + size_t conditionalBatchDeleteKeyLimit() const override { return conditional_batch_delete_supported ? kBulkDeleteMaxKeys : 1; } + void pinConditionalBatchDeleteSupport(const bool supported) override { conditional_batch_delete_supported = supported; } /// Creates the key when `expected_value` is empty, or replaces the incarnation it names. A /// refused precondition leaves the store unchanged. Value enforcement can be disabled with @@ -227,6 +231,7 @@ class InMemoryBackend : public Backend std::vector armed_bulk_remove_failures_; std::function before_bulk_remove_hook_; size_t bulk_remove_calls_ = 0; + std::atomic_bool conditional_batch_delete_supported = false; }; } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp index 4782bd3f10116..90d4b410cee3b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.cpp @@ -154,4 +154,13 @@ void InstrumentedBackend::removeManyWriteOnce(const std::vector & incrementCasEvent(classifyCasNs(key.str()), CasOp::Delete); } +std::vector InstrumentedBackend::removeManyByConditional( + const std::vector & removals, TransportAccess & access) +{ + std::vector results = inner->removeManyByConditional(removals, access); + for (const RawConditionalRemove & removal : removals) + incrementCasEvent(classifyCasNs(removal.key), CasOp::Delete); + return results; +} + } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h index 5d1f3f36e9536..0f21488d1cf32 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasInstrumentedBackend.h @@ -130,6 +130,8 @@ class InstrumentedBackend final : public Backend /// carried, the request counter says how many requests it took. Out of line, like `publish`, so /// the header need not declare the `CASBulkDeleteRequests` extern. void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override; + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override; /// Count a create and a replacement separately, and each of them separately from its refusal: /// they cost the same one request, but a pool whose creates are mostly refused and one whose @@ -162,6 +164,8 @@ class InstrumentedBackend final : public Backend uint64_t attemptTimeoutMs() const override { return inner->attemptTimeoutMs(); } uint64_t attemptEnvelopeMs() const override { return inner->attemptEnvelopeMs(); } size_t bulkDeleteKeyLimit() const override { return inner->bulkDeleteKeyLimit(); } + size_t conditionalBatchDeleteKeyLimit() const override { return inner->conditionalBatchDeleteKeyLimit(); } + void pinConditionalBatchDeleteSupport(const bool supported) override { inner->pinConditionalBatchDeleteSupport(supported); } bool refreshCredentials() override { return inner->refreshCredentials(); } private: diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp index 00e632e7a802c..a26173b4aa421 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.cpp @@ -1040,6 +1040,18 @@ size_t ObjectStorageBackend::bulkDeleteKeyLimit() const return mode == Mode::Native ? object_storage->batchDeleteKeyLimit() : kBulkDeleteMaxKeys; } +size_t ObjectStorageBackend::conditionalBatchDeleteKeyLimit() const +{ + if (mode != Mode::Native || !conditional_batch_delete_supported.load()) + return 1; + return object_storage->configuredBatchDeleteKeyLimit(); +} + +void ObjectStorageBackend::pinConditionalBatchDeleteSupport(const bool supported) +{ + conditional_batch_delete_supported.store(supported); +} + void ObjectStorageBackend::removeManyWriteOnce(const std::vector & keys, TransportAccess & access) { if (keys.empty()) @@ -1067,6 +1079,48 @@ void ObjectStorageBackend::removeManyWriteOnce(const std::vector & } } +std::vector ObjectStorageBackend::removeManyByConditional( + const std::vector & removals, TransportAccess & access) +{ + if (removals.empty()) + return {}; + if (mode != Mode::Native) + { + std::vector results; + results.reserve(removals.size()); + for (const RawConditionalRemove & removal : removals) + results.push_back(removeUnder(removal.key, removal.expected_token, controlRequest(access.attemptNo()))); + return results; + } + + std::vector objects; + objects.reserve(removals.size()); + for (const RawConditionalRemove & removal : removals) + { + if (!isValidTokenValue(dialect(), removal.expected_token)) + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS backend: malformed conditional removal token for {}", removal.key); + objects.push_back({StoredObject(removal.key), removal.expected_token}); + } + const auto storage_results = object_storage->removeObjectsIfTokensMatch(objects, controlRequest(access.attemptNo())); + if (storage_results.size() != removals.size()) + throw Exception(ErrorCodes::LOGICAL_ERROR, "CAS backend: conditional batch removal returned {} results for {} keys", storage_results.size(), removals.size()); + + std::vector results; + results.reserve(storage_results.size()); + for (const ConditionalRemoveResult & result : storage_results) + { + switch (result.outcome) + { + case ConditionalRemoveOutcome::Removed: + results.push_back(result.created_delete_marker ? RawRemoval::DeleteMarker : RawRemoval::Removed); + break; + case ConditionalRemoveOutcome::TokenMismatch: results.push_back(RawRemoval::Mismatch); break; + case ConditionalRemoveOutcome::NotFound: results.push_back(RawRemoval::Gone); break; + } + } + return results; +} + Backend::RawListPage ObjectStorageBackend::list(const String & prefix, const String & cursor, size_t limit, TransportAccess & access) { return listUnder(prefix, cursor, limit, controlRequest(access.attemptNo())); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h index a738a49a63244..405216abd66e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasObjectStorageBackend.h @@ -101,10 +101,14 @@ class ObjectStorageBackend final : public Backend /// under the control-plane profile; `EmulatedSingleProcess` deletes each present key under the /// emulation lock with the same token bookkeeping as the single-key delete. void removeManyWriteOnce(const std::vector & keys, TransportAccess & access) override; + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override; /// Native: the object storage's own limit. Emulated: `kBulkDeleteMaxKeys`, since this backend deletes /// the keys one by one under its emulation lock. size_t bulkDeleteKeyLimit() const override; + size_t conditionalBatchDeleteKeyLimit() const override; + void pinConditionalBatchDeleteSupport(bool supported) override; /// Native mints its store's own dialect (ETag or GCS generation); the emulated adapter mints its /// own values. Dialect dialect() const override { return mode == Mode::Native ? native_token_type : Dialect::Emulated; } @@ -220,6 +224,7 @@ class ObjectStorageBackend final : public Backend private: const ObjectStoragePtr object_storage; const Mode mode; + std::atomic_bool conditional_batch_delete_supported = false; Dialect native_token_type = Dialect::ETag; /// See the constructor: what the READ-class requests (read, head, list, remove) carry. const bool single_attempt_control_plane; diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp index 481910f1d91f7..b9ab87b79f1e9 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.cpp @@ -1,5 +1,8 @@ #include #include +#include + +#include namespace DB { @@ -42,30 +45,69 @@ Removal removeOrReportDeleteMarker(CasOperation & op, const String & key, const } } +/// Best-effort cleanup uses a fresh token and skips keys that are already absent. +void cleanupProbeKeys(CasOperation & op, std::initializer_list keys) noexcept +{ + try + { + for (const String * cleanup_key : keys) + { + const auto current = op.head(*cleanup_key, Retry::standard()); + if (current) + op.remove(*cleanup_key, current->etag, Retry::standard()); + } + } + catch (...) {} /// NOLINT(bugprone-empty-catch) } -void runCapabilityProbe(CasOperation & op, const String & probe_prefix) +} + +bool probeConditionalBatchDelete(CasOperation & op, const String & probe_prefix, const ConditionalRemove & absent_key) +{ + const String first_matching_key = probe_prefix + "/batch-first"; + const String replaced_key = probe_prefix + "/batch-second"; + const String second_matching_key = probe_prefix + "/batch-third"; + SCOPE_EXIT({ cleanupProbeKeys(op, {&first_matching_key, &replaced_key, &second_matching_key}); }); + + const Etag first_matching_token = std::get(op.create(first_matching_key, "probe-batch-first", Retry::standard())).etag; + const Etag stale_token = std::get(op.create(replaced_key, "probe-batch-second-v1", Retry::standard())).etag; + const String replacement_bytes = "probe-batch-second-v2"; + (void)std::get(op.replace(replaced_key, replacement_bytes, stale_token, Retry::standard())); + const Etag second_matching_token = std::get(op.create(second_matching_key, "probe-batch-third", Retry::standard())).etag; + try + { + const auto removals = op.removeManyByConditional( + {{first_matching_key, first_matching_token}, {replaced_key, stale_token}, + {second_matching_key, second_matching_token}, absent_key}, + Retry::standard(), kBulkDeleteMaxKeys); + if (removals.size() != 4 || removals[0] != Removal::Removed || removals[1] != Removal::Mismatch + || removals[2] != Removal::Removed || removals[3] != Removal::Gone) + return false; + + /// Check before cleanup, which would otherwise hide a successful response without deletion. + const auto first = op.head(first_matching_key, Retry::standard()); + const auto replacement = op.read(replaced_key, Retry::standard()); + const auto second = op.head(second_matching_key, Retry::standard()); + return !first && !second && replacement && replacement->bytes == replacement_bytes; + } + catch (const DB::Exception & e) + { + if (e.code() != DB::ErrorCodes::NOT_IMPLEMENTED) + throw; + return false; + } +} + +CapabilityProbeResult runCapabilityProbe(CasOperation & op, const String & probe_prefix) { // Sub-directory style ("probe_prefix/token") ensures that list(probe_prefix, …) works for both the // in-memory backend (prefix match) and the LocalObjectStorage backend (directory listing). const String key = probe_prefix + "/token"; // Best-effort cleanup — runs at function exit regardless of outcome. - auto cleanup = [&]() noexcept - { - // Skip the remove when HEAD says the key is already gone (the happy path: the battery's own - // delete already ran). `Etag` can only be minted from an actual HEAD/read observation, so - // an unconditional "delete with whatever precondition" this backend never saw is not - // constructible here — the gate below is the only way to reach `remove` at all. - try - { - const auto h = op.head(key, Retry::standard()); - if (h) - op.remove(key, h->etag, Retry::standard()); - } - catch (...) {} /// NOLINT(bugprone-empty-catch) - }; + auto cleanup = [&]() noexcept { cleanupProbeKeys(op, {&key}); }; + CapabilityProbeResult result; try { // ---- Step 1: create fresh -> Committed; read-after-write returns the bytes. ---- @@ -195,6 +237,9 @@ void runCapabilityProbe(CasOperation & op, const String & probe_prefix) "CasProbe: list-after-delete failed — the deleted probe key '{}' is still visible in the listing under prefix '{}'", key, probe_prefix); } + + /// Step 7 verified that `key` was gone, so reuse its token for the absent entry. + result.conditional_batch_delete_supported = probeConditionalBatchDelete(op, probe_prefix, {key, t2}); } catch (...) { @@ -206,6 +251,7 @@ void runCapabilityProbe(CasOperation & op, const String & probe_prefix) // Normal-exit cleanup (the key was already deleted in step 7). Call cleanup anyway to handle any // partial state edge cases — it is a no-op if the key is gone. cleanup(); + return result; } } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h index 99ec36581af29..783261ddbcd27 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasProbe.h @@ -5,6 +5,11 @@ namespace DB::Cas { +struct CapabilityProbeResult +{ + bool conditional_batch_delete_supported = false; +}; + /// Run the capability battery against `op`, using throwaway keys under `probe_prefix`. /// /// `op` must already be admitted. The two store-level precondition hooks — @@ -31,6 +36,6 @@ namespace DB::Cas /// Cleanup of the probe key is best-effort and runs unconditionally: after the battery completes, or on /// the failure path immediately before the check-failing exception is rethrown. Cleanup itself suppresses /// exceptions so that it cannot hide the capability-check failure. -void runCapabilityProbe(CasOperation & op, const String & probe_prefix); +CapabilityProbeResult runCapabilityProbe(CasOperation & op, const String & probe_prefix); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp index e541135fbd7ae..01c7786ccad9b 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.cpp @@ -19,9 +19,11 @@ #include +#include #include #include #include +#include #include namespace ProfileEvents @@ -648,6 +650,70 @@ void CasOperation::removeManyWriteOnce(const std::vector & keys, c }); } +namespace +{ +Removal toConditionalRemoval(const Backend::RawRemoval raw) +{ + switch (raw) + { + case Backend::RawRemoval::Removed: return Removal::Removed; + case Backend::RawRemoval::Gone: return Removal::Gone; + case Backend::RawRemoval::Mismatch: return Removal::Mismatch; + case Backend::RawRemoval::DeleteMarker: + throw Exception(ErrorCodes::CAS_DELETE_MARKER, + "CAS conditional batch removal archived a noncurrent version instead of reclaiming an object: " + "the bucket has object versioning enabled"); + } + UNREACHABLE(); +} +} + +std::vector CasOperation::removeManyByConditional( + const std::vector & removals, const Retry & policy) +{ + const size_t limit = std::min(kBulkDeleteMaxKeys, owner.backend->conditionalBatchDeleteKeyLimit()); + return removeManyByConditional(removals, policy, limit); +} + +std::vector CasOperation::removeManyByConditional( + const std::vector & removals, const Retry & policy, const size_t limit) +{ + if (removals.size() > limit) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS removeManyByConditional: {} keys in one chunk, the limit is {}; the consumer chunks its input", + removals.size(), limit); + if (removals.empty()) + return {}; + + std::set keys; + std::vector raw_removals; + raw_removals.reserve(removals.size()); + for (const ConditionalRemove & removal : removals) + { + if (!keys.emplace(removal.key).second) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS removeManyByConditional: duplicate key '{}' in one chunk", removal.key); + raw_removals.push_back({removal.key, owner.valueFor(removal.key, removal.expected_token)}); + } + + const Retry::Bound bound = owner.bindPolicy(policy); + const String subject = fmt::format("{} (+{} keys)", removals.front().key, removals.size() - 1); + const std::vector raw_results = readLoop( + "removeManyByConditional", subject, policy, bound, [&](auto & access) + { + return owner.backend->removeManyByConditional(raw_removals, access); + }); + if (raw_results.size() != removals.size()) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "CAS removeManyByConditional: backend returned {} results for {} keys", raw_results.size(), removals.size()); + + std::vector results; + results.reserve(raw_results.size()); + for (const Backend::RawRemoval raw : raw_results) + results.push_back(toConditionalRemoval(raw)); + return results; +} + Removal CasOperation::remove(const String & key, const Etag & seen, const Retry & policy) { return removeUnder(key, owner.valueFor(key, seen), policy, owner.bindPolicy(policy)); diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h index 05eca89409610..b541b17e27168 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasRequests.h @@ -34,6 +34,8 @@ namespace DB::ErrorCodes namespace DB::Cas { +class CasOperation; + /// TRUE when the store's own answer proves this write never applied: a malformed request, an entity /// too large, an access denial, or a credential failure. Whether a STALE CREDENTIAL explains it is not /// asked here -- the engine asks the backend for fresh credentials first, and refuses only when no @@ -280,6 +282,12 @@ class CasRequests /// The cap on one `removeManyWriteOnce` chunk -- also the ceiling a batch-delete request can carry. inline constexpr size_t kBulkDeleteMaxKeys = 1000; +struct ConditionalRemove +{ + String key; + Etag expected_token; +}; + /// One admitted operation: the unit a policy, a fence generation and a liveness predicate apply to. /// Move-only, and every request it makes re-checks its admission -- before each attempt, before each /// sleep, and once more after a proven commit, so a write whose fence was lost while it was in flight @@ -342,6 +350,7 @@ class CasOperation /// success. Throws when the policy is exhausted. More keys than the cap is a caller bug: the /// consumer chunks, so that every chunk that succeeded is recorded before a later one can fail. void removeManyWriteOnce(const std::vector & keys, const Retry & policy); + std::vector removeManyByConditional(const std::vector & removals, const Retry & policy); /// The one primitive that reports failure as a value, so the policy reissues on the OUTCOME: /// `Indeterminate` is retried, the four authoritative outcomes return at once, and an /// `Indeterminate` that outlives the bound is returned rather than thrown. A lost fence or a spent @@ -363,6 +372,11 @@ class CasOperation WriteResult readModifyWriteOnPresence(const String & key, const DecideOnMeta & decide, const Retry & policy); private: + friend bool probeConditionalBatchDelete(CasOperation & op, const String & probe_prefix, const ConditionalRemove & absent_key); + + std::vector removeManyByConditional( + const std::vector & removals, const Retry & policy, const size_t limit); + friend class CasRequests; /// The lane waits on this operation's own admission and reads and writes through its verbs; it /// needs the gate, the reservation, the resolve read and the give-up helpers, never the transport. diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h index 8f69d4464347d..58a78ae85e4e8 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Backend/CasThrottlingBackend.h @@ -95,6 +95,14 @@ class ThrottlingBackend final : public Backend inner->removeManyWriteOnce(keys, access); } + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + for (const RawConditionalRemove & removal : removals) + refuseOrPass(removal.key); + return inner->removeManyByConditional(removals, access); + } + std::expected write(const String & key, const String & bytes, const std::optional & expected_value, TransportAccess & access) override { @@ -126,6 +134,8 @@ class ThrottlingBackend final : public Backend uint64_t attemptTimeoutMs() const override { return inner->attemptTimeoutMs(); } uint64_t attemptEnvelopeMs() const override { return inner->attemptEnvelopeMs(); } size_t bulkDeleteKeyLimit() const override { return inner->bulkDeleteKeyLimit(); } + size_t conditionalBatchDeleteKeyLimit() const override { return inner->conditionalBatchDeleteKeyLimit(); } + void pinConditionalBatchDeleteSupport(const bool supported) override { inner->pinConditionalBatchDeleteSupport(supported); } bool refreshCredentials() override { return inner->refreshCredentials(); } void checkPoolPreconditions() override { inner->checkPoolPreconditions(); } void checkSkipAccessCheckSupport() override { inner->checkSkipAccessCheckSupport(); } diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp index e6b9afe6f155b..98dd06e6ac244 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.cpp @@ -580,29 +580,66 @@ size_t Gc::bulkDeleteChunkKeys() const return std::clamp(std::min(configured, storage_limit), 1, kBulkDeleteMaxKeys); } -Gc::RedeleteIo Gc::performRedeleteIo(const RetiredEntry & entry, const Layout & layout, CasOperation & op) +size_t Gc::conditionalDeleteKeysPerChunk() const { - RedeleteIo io; - io.blob_key = layout.blobKey(entry.ref); + const uint64_t configured_limit = store->poolConfig().gc_bulk_delete_chunk_keys; + const uint64_t storage_limit = store->poolBackendPtr()->conditionalBatchDeleteKeyLimit(); + return std::clamp(std::min(configured_limit, storage_limit), 1, kBulkDeleteMaxKeys); +} + +void Gc::readRetiredBlobState( + const RetiredEntry & entry, const Layout & layout, CasOperation & op, RedeleteBlobState & redelete_blob_state) +{ + redelete_blob_state.blob_key = layout.blobKey(entry.ref); /// The condemned incarnation is a PERSISTED pair and cannot itself be a precondition, so /// the round observes the blob and compares the two renderings. Observing first also /// settles the absent case without spending a conditional delete against a key that is /// already gone. - const std::optional observed = op.head(io.blob_key, Retry::standard()); - if (observed) - io.del = entry.token.matches(observed->etag) ? op.remove(io.blob_key, observed->etag, Retry::standard()) : Removal::Mismatch; - return io; + const std::optional blob_data = op.head(redelete_blob_state.blob_key, Retry::standard()); + if (!blob_data) + redelete_blob_state.removal_result = Removal::Gone; + else if (entry.token.matches(blob_data->etag)) + redelete_blob_state.live_token = blob_data->etag; + else + redelete_blob_state.removal_result = Removal::Mismatch; +} + +std::vector Gc::prepareConditionalDeleteChunks( + const std::vector & redelete_blob_states, const size_t max_keys_per_chunk) +{ + std::vector chunks; + std::optional chunk; + for (size_t i = 0; i < redelete_blob_states.size(); ++i) + { + const RedeleteBlobState & redelete_blob_state = redelete_blob_states[i]; + if (redelete_blob_state.error || !redelete_blob_state.live_token) + continue; + + if (!chunk) + chunk.emplace(); + chunk->blob_state_indices.push_back(i); + chunk->conditional_removals.push_back({redelete_blob_state.blob_key, *redelete_blob_state.live_token}); + if (chunk->blob_state_indices.size() == max_keys_per_chunk) + { + chunks.push_back(std::move(*chunk)); + chunk.reset(); + } + } + if (chunk) + chunks.push_back(std::move(*chunk)); + return chunks; } -void Gc::applyRedeleteOutcome(RedeleteRoundContext & ctx, const RetiredEntry & entry, const RedeleteIo & io) +void Gc::recordRedeleteSuccess(RedeleteRoundContext & ctx, const RetiredEntry & entry, const RedeleteBlobState & redelete_blob_state) { - if (const auto & hook = store->poolConfig().gc_redelete_apply_hook_for_test) + if (const auto & hook = store->poolConfig().gc_redelete_record_hook_for_test) hook(entry.ref); - const OutcomeKind outcome_kind = io.del == Removal::Removed ? OutcomeKind::Deleted - : io.del == Removal::Gone ? OutcomeKind::Absent - : OutcomeKind::Replaced; + const Removal removal_result = *redelete_blob_state.removal_result; + const OutcomeKind outcome_kind = removal_result == Removal::Removed ? OutcomeKind::Deleted + : removal_result == Removal::Gone ? OutcomeKind::Absent + : OutcomeKind::Replaced; OutcomeEntry outcome{.kind = entry.kind, .ref = entry.ref, .token = entry.token, .outcome = outcome_kind}; - const String del_outcome{removalName(io.del)}; + const String removal_outcome{removalName(removal_result)}; /// The single content-delete site is attributable per row. A mismatch (a writer recreated /// the incarnation) is terminal-OK: the fresh incarnation is a live object. EventEmitter{*store}.emit( @@ -614,11 +651,11 @@ void Gc::applyRedeleteOutcome(RedeleteRoundContext & ctx, const RetiredEntry & e e.token = renderIncarnation(entry.token); e.round = ctx.new_round; e.gen = ctx.snap_generation; - e.outcome = del_outcome; + e.outcome = removal_outcome; e.reason = "delete_pending published by a prior pass; exact-incarnation delete (pre-CAS)"; - e.detail = {{"condemn_round", std::to_string(entry.condemn_round)}, {"key", io.blob_key}}; + e.detail = {{"condemn_round", std::to_string(entry.condemn_round)}, {"key", redelete_blob_state.blob_key}}; }); - /// The audit row is observability only -- the delete in `performRedeleteIo` already executed + /// The audit row is observability only -- the delete in `redeleteBlobs` already executed /// regardless of this cap. Skipping it here bounds the per-shard `GcOutcomes` body without /// skipping or deferring any destructive work. ctx.outcome_tallies[ctx.shard].add(outcome_kind); @@ -626,140 +663,277 @@ void Gc::applyRedeleteOutcome(RedeleteRoundContext & ctx, const RetiredEntry & e ctx.outcomes[ctx.shard].entries.push_back(std::move(outcome)); ++ctx.report.redeleted; ProfileEvents::increment(ProfileEvents::CASGCRetiredRedeleted); +} + +void Gc::scheduleBlobMetaDeletes( + const std::vector & entries, + const std::vector & redelete_blob_states, + const uint64_t newest_round) +{ /// Drop the per-hash meta only on a removal or a proven absence — a mismatch means a /// writer already published a fresh incarnation at this hash, and that writer's own /// republication path (`PartWriteTxn::ensureBlobPresent`) reconciles the meta back to Clean /// right after the publication; deleting it here would race that legitimate Clean write for /// no reason (the meta is advisory, but there is no reason to touch it on that path at all). - if (io.del == Removal::Removed || io.del == Removal::Gone) + for (size_t i = 0; i < redelete_blob_states.size(); ++i) + { + const RedeleteBlobState & redelete_blob_state = redelete_blob_states[i]; + if (redelete_blob_state.error) + continue; + if (redelete_blob_state.removal_result == Removal::Removed || redelete_blob_state.removal_result == Removal::Gone) + meta_writer->scheduleConfirmedMetaDelete(entries[i].ref, newest_round); + } +} + +std::string_view Gc::redeleteFailureName(const RedeleteFailureKind failure_kind) +{ + switch (failure_kind) { - /// The round before this one, not `entry.condemn_round`: the graduation gate re-stamps a marker it - /// refused at its own, later round. Deleting a marker is safe for writers that adopt only on `Clean`; - /// a 26.6.4 writer still backfills an absent marker and adopts. - meta_writer->scheduleConfirmedMetaDelete(entry.ref, ctx.new_round - 1); + case RedeleteFailureKind::ReadBlobJob: return "read blob job"; + case RedeleteFailureKind::DeleteChunkJob: return "delete chunk job"; + case RedeleteFailureKind::ScheduleReadBlobJob: return "scheduling read blob job"; + case RedeleteFailureKind::ScheduleDeleteChunkJob: return "scheduling delete chunk job"; } + UNREACHABLE(); } -void Gc::reportRedeleteFailure( +void Gc::recordRedeleteFailure( RedeleteRoundContext & ctx, const RetiredEntry & entry, const Layout & layout, - std::exception_ptr exception, - std::optional unsent_count) + const RedeleteFailureKind failure_kind, + const std::exception_ptr exception, + const std::optional unscheduled_blob_count) { ProfileEvents::increment(ProfileEvents::CASGCRetiredRedeleteFailed); ++ctx.report.redelete_failed; LOG_WARNING(logger, - "CAS gc: pending delete of blob {} (key `{}`, condemned at round {}) failed; the entry stays delete_pending " + "CAS gc: {} failed for blob {} (key `{}`, condemned at round {}); the entry stays delete_pending " "and is retried in the next round: {}{}", - blobIdOf(entry.ref), layout.blobKey(entry.ref), entry.condemn_round, getExceptionMessage(exception, false), - unsent_count ? fmt::format("; {} re-delete entries were not submitted", *unsent_count) : String{}); + redeleteFailureName(failure_kind), blobIdOf(entry.ref), layout.blobKey(entry.ref), entry.condemn_round, getExceptionMessage(exception, false), + unscheduled_blob_count ? fmt::format("; {} blob {} were not submitted", *unscheduled_blob_count, + failure_kind == RedeleteFailureKind::ScheduleReadBlobJob ? "reads" : "deletions") : String{}); } -void Gc::redeleteBlobs( - RedeleteRoundContext & ctx, - const std::vector & entries, +Gc::IoJobsResult Gc::runRedeleteIoJobs( ThreadPool & pool, - size_t concurrency, - const Layout & layout, - CasOperation & op) + const size_t concurrency, + const size_t job_count, + const size_t refuse_offset, + CasOperation & op, + const std::function & job) { - std::vector io_results(entries.size()); - std::vector io_errors(entries.size()); - std::optional schedule_error; - size_t scheduled = 0; - if (concurrency <= 1 || entries.size() <= 1) + IoJobsResult result; + result.scheduled_job_errors.resize(job_count); + if (concurrency <= 1 || job_count <= 1) { - for (size_t i = 0; i < entries.size(); ++i) + for (; result.scheduled < job_count; ++result.scheduled) { try { - io_results[i] = performRedeleteIo(entries[i], layout, op); + job(result.scheduled, op); } catch (...) { - io_errors[i] = std::current_exception(); + result.scheduled_job_errors[result.scheduled] = std::current_exception(); } } - scheduled = entries.size(); + return result; } - else - { - ThreadPoolCallbackRunnerLocal runner(pool, ThreadName::CAS_GC_REDELETE); - std::vector::Task>> handles; - handles.reserve(entries.size()); - SCOPE_EXIT_SAFE({ ThreadPoolCallbackRunnerLocal::waitForAllToFinish(handles); }); - const uint64_t admitted_generation = op.generation(); - try + result.on_pool = true; + ThreadPoolCallbackRunnerLocal runner(pool, ThreadName::CAS_GC_REDELETE); + std::vector::Task>> handles; + handles.reserve(job_count); + + SCOPE_EXIT_SAFE({ ThreadPoolCallbackRunnerLocal::waitForAllToFinish(handles); }); + + const uint64_t admitted_generation = op.generation(); + try + { + for (; result.scheduled < job_count; ++result.scheduled) { - for (; scheduled < entries.size(); ++scheduled) + if (io_pool_refuse_at_for_test == refuse_offset + result.scheduled) { - if (io_pool_refuse_at_for_test == scheduled) - { - io_pool_refuse_at_for_test.reset(); - throw Exception(ErrorCodes::CANNOT_SCHEDULE_TASK, "Injected CAS GC re-delete enqueue refusal at index {}", scheduled); - } - handles.emplace_back(runner.enqueueAndGiveOwnership( - [slot = &io_results[scheduled], - entry = entries[scheduled], - layout_ptr = &layout, - pool_store = store, - admitted_generation] - { - CasOperation job_op = pool_store->openRequests().resume(admitted_generation); - *slot = performRedeleteIo(entry, *layout_ptr, job_op); - })); + io_pool_refuse_at_for_test.reset(); + throw Exception( + ErrorCodes::CANNOT_SCHEDULE_TASK, + "Injected CAS GC re-delete enqueue refusal at index {}", + refuse_offset + result.scheduled); } + handles.emplace_back(runner.enqueueAndGiveOwnership( + [&job, index = result.scheduled, pool_store = store, admitted_generation] + { + CasOperation job_op = pool_store->openRequests().resume(admitted_generation); + job(index, job_op); + })); + } + } + catch (...) + { + result.job_schedule_error = IndexedException{result.scheduled, std::current_exception()}; + } + + ThreadPoolCallbackRunnerLocal::waitForAllToFinish(handles); + for (size_t i = 0; i < handles.size(); ++i) + { + try + { + handles[i]->future.get(); } catch (...) { - schedule_error = IndexedException{scheduled, std::current_exception()}; + result.scheduled_job_errors[i] = std::current_exception(); } - ctx.jobs_scheduled += handles.size(); + } + return result; +} - ThreadPoolCallbackRunnerLocal::waitForAllToFinish(handles); - for (size_t i = 0; i < handles.size(); ++i) +void Gc::redeleteBlobs( + RedeleteRoundContext & ctx, + const std::vector & entries, + ThreadPool & pool, + const size_t concurrency, + const Layout & layout, + CasOperation & op) +{ + std::vector redelete_blob_states(entries.size()); + const IoJobsResult read_blob_jobs = runRedeleteIoJobs( + pool, + concurrency, + entries.size(), + 0, + op, + [&](const size_t index, CasOperation & job_op) { readRetiredBlobState(entries[index], layout, job_op, redelete_blob_states[index]); }); + + if (read_blob_jobs.on_pool) + ctx.read_blob_jobs_scheduled += read_blob_jobs.scheduled; + std::optional first_error + = recordReadBlobJobFailures(ctx, entries, layout, redelete_blob_states, read_blob_jobs); + + const std::vector delete_chunks + = prepareConditionalDeleteChunks(redelete_blob_states, conditionalDeleteKeysPerChunk()); + + const IoJobsResult delete_chunk_jobs = runRedeleteIoJobs( + pool, + concurrency, + delete_chunks.size(), + entries.size(), + op, + [&](const size_t chunk_index, CasOperation & job_op) { - try - { - handles[i]->future.get(); - } - catch (...) - { - io_errors[i] = std::current_exception(); - } - } + const ConditionalDeleteChunk & delete_chunk = delete_chunks[chunk_index]; + const auto removal_results = job_op.removeManyByConditional(delete_chunk.conditional_removals, Retry::standard()); + + for (size_t pos = 0; pos < delete_chunk.blob_state_indices.size(); ++pos) + redelete_blob_states[delete_chunk.blob_state_indices[pos]].removal_result = removal_results[pos]; + }); + + if (delete_chunk_jobs.on_pool) + ctx.delete_chunk_jobs_scheduled += delete_chunk_jobs.scheduled; + ctx.delete_chunks += delete_chunk_jobs.scheduled; + + const std::optional delete_error + = recordDeleteChunkJobFailures(ctx, entries, layout, redelete_blob_states, delete_chunks, delete_chunk_jobs); + + if (delete_error && (!first_error || delete_error->index < first_error->index)) + first_error = delete_error; + + recordRedeleteSuccesses(ctx, entries, redelete_blob_states, read_blob_jobs.scheduled); + scheduleBlobMetaDeletes(entries, redelete_blob_states, ctx.new_round - 1); + + if (first_error) + std::rethrow_exception(first_error->exception); +} + +std::optional Gc::recordReadBlobJobFailures( + RedeleteRoundContext & ctx, + const std::vector & entries, + const Layout & layout, + std::vector & redelete_blob_states, + const IoJobsResult & read_blob_jobs) +{ + std::optional first_error; + for (size_t i = 0; i < read_blob_jobs.scheduled; ++i) + { + const std::exception_ptr & error = read_blob_jobs.scheduled_job_errors[i]; + redelete_blob_states[i].error = error; + if (!error) + continue; + ++ctx.read_blob_jobs_failed; + recordRedeleteFailure(ctx, entries[i], layout, RedeleteFailureKind::ReadBlobJob, error); + if (!first_error || i < first_error->index) + first_error = IndexedException{i, error}; + } + if (read_blob_jobs.job_schedule_error) + { + ++ctx.read_blob_job_schedule_failures; + const IndexedException & error = *read_blob_jobs.job_schedule_error; + chassert(error.index < entries.size()); + recordRedeleteFailure(ctx, entries[error.index], layout, RedeleteFailureKind::ScheduleReadBlobJob, + error.exception, entries.size() - error.index); + if (!first_error || error.index < first_error->index) + first_error = error; } + return first_error; +} +std::optional Gc::recordDeleteChunkJobFailures( + RedeleteRoundContext & ctx, + const std::vector & entries, + const Layout & layout, + std::vector & redelete_blob_states, + const std::vector & delete_chunks, + const IoJobsResult & delete_chunk_jobs) +{ std::optional first_error; - for (size_t i = 0; i < scheduled; ++i) + for (size_t chunk_index = 0; chunk_index < delete_chunk_jobs.scheduled; ++chunk_index) { - if (!io_errors[i]) + const std::exception_ptr & error = delete_chunk_jobs.scheduled_job_errors[chunk_index]; + if (!error) continue; - reportRedeleteFailure(ctx, entries[i], layout, io_errors[i]); - if (!first_error) - first_error = IndexedException{i, io_errors[i]}; + ++ctx.delete_chunk_jobs_failed; + for (const size_t index : delete_chunks[chunk_index].blob_state_indices) + { + redelete_blob_states[index].error = error; + recordRedeleteFailure(ctx, entries[index], layout, RedeleteFailureKind::DeleteChunkJob, error); + if (!first_error || index < first_error->index) + first_error = IndexedException{index, error}; + } } - if (schedule_error) + if (delete_chunk_jobs.job_schedule_error) { - reportRedeleteFailure( - ctx, entries[schedule_error->index], layout, schedule_error->exception, entries.size() - schedule_error->index); - if (!first_error || schedule_error->index < first_error->index) - first_error = schedule_error; + ++ctx.delete_chunk_job_schedule_failures; + const size_t first_unscheduled_chunk_index = delete_chunk_jobs.job_schedule_error->index; + chassert(first_unscheduled_chunk_index < delete_chunks.size()); + size_t unscheduled_delete_blob_count = 0; + for (size_t chunk_index = first_unscheduled_chunk_index; chunk_index < delete_chunks.size(); ++chunk_index) + unscheduled_delete_blob_count += delete_chunks[chunk_index].blob_state_indices.size(); + + const size_t index = delete_chunks[first_unscheduled_chunk_index].blob_state_indices.front(); + const std::exception_ptr & error = delete_chunk_jobs.job_schedule_error->exception; + recordRedeleteFailure(ctx, entries[index], layout, RedeleteFailureKind::ScheduleDeleteChunkJob, + error, unscheduled_delete_blob_count); + if (!first_error || index < first_error->index) + first_error = IndexedException{index, error}; } + return first_error; +} - /// Every submitted delete of the batch has already run. If `applyRedeleteOutcome` throws, the batch stays deleted +void Gc::recordRedeleteSuccesses( + RedeleteRoundContext & ctx, + const std::vector & entries, + const std::vector & redelete_blob_states, + const size_t scheduled_read_blob_job_count) +{ + /// Every submitted delete of the batch has already run. If `recordRedeleteSuccess` throws, the batch stays deleted /// but unrecorded: the round fails before its outcome logs and `gc/state` commit, every entry stays /// `delete_pending`, and the next round records the deleted ones as `Absent`. - for (size_t i = 0; i < scheduled; ++i) + for (size_t i = 0; i < scheduled_read_blob_job_count; ++i) { - if (!io_errors[i]) - applyRedeleteOutcome(ctx, entries[i], io_results[i]); + if (!redelete_blob_states[i].error && redelete_blob_states[i].removal_result) + recordRedeleteSuccess(ctx, entries[i], redelete_blob_states[i]); } - - if (first_error) - std::rethrow_exception(first_error->exception); } RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool allow_steal, UniversePolicy policy, @@ -1057,12 +1231,22 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al pending_deletes_timer.emplace(phase_sink, "pending_deletes"); const uint64_t redeleted_before = report.redeleted; const uint64_t graduated_before = report.graduated; - const uint64_t redelete_failed_before = report.redelete_failed; - uint64_t redelete_jobs_scheduled = 0; + uint64_t read_blob_jobs_scheduled = 0; + uint64_t delete_chunk_jobs_scheduled = 0; + uint64_t read_blob_jobs_failed = 0; + uint64_t delete_chunk_jobs_failed = 0; + uint64_t read_blob_job_schedule_failures = 0; + uint64_t delete_chunk_job_schedule_failures = 0; + uint64_t delete_chunks = 0; const auto record_redelete_job_metrics = [&] { - pending_deletes_timer->metric("jobs_scheduled", redelete_jobs_scheduled); - pending_deletes_timer->metric("jobs_failed", report.redelete_failed - redelete_failed_before); + pending_deletes_timer->metric("read_blob_jobs_scheduled", read_blob_jobs_scheduled); + pending_deletes_timer->metric("delete_chunk_jobs_scheduled", delete_chunk_jobs_scheduled); + pending_deletes_timer->metric("read_blob_jobs_failed", read_blob_jobs_failed); + pending_deletes_timer->metric("delete_chunk_jobs_failed", delete_chunk_jobs_failed); + pending_deletes_timer->metric("read_blob_job_schedule_failures", read_blob_job_schedule_failures); + pending_deletes_timer->metric("delete_chunk_job_schedule_failures", delete_chunk_job_schedule_failures); + pending_deletes_timer->metric("delete_chunks", delete_chunks); }; SCOPE_EXIT_SAFE({ if (pending_deletes_timer) @@ -1098,8 +1282,16 @@ RoundReport Gc::runRegularRound(std::function on_lease_acquired, bool al .report = report, .outcomes = outcomes, .outcome_tallies = outcome_tallies, - .jobs_scheduled = redelete_jobs_scheduled}; + .read_blob_jobs_scheduled = read_blob_jobs_scheduled, + .delete_chunk_jobs_scheduled = delete_chunk_jobs_scheduled, + .read_blob_jobs_failed = read_blob_jobs_failed, + .delete_chunk_jobs_failed = delete_chunk_jobs_failed, + .read_blob_job_schedule_failures = read_blob_job_schedule_failures, + .delete_chunk_job_schedule_failures = delete_chunk_job_schedule_failures, + .delete_chunks = delete_chunks}; + redeleteBlobs(redelete_ctx, redelete_now, *io_pool, store->poolConfig().gc_io_concurrency, layout, op); + for (const RetiredEntry & entry : merge.spared) { /// A fresh dedup-adopt raced the condemn (see the matching CasGcFold Debug log emitted diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h index fd0cc34fde5d9..0181d0704b7e0 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Gc/CasGc.h @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -528,6 +529,8 @@ class Gc /// limit, clamped to [1, `kBulkDeleteMaxKeys`] because an empty request would never advance. size_t bulkDeleteChunkKeys() const; + size_t conditionalDeleteKeysPerChunk() const; + void setRebuildEdgeBudgetForTest(uint64_t n) { rebuild_edge_budget_override = n; } /// TEST SEAM: disable the round's journal trim so a folded event stays in the journal @@ -559,6 +562,24 @@ class Gc /// (throwaway keys) remove non-content objects they own. Adding a second content-delete site is a /// protocol defect. + struct RedeleteBlobState + { + String blob_key; + std::optional live_token; + std::optional removal_result; + std::exception_ptr error; + }; + + struct ConditionalDeleteChunk + { + /// `conditional_removals[i]` targets `redelete_blob_states[blob_state_indices[i]]`. + std::vector blob_state_indices; + std::vector conditional_removals; + }; + + static std::vector prepareConditionalDeleteChunks( + const std::vector & redelete_blob_states, size_t max_keys_per_chunk); + private: /// Lease acquire/renew/steal per the documented observation protocol. On success `state` holds the /// committed gc/state (with our lease) and `state_etag` the etag that write created. @@ -760,18 +781,30 @@ class Gc /// every destructive-work family the round touches — see `GcRoundWorkBudget`. GcRoundWorkBudget & work_budget); - struct RedeleteIo - { - String blob_key; - Removal del = Removal::Gone; - }; - struct IndexedException { size_t index; std::exception_ptr exception; }; + enum class RedeleteFailureKind + { + ReadBlobJob, + DeleteChunkJob, + ScheduleReadBlobJob, + ScheduleDeleteChunkJob, + }; + + static std::string_view redeleteFailureName(RedeleteFailureKind failure_kind); + + struct IoJobsResult + { + bool on_pool = false; + size_t scheduled = 0; + std::vector scheduled_job_errors; + std::optional job_schedule_error; + }; + /// The settlement decisions of one shard in a round, counted where each is made: a decision whose /// audit row the outcome-log limits dropped still counts. struct OutcomeTally @@ -794,19 +827,61 @@ class Gc RoundReport & report; std::map & outcomes; std::map & outcome_tallies; - uint64_t & jobs_scheduled; + uint64_t & read_blob_jobs_scheduled; + uint64_t & delete_chunk_jobs_scheduled; + uint64_t & read_blob_jobs_failed; + uint64_t & delete_chunk_jobs_failed; + uint64_t & read_blob_job_schedule_failures; + uint64_t & delete_chunk_job_schedule_failures; + uint64_t & delete_chunks; }; - static RedeleteIo performRedeleteIo(const RetiredEntry & entry, const Layout & layout, CasOperation & op); + static void readRetiredBlobState( + const RetiredEntry & entry, const Layout & layout, CasOperation & op, RedeleteBlobState & redelete_blob_state); - void applyRedeleteOutcome(RedeleteRoundContext & ctx, const RetiredEntry & entry, const RedeleteIo & io); + void recordRedeleteSuccess(RedeleteRoundContext & ctx, const RetiredEntry & entry, const RedeleteBlobState & redelete_blob_state); - void reportRedeleteFailure( + void recordRedeleteFailure( RedeleteRoundContext & ctx, const RetiredEntry & entry, const Layout & layout, + RedeleteFailureKind failure_kind, std::exception_ptr exception, - std::optional unsent_count = std::nullopt); + std::optional unscheduled_blob_count = std::nullopt); + + std::optional recordReadBlobJobFailures( + RedeleteRoundContext & ctx, + const std::vector & entries, + const Layout & layout, + std::vector & redelete_blob_states, + const IoJobsResult & read_blob_jobs); + + std::optional recordDeleteChunkJobFailures( + RedeleteRoundContext & ctx, + const std::vector & entries, + const Layout & layout, + std::vector & redelete_blob_states, + const std::vector & delete_chunks, + const IoJobsResult & delete_chunk_jobs); + + void recordRedeleteSuccesses( + RedeleteRoundContext & ctx, + const std::vector & entries, + const std::vector & redelete_blob_states, + size_t scheduled_read_blob_job_count); + + void scheduleBlobMetaDeletes( + const std::vector & entries, + const std::vector & redelete_blob_states, + uint64_t newest_round); + + IoJobsResult runRedeleteIoJobs( + ThreadPool & pool, + size_t concurrency, + size_t job_count, + size_t refuse_offset, + CasOperation & op, + const std::function & job); void redeleteBlobs( RedeleteRoundContext & ctx, diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp index 3cb1a0b783caa..7d9b85144c91f 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.cpp @@ -543,7 +543,9 @@ PoolPtr Pool::open(BackendPtr backend, PoolConfig config) /// namespace only (never the content planes) — acceptable. const UInt128 probe_uid = (static_cast(thread_local_rng()) << 64) | thread_local_rng(); CasOperation probe_op = bootstrap_requests.admit(); - runCapabilityProbe(probe_op, config.pool_prefix + "/_probe/" + u128ToHex(probe_uid)); + const CapabilityProbeResult probe_result = runCapabilityProbe( + probe_op, config.pool_prefix + "/_probe/" + u128ToHex(probe_uid)); + backend->pinConditionalBatchDeleteSupport(probe_result.conditional_batch_delete_supported); } else { diff --git a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h index 759584fae76a1..b842e82ae9a84 100644 --- a/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h +++ b/src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasPool.h @@ -215,7 +215,7 @@ struct PoolConfig std::function teardown_phase2_throw_for_test = {}; std::function teardown_phase3_throw_for_test = {}; - std::function gc_redelete_apply_hook_for_test = {}; + std::function gc_redelete_record_hook_for_test = {}; std::optional gc_io_pool_refuse_at_for_test = std::nullopt; /// TEST SEAM: replaces the default per-shard byte limit of a round's outcome log. std::optional gc_outcome_log_byte_limit_for_test = std::nullopt; diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp index c763863d1a729..8a902b11c52dc 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.cpp @@ -67,6 +67,17 @@ ObjectStorageIteratorPtr IObjectStorage::iterate( return std::make_shared(std::move(files)); } +std::vector IObjectStorage::removeObjectsIfTokensMatch( + const std::vector & objects, const ObjectStorageControlRequest & request) +{ + if (objects.size() > 1) + throw Exception(ErrorCodes::NOT_IMPLEMENTED, + "Conditional batch object removal is not implemented for {} object storage", getName()); + if (objects.empty()) + return {}; + return {removeObjectIfTokenMatches(objects.front().object, objects.front().expected_token, request)}; +} + ObjectStorageIteratorPtr IObjectStorage::iterate( const std::string & path_prefix, size_t max_keys, diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h index d9d17af0f2eda..81eda5739a56f 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h @@ -297,6 +297,12 @@ struct ConditionalRemoveResult bool created_delete_marker = false; /// backend reported a versioning delete marker }; +struct ConditionalRemoveObject +{ + StoredObject object; + std::string expected_token; +}; + /// Base class for all object storages which implement some subset of ordinary filesystem operations. /// /// Examples of object storages are S3, Azure Blob Storage, HDFS. @@ -447,6 +453,9 @@ class IObjectStorage virtual ConditionalRemoveResult removeObjectIfTokenMatches( const StoredObject & object, const std::string & etag, const ObjectStorageControlRequest & request); + virtual std::vector removeObjectsIfTokensMatch( + const std::vector & objects, const ObjectStorageControlRequest & request); + /// Removes every object in ONE request with no per-key precondition; an absent object is success. /// Content-addressed callers use it for write-once keys only, at most 1000 per call. Throws on a /// request-level failure and on any per-key error other than "not found", naming the failed keys. @@ -461,6 +470,10 @@ class IObjectStorage /// suits a storage without a batch delete. virtual size_t batchDeleteKeyLimit() const { return 1; } + /// The configured key limit for a multi-object delete request, independent of any learned + /// capability. A caller that probes conditional delete support can apply its own capability gate. + virtual size_t configuredBatchDeleteKeyLimit() const { return 1; } + /// Copy object with different attributes if required virtual void copyObject( /// NOLINT const StoredObject & object_from, diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp index 41c3ba97b0b47..c50348613eab7 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.cpp @@ -43,6 +43,8 @@ #include #include +#include + namespace ProfileEvents { extern const Event S3ListObjects; @@ -641,6 +643,141 @@ ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatches( object, etag, clientForRetryProfile(request), request.attempt_number); }); } +std::vector S3ObjectStorage::removeObjectsIfTokensMatch( + const std::vector & objects, const ObjectStorageControlRequest & request) +{ + if (objects.empty()) + return {}; + if (objects.size() == 1) + return {removeObjectIfTokenMatches(objects.front().object, objects.front().expected_token, request)}; + + std::vector identifiers; + identifiers.reserve(objects.size()); + for (const ConditionalRemoveObject & object : objects) + { + Aws::S3::Model::ObjectIdentifier identifier; + identifier.SetKey(object.object.remote_path); + identifier.SetETag(object.expected_token); + identifiers.push_back(std::move(identifier)); + } + Aws::S3::Model::Delete delete_request; + delete_request.SetObjects(std::move(identifiers)); + delete_request.SetQuiet(false); + S3::DeleteObjectsRequest request_s3; + request_s3.SetBucket(uri.bucket); + request_s3.SetDelete(std::move(delete_request)); + if (request.attempt_number != 0) + S3::setClickHouseAttemptNumber(request_s3, request.attempt_number); + auto blob_storage_log = BlobStorageLogWriter::create(disk_name); + const auto log_object = [&](const StoredObject & object, const Int32 error_code, const String & error_message) + { + if (blob_storage_log) + blob_storage_log->addEvent( + BlobStorageLogElement::EventType::Delete, + uri.bucket, + object.remote_path, + object.local_path, + object.bytes_size, + 0, + error_code, + error_message); + }; + const auto log_all = [&](const Int32 error_code, const String & error_message) + { + for (const auto & object : objects) + log_object(object.object, error_code, error_message); + }; + const auto outcome = refreshAndRetryOnExpiredCredentials([&] + { + ProfileEvents::increment(ProfileEvents::DiskS3DeleteObjects); + auto current_outcome = clientForRetryProfile(request)->DeleteObjects(request_s3); + if (!current_outcome.IsSuccess()) + { + const auto & error = current_outcome.GetError(); + log_all(static_cast(error.GetErrorType()), error.GetMessage()); + if (error.GetExceptionName() == "NotImplemented") + throw Exception(ErrorCodes::NOT_IMPLEMENTED, "{} does not support conditional DeleteObjects", getName()); + throw S3Exception( + PreformattedMessage::create("{} while conditionally removing objects from S3", error.GetMessage()), + error.GetErrorType(), error.GetExceptionName()); + } + return current_outcome; + }); + std::unordered_set expected_keys; + expected_keys.reserve(objects.size()); + for (const ConditionalRemoveObject & object : objects) + expected_keys.insert(object.object.remote_path); + + std::unordered_set response_keys; + response_keys.reserve(objects.size()); + for (const auto & deleted : outcome.GetResult().GetDeleted()) + { + const auto & key = deleted.GetKey(); + if (!expected_keys.contains(key) || !response_keys.insert(key).second) + { + log_all(static_cast(Aws::S3::S3Errors::UNKNOWN), "DeleteObjects response has unexpected or duplicate keys"); + throw Exception(ErrorCodes::LOGICAL_ERROR, "S3 conditional DeleteObjects response has unexpected or duplicate key {}", key); + } + } + for (const auto & error : outcome.GetResult().GetErrors()) + { + const auto & key = error.GetKey(); + if (!expected_keys.contains(key) || !response_keys.insert(key).second) + { + log_all(static_cast(Aws::S3::S3Errors::UNKNOWN), "DeleteObjects response has unexpected or duplicate keys"); + throw Exception(ErrorCodes::LOGICAL_ERROR, "S3 conditional DeleteObjects response has unexpected or duplicate key {}", key); + } + } + if (response_keys.size() != objects.size()) + { + log_all(static_cast(Aws::S3::S3Errors::UNKNOWN), "DeleteObjects response omitted keys"); + throw Exception(ErrorCodes::LOGICAL_ERROR, "S3 conditional DeleteObjects response omitted keys"); + } + + std::vector results; + results.reserve(objects.size()); + std::optional first_error_code; + String first_error_name; + String first_error_message; + String first_error_key; + for (const ConditionalRemoveObject & object : objects) + { + const auto deleted = std::find_if(outcome.GetResult().GetDeleted().begin(), outcome.GetResult().GetDeleted().end(), + [&](const auto & entry) { return entry.GetKey() == object.object.remote_path; }); + if (deleted != outcome.GetResult().GetDeleted().end()) + { + log_object(object.object, 0, ""); + results.push_back({ConditionalRemoveOutcome::Removed, deleted->GetDeleteMarker()}); + continue; + } + const auto error = std::find_if(outcome.GetResult().GetErrors().begin(), outcome.GetResult().GetErrors().end(), + [&](const auto & entry) { return entry.GetKey() == object.object.remote_path; }); + if (error == outcome.GetResult().GetErrors().end()) + { + log_all(static_cast(Aws::S3::S3Errors::UNKNOWN), "DeleteObjects response omitted keys"); + throw Exception(ErrorCodes::LOGICAL_ERROR, "S3 conditional DeleteObjects response omitted key {}", object.object.remote_path); + } + const auto code = classifyDeleteObjectsErrorCode(error->GetCode()); + log_object(object.object, static_cast(code), error->GetMessage()); + if (error->GetCode() == "PreconditionFailed") + results.push_back({ConditionalRemoveOutcome::TokenMismatch, false}); + else if (error->GetCode() == "NoSuchKey") + results.push_back({ConditionalRemoveOutcome::NotFound, false}); + else if (!first_error_code) + { + first_error_code = code; + first_error_name = error->GetCode(); + first_error_message = error->GetMessage(); + first_error_key = error->GetKey(); + } + } + if (first_error_code) + throw S3Exception( + PreformattedMessage::create("{} while conditionally removing object {} from S3", first_error_message, first_error_key), + *first_error_code, first_error_name); + return results; +} + ConditionalRemoveResult S3ObjectStorage::removeObjectIfTokenMatchesImpl( const StoredObject & object, const std::string & etag, const std::shared_ptr & used_client, size_t attempt_seed) @@ -811,6 +948,11 @@ size_t S3ObjectStorage::batchDeleteKeyLimit() const { if (const auto supported = s3_capabilities.isBatchDeleteSupported(); supported.has_value() && !*supported) return 1; + return configuredBatchDeleteKeyLimit(); +} + +size_t S3ObjectStorage::configuredBatchDeleteKeyLimit() const +{ return std::max(1, s3_settings.get()->request_settings[S3RequestSetting::objects_chunk_size_to_delete]); } diff --git a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h index 2d285e289d9b0..f94e57955c451 100644 --- a/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h +++ b/src/Disks/DiskObjectStorage/ObjectStorages/S3/S3ObjectStorage.h @@ -130,6 +130,9 @@ class S3ObjectStorage : public IObjectStorage ConditionalRemoveResult removeObjectIfTokenMatches( const StoredObject & object, const std::string & etag, const ObjectStorageControlRequest & request) override; + std::vector removeObjectsIfTokensMatch( + const std::vector & objects, const ObjectStorageControlRequest & request) override; + /// One `DeleteObjects` for the given objects (the caller chunks to at most 1000); absence is success. /// Exactly one object is always a plain `DeleteObject` instead (never gated on `s3_capabilities`: a /// single physical request per call, so there is nothing here for that capability to say no to). @@ -142,6 +145,7 @@ class S3ObjectStorage : public IObjectStorage /// `objects_chunk_size_to_delete`, at least 1, while `DeleteObjects` is not known unsupported; 1 after. size_t batchDeleteKeyLimit() const override; + size_t configuredBatchDeleteKeyLimit() const override; void tagObjects(const StoredObjects & objects, const std::string & tag_key, const std::string & tag_value) override; diff --git a/src/Disks/tests/cas_gc_marker_test_support.h b/src/Disks/tests/cas_gc_marker_test_support.h index 0dbcbeb17baea..3f43697883398 100644 --- a/src/Disks/tests/cas_gc_marker_test_support.h +++ b/src/Disks/tests/cas_gc_marker_test_support.h @@ -46,13 +46,13 @@ inline BlobRef gateRef(const UInt128 & hash) /// A pool that folds every round, so a round's number is the number of rounds run. template PoolPtr openGatePool(std::shared_ptr backend, uint64_t io_concurrency = 1, uint64_t graduation_budget = 0, - std::function apply_hook = {}, uint64_t gc_shards = 1) + std::function record_hook = {}, uint64_t gc_shards = 1) { PoolConfig config{.pool_prefix = "p", .server_root_id = "test", .gc_fold_max_defer_rounds = 0, .gc_io_concurrency = io_concurrency}; config.gc_shards = gc_shards; config.gc_round_graduation_budget = graduation_budget; - config.gc_redelete_apply_hook_for_test = std::move(apply_hook); + config.gc_redelete_record_hook_for_test = std::move(record_hook); return Pool::open(std::move(backend), std::move(config)); } diff --git a/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp b/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp index 3ce05af764752..96cbc00d979d8 100644 --- a/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp +++ b/src/Disks/tests/gtest_cas_bulk_delete_backend.cpp @@ -141,6 +141,27 @@ TEST(CASBulkDeleteBackend, InstrumentedCountsOneRequestAndOneDeletePerKeyClass) EXPECT_EQ(ProfileEvents::global_counters[ProfileEvents::CASRootDelete] - root_before, 1u); } +TEST(CASConditionalBatchDeleteBackend, InstrumentedBackendForwardsMultiKeyConditionalRemove) +{ + auto inner = std::make_shared(); + inner->pinConditionalBatchDeleteSupport(true); + auto backend = std::make_shared(inner); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const String first_key = "p/conditional/instrumented/first"; + const String second_key = "p/conditional/instrumented/second"; + const Etag first_token = std::get(op.create(first_key, "v1", Retry::once())).etag; + const Etag second_token = std::get(op.create(second_key, "v1", Retry::once())).etag; + + EXPECT_EQ(backend->conditionalBatchDeleteKeyLimit(), kBulkDeleteMaxKeys); + const std::vector results = op.removeManyByConditional( + {{first_key, first_token}, {second_key, second_token}}, Retry::once()); + + EXPECT_EQ(results, (std::vector{Removal::Removed, Removal::Removed})); + EXPECT_FALSE(op.head(first_key, Retry::once()).has_value()); + EXPECT_FALSE(op.head(second_key, Retry::once()).has_value()); +} + #if USE_AWS_S3 TEST(CASBulkDeleteBackend, ThrottlingBackendForwardsStorageLimit) { diff --git a/src/Disks/tests/gtest_cas_bulk_delete_engine.cpp b/src/Disks/tests/gtest_cas_bulk_delete_engine.cpp index 01ace5ad336c3..426f11e645e1f 100644 --- a/src/Disks/tests/gtest_cas_bulk_delete_engine.cpp +++ b/src/Disks/tests/gtest_cas_bulk_delete_engine.cpp @@ -4,7 +4,12 @@ #include #include #include +#include #include +#include +#include +#include +#include /// The engine sends one chunk of at most 1000 write-once keys as one request under the ordinary /// attempt loop; a failed attempt reissues the whole chunk, which is sound because a key the failed @@ -19,6 +24,12 @@ using namespace DB::Cas; using DB::Cas::tests::CountingBackend; using DB::Cas::tests::openRequestsForTest; +namespace DB::ErrorCodes +{ + extern const int CAS_DELETE_MARKER; + extern const int LOGICAL_ERROR; +} + namespace { @@ -34,6 +45,34 @@ std::vector manifestKeys(uint32_t count) return keys; } +class UnlimitedConditionalBatchBackend : public InMemoryBackend +{ +public: + size_t conditionalBatchDeleteKeyLimit() const override { return kBulkDeleteMaxKeys + 1; } +}; + +class RecordingConditionalBackend : public InMemoryBackend +{ +public: + std::vector> attempts; + bool lose_first_response = false; + + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + attempts.push_back(removals); + if (lose_first_response && attempts.size() == 1 && removals.size() == 2) + { + const auto & first = removals[0]; + const auto & second = removals[1]; + EXPECT_EQ(remove(first.key, first.expected_token, access), RawRemoval::Removed); + EXPECT_TRUE(write(second.key, "replacement", second.expected_token, access).has_value()); + throw Poco::TimeoutException("lost conditional delete response"); + } + return InMemoryBackend::removeManyByConditional(removals, access); + } +}; + } TEST(CASBulkDeleteEngine, AFailedAttemptReissuesTheWholeChunkAndAlreadyDeletedKeysAreSuccess) @@ -70,6 +109,143 @@ TEST(CASBulkDeleteEngine, AnEmptyChunkIsNoRequest) EXPECT_EQ(backend->bulkRemoveCalls(), 0u); } +TEST(CASConditionalBatchDeleteEngine, ReturnsMixedOutcomesInInputOrder) +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + const String removed_key = "p/conditional/removed"; + const String replaced_key = "p/conditional/replaced"; + const String gone_key = "p/conditional/gone"; + + const Etag removed_token = std::get(op.create(removed_key, "v1", Retry::once())).etag; + const Etag stale_token = std::get(op.create(replaced_key, "v1", Retry::once())).etag; + const Etag gone_token = std::get(op.create(gone_key, "v1", Retry::once())).etag; + ASSERT_TRUE(std::holds_alternative(op.replace(replaced_key, "v2", stale_token, Retry::once()))); + ASSERT_EQ(op.remove(gone_key, gone_token, Retry::once()), Removal::Removed); + + const std::vector results = op.removeManyByConditional( + {{removed_key, removed_token}, {replaced_key, stale_token}, {gone_key, gone_token}}, Retry::once()); + + ASSERT_EQ(results.size(), 3u); + EXPECT_EQ(results[0], Removal::Removed); + EXPECT_EQ(results[1], Removal::Mismatch); + EXPECT_EQ(results[2], Removal::Gone); + EXPECT_FALSE(op.head(removed_key, Retry::once()).has_value()); + const auto replaced = op.read(replaced_key, Retry::once()); + ASSERT_TRUE(replaced.has_value()); + EXPECT_EQ(replaced->bytes, "v2"); +} + +TEST(CASConditionalBatchDeleteEngine, PartialSuccessRetryKeepsOriginalTokens) +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + backend->lose_first_response = true; + DB::Cas::tests::FakeClock clock; + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), nullptr, kDefaultRequestWindowMs); + auto op = requests.admit(); + const auto first = std::get(op.create("a", "first", Retry::once())).etag; + const auto second = std::get(op.create("b", "second", Retry::once())).etag; + + const auto results = op.removeManyByConditional({{"a", first}, {"b", second}}, Retry::standard()); + + EXPECT_EQ(results, (std::vector{Removal::Gone, Removal::Mismatch})); + ASSERT_EQ(backend->attempts.size(), 2u); + for (const auto & attempt : backend->attempts) + { + ASSERT_EQ(attempt.size(), 2u); + EXPECT_EQ(attempt[0].key, "a"); + EXPECT_EQ(attempt[0].expected_token, PersistedEtag::capture(first).value); + EXPECT_EQ(attempt[1].key, "b"); + EXPECT_EQ(attempt[1].expected_token, PersistedEtag::capture(second).value); + } + EXPECT_FALSE(op.head("a", Retry::once())); + const auto replacement = op.read("b", Retry::once()); + ASSERT_TRUE(replacement); + EXPECT_EQ(replacement->bytes, "replacement"); +} + +TEST(CASConditionalBatchDeleteEngine, DeleteMarkerFailsTheChunk) +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + auto requests = openRequestsForTest(backend); + auto op = requests.admit(); + const auto first = std::get(op.create("a", "body", Retry::once())).etag; + const auto second = std::get(op.create("b", "body", Retry::once())).etag; + ASSERT_EQ(op.remove("a", first, Retry::once()), Removal::Removed); + backend->setSimulateDeleteMarkers(true); + + DB::Cas::tests::expectThrowsCode(DB::ErrorCodes::CAS_DELETE_MARKER, [&] + { + op.removeManyByConditional({{"a", first}, {"b", second}}, Retry::once()); + }); +} + +TEST(CASConditionalBatchDeleteEngine, EmptyInputMakesNoRequestAndAFullChunkMakesOne) +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + auto requests = openRequestsForTest(backend); + auto op = requests.admit(); + EXPECT_TRUE(op.removeManyByConditional({}, Retry::once()).empty()); + EXPECT_TRUE(backend->attempts.empty()); + + std::vector chunk; + for (size_t i = 0; i < kBulkDeleteMaxKeys; ++i) + { + const String key = std::to_string(i); + chunk.push_back({key, std::get(op.create(key, "body", Retry::once())).etag}); + } + EXPECT_EQ(op.removeManyByConditional(chunk, Retry::once()), std::vector(chunk.size(), Removal::Removed)); + ASSERT_EQ(backend->attempts.size(), 1u); + EXPECT_EQ(backend->attempts.front().size(), chunk.size()); +} + +TEST(CASConditionalBatchDeleteEngineDeathTest, RejectsDuplicateKeysAndForeignTokensBeforeSending) +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + auto requests = openRequestsForTest(backend); + auto op = requests.admit(); + auto other_requests = openRequestsForTest(std::make_shared()); + auto other_op = other_requests.admit(); + const auto token = std::get(op.create("a", "body", Retry::once())).etag; + const auto foreign = std::get(other_op.create("b", "body", Retry::once())).etag; + const ConditionalRemove first{"a", token}; + const auto expect_rejected = [&](const ConditionalRemove & invalid, const String & expected_message) + { + SCOPED_TRACE(expected_message); + const std::vector chunk{first, invalid}; + EXPECT_DEATH( + { + Poco::AutoPtr channel(new Poco::ConsoleChannel(std::cerr)); + auto & logger = Poco::Logger::root(); + logger.setChannel(channel); + logger.setLevel(Poco::Message::PRIO_FATAL); + DB::abort_on_logical_error.store(true, std::memory_order_relaxed); + op.removeManyByConditional(chunk, Retry::once()); + }, + expected_message); +#if !defined(DEBUG_OR_SANITIZER_BUILD) + DB::Cas::tests::expectThrowsCodeWithMessage( + DB::ErrorCodes::LOGICAL_ERROR, expected_message, [&] { op.removeManyByConditional(chunk, Retry::once()); }); +#endif + }; + + expect_rejected(first, "duplicate key 'a' in one chunk"); + expect_rejected({"b", token}, fmt::format( + "incarnation of 'a' observed on backend {} cannot be the precondition for 'b' on backend {}", + token.backendId(), backend->backendId())); + expect_rejected({"b", foreign}, fmt::format( + "incarnation of 'b' observed on backend {} cannot be the precondition for 'b' on backend {}", + foreign.backendId(), backend->backendId())); + EXPECT_TRUE(backend->attempts.empty()); + EXPECT_TRUE(op.head("a", Retry::once())); +} + TEST(CASBulkDeleteEngine, ExactlyOneThousandKeysIsOneRequest) { auto backend = std::make_shared(); @@ -89,3 +265,28 @@ TEST(CASBulkDeleteEngineDeathTest, MoreThanOneThousandKeysIsACallerBug) EXPECT_EQ(backend->bulkRemoveCalls(), 0u); } #endif + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASConditionalBatchDeleteEngineDeathTest, MoreThanOneThousandKeysIsRejectedEvenIfBackendReportsALargerLimit) +#else +TEST(CASConditionalBatchDeleteEngine, MoreThanOneThousandKeysIsRejectedEvenIfBackendReportsALargerLimit) +#endif +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + CasRequests requests = openRequestsForTest(backend); + CasOperation op = requests.admit(); + std::vector removals; + removals.reserve(kBulkDeleteMaxKeys + 1); + for (size_t i = 0; i <= kBulkDeleteMaxKeys; ++i) + { + const String key = "p/conditional/limit/" + std::to_string(i); + removals.push_back({key, std::get(op.create(key, "v", Retry::once())).etag}); + } + +#if defined(DEBUG_OR_SANITIZER_BUILD) + EXPECT_DEATH({ op.removeManyByConditional(removals, Retry::once()); }, "CAS removeManyByConditional: 1001 keys"); +#else + EXPECT_THROW(op.removeManyByConditional(removals, Retry::once()), DB::Exception); +#endif +} diff --git a/src/Disks/tests/gtest_cas_gc_meta_writer.cpp b/src/Disks/tests/gtest_cas_gc_meta_writer.cpp index 546bbfdf67116..6663b2c154268 100644 --- a/src/Disks/tests/gtest_cas_gc_meta_writer.cpp +++ b/src/Disks/tests/gtest_cas_gc_meta_writer.cpp @@ -178,7 +178,7 @@ TEST(CASGcMetaWriter, ThrowingRoundDrainsBeforeReturning) namespace { -/// The delete job, scheduled the way `Gc::applyRedeleteOutcome` schedules it, and drained. +/// The delete job, scheduled the way `Gc::scheduleBlobMetaDeletes` schedules it, and drained. void runConfirmedMetaDelete(Gc & gc, const BlobRef & ref, uint64_t newest_round) { gc.metaWriterForTest().scheduleConfirmedMetaDelete(ref, newest_round); diff --git a/src/Disks/tests/gtest_cas_gc_redelete_concurrency.cpp b/src/Disks/tests/gtest_cas_gc_redelete_concurrency.cpp index a74965533d009..95c3912ccdd62 100644 --- a/src/Disks/tests/gtest_cas_gc_redelete_concurrency.cpp +++ b/src/Disks/tests/gtest_cas_gc_redelete_concurrency.cpp @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -35,6 +36,7 @@ extern const Event CASGCRetiredRedeleteFailed; namespace DB::ErrorCodes { extern const int CANNOT_SCHEDULE_TASK; +extern const int NETWORK_ERROR; } using namespace DB::Cas; @@ -82,6 +84,15 @@ class WorkerFaultBackend : public InMemoryBackend return InMemoryBackend::remove(key, expected_value, access); } + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + for (const RawConditionalRemove & removal : removals) + if (takeFault(remove_keys, removal.key)) + throw std::runtime_error("injected worker remove fault for key " + removal.key); + return InMemoryBackend::removeManyByConditional(removals, access); + } + std::optional head(const String & key, TransportAccess & access) override { if (takeFault(head_keys, key)) @@ -111,6 +122,39 @@ class WorkerFaultBackend : public InMemoryBackend std::set head_keys; }; +class RedeleteDiagnosticsBackend : public InMemoryBackend +{ +public: + std::set blob_keys; + std::function read_blob_hook; + std::function &, TransportAccess &)> delete_chunk_hook; + + std::optional head(const String & key, TransportAccess & access) override + { + if (!read_blob_hook || !blob_keys.contains(key)) + return InMemoryBackend::head(key, access); + + SCOPE_EXIT({ LOG_WARNING(logger, "Test blob read finished"); }); + read_blob_hook(key); + return InMemoryBackend::head(key, access); + } + + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + if (!delete_chunk_hook) + return InMemoryBackend::removeManyByConditional(removals, access); + + LOG_WARNING(logger, "Test delete chunk started"); + SCOPE_EXIT({ LOG_WARNING(logger, "Test delete chunk finished"); }); + delete_chunk_hook(removals, access); + return InMemoryBackend::removeManyByConditional(removals, access); + } + +private: + LoggerPtr logger = getLogger("CasGc"); +}; + class BlobRemoveWitnessBackend : public InMemoryBackend { public: @@ -165,6 +209,16 @@ class BlobRemoveWitnessBackend : public InMemoryBackend return result; } + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + std::vector results; + results.reserve(removals.size()); + for (const RawConditionalRemove & removal : removals) + results.push_back(remove(removal.key, removal.expected_token, access)); + return results; + } + private: std::set blob_keys; size_t k_overlap = 1; @@ -184,7 +238,8 @@ PoolPtr openPoolWithIoConcurrency( uint64_t gc_shards = 1, uint64_t outcome_entry_budget = 0, CasEventSink event_sink = {}, - std::optional refuse_at_for_test = std::nullopt) + std::optional refuse_at_for_test = std::nullopt, + uint64_t gc_bulk_delete_chunk_keys = kBulkDeleteMaxKeys) { PoolConfig config{.pool_prefix = "p", .server_root_id = "test"}; config.gc_io_concurrency = concurrency; @@ -192,6 +247,8 @@ PoolPtr openPoolWithIoConcurrency( config.gc_round_outcome_entry_budget = outcome_entry_budget; config.event_sink = std::move(event_sink); config.gc_io_pool_refuse_at_for_test = refuse_at_for_test; + config.gc_bulk_delete_chunk_keys = gc_bulk_delete_chunk_keys; + config.skip_access_check = true; return Pool::open(std::move(backend), std::move(config)); } @@ -358,8 +415,62 @@ TEST(CASGCRedeleteConcurrency, ParallelRedeleteReclaimsEveryBlob) EXPECT_EQ(redeleted, kBlobs); EXPECT_EQ(deleted, kBlobs); EXPECT_EQ(redelete_failed, 0u); - EXPECT_EQ(rows->total("jobs_scheduled"), kBlobs); - EXPECT_EQ(rows->total("jobs_failed"), 0u); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), kBlobs); + EXPECT_EQ(rows->total("delete_chunk_jobs_scheduled"), kBlobs); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 0u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 0u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); +} + +TEST(CASGCRedeleteConcurrency, ConditionalDeleteBatchesAreRecorded) +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + auto store = openPoolWithIoConcurrency(backend, 4, 1, 0, {}, std::nullopt, 2); + Gc gc(store, kGc); + publishThenDrop(*backend, store, gc); + ASSERT_NO_FATAL_FAILURE(driveUntilAllGraduated(*backend, gc, store, kReplacedBlob)); + + auto rows = std::make_shared(); + gc.setPhaseSink(recordPendingDeletes(rows)); + const RoundReport rep = runRegularRoundReclaiming(gc); + store->renewWatermarkOnce(); + + EXPECT_EQ(rep.deleted, kBlobs); + ASSERT_EQ(rows->rows.size(), 1u); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), kBlobs); + EXPECT_EQ(rows->total("delete_chunk_jobs_scheduled"), 3u); + EXPECT_EQ(rows->total("delete_chunks"), 3u); + ASSERT_TRUE(rows->lastRowHas("read_blob_jobs_failed")); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_jobs_failed")); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 0u); + ASSERT_TRUE(rows->lastRowHas("read_blob_job_schedule_failures")); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 0u); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_job_schedule_failures")); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); +} + +TEST(CASGCRedeleteConcurrency, DeleteWaveEnqueueRefusalWaitsForSubmittedChunks) +{ + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + auto store = openPoolWithIoConcurrency(backend, 4, 1, 0, {}, kBlobs + 1, 2); + Gc gc(store, kGc); + publishThenDrop(*backend, store, gc); + ASSERT_NO_FATAL_FAILURE(driveUntilAllGraduated(*backend, gc, store, kReplacedBlob)); + + RoundReport progress; + EXPECT_THROW( + gc.runRegularRound({}, /*allow_steal*/ true, UniversePolicy::Authoritative, &progress), + DB::Exception); + store->renewWatermarkOnce(); + + EXPECT_EQ(progress.redeleted, 2u); + EXPECT_EQ(progress.redelete_failed, 1u); + EXPECT_EQ(runRegularRoundReclaiming(gc).redeleted, kBlobs); + EXPECT_TRUE(allBlobsAbsent(*backend, *store)); } TEST(CASGCRedeleteConcurrency, BlobDeletesActuallyOverlap) @@ -402,8 +513,19 @@ TEST(CASGCRedeleteConcurrency, ConcurrencyOneDeletesSequentially) EXPECT_TRUE(allBlobsAbsent(*backend, *store)); EXPECT_EQ(backend->blobRemoves(), kBlobs); EXPECT_EQ(backend->peakInFlight(), 1u); - ASSERT_TRUE(rows->lastRowHas("jobs_scheduled")); - EXPECT_EQ(rows->total("jobs_scheduled"), 0u) << "concurrency 1 must not use the pool"; + ASSERT_TRUE(rows->lastRowHas("read_blob_jobs_scheduled")); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_jobs_scheduled")); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), 0u) << "concurrency 1 must not use the pool"; + EXPECT_EQ(rows->total("delete_chunk_jobs_scheduled"), 0u); + EXPECT_EQ(rows->total("delete_chunks"), kBlobs); + ASSERT_TRUE(rows->lastRowHas("read_blob_jobs_failed")); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_jobs_failed")); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 0u); + ASSERT_TRUE(rows->lastRowHas("read_blob_job_schedule_failures")); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 0u); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_job_schedule_failures")); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); } TEST(CASGCRedeleteConcurrency, WorkerRemoveFaultKeepsSiblingOutcomesAndPoolAlive) @@ -505,40 +627,6 @@ TEST(CASGCRedeleteConcurrency, SameOutcomesAtConcurrencyOneAndFour) namespace { -class RemoveRaceBackend : public InMemoryBackend -{ -public: - void armAgainstOtherThreads(String key) - { - raced_key = std::move(key); - owner = std::this_thread::get_id(); - armed.store(true); - } - - bool racedOnWorker() const { return raced_on_worker.load(); } - - bool replacementCommitted() const { return replacement_committed.load(); } - - RawRemoval remove(const String & key, const String & expected_value, TransportAccess & access) override - { - if (armed.load() && key == raced_key && armed.exchange(false)) - { - raced_on_worker.store(std::this_thread::get_id() != owner); - const std::optional current = InMemoryBackend::read(key, access); - if (current) - replacement_committed.store(InMemoryBackend::write(key, current->bytes, current->value, access).has_value()); - } - return InMemoryBackend::remove(key, expected_value, access); - } - -private: - String raced_key; - std::thread::id owner; - std::atomic armed{false}; - std::atomic raced_on_worker{false}; - std::atomic replacement_committed{false}; -}; - BlobRef blobRefOf(uint64_t blob) { return BlobRef{BlobHashAlgo::CityHash128, BlobDigest::fromU128(blobHash(blob))}; @@ -625,18 +713,45 @@ TEST(CASGCRedeleteConcurrency, ReplacedBlobInsideParallelBatchSurvivesWhileSibli TEST(CASGCRedeleteConcurrency, BlobReplacedBetweenHeadAndDeleteSurvivesWhileSiblingsDelete) { - auto backend = std::make_shared(); - auto store = openPoolWithIoConcurrency(backend, 4); + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + auto store = openPoolWithIoConcurrency(backend, 4, 1, 0, {}, std::nullopt, 2); Gc gc(store, kGc); publishThenDrop(*backend, store, gc); ASSERT_NO_FATAL_FAILURE(driveUntilAllGraduated(*backend, gc, store, kReplacedBlob)); - backend->armAgainstOtherThreads(blobKeyOf(*store, kReplacedBlob)); + const String raced_key = blobKeyOf(*store, kReplacedBlob); + const auto blob_keys = allBlobKeys(*store); + const auto caller = std::this_thread::get_id(); + std::optional replacement_token; + backend->delete_chunk_hook = [&](const std::vector & removals, TransportAccess & access) + { + for (const auto & removal : removals) + { + if (removal.key != raced_key) + continue; + EXPECT_NE(std::this_thread::get_id(), caller); + ASSERT_EQ(removals.size(), 2u); + EXPECT_TRUE(std::any_of(removals.begin(), removals.end(), [&](const auto & sibling) + { + return sibling.key != raced_key && blob_keys.contains(sibling.key); + })); + const auto original = backend->read(raced_key, access); + ASSERT_TRUE(original); + const auto written = backend->write(raced_key, original->bytes, removal.expected_token, access); + ASSERT_TRUE(written.has_value()); + replacement_token = *written; + } + }; const RoundReport rep = runRegularRoundReclaiming(gc); + backend->delete_chunk_hook = {}; store->renewWatermarkOnce(); - EXPECT_TRUE(backend->replacementCommitted()) << "the race never replaced the blob, so the 412 path is untested"; - EXPECT_TRUE(backend->racedOnWorker()) << "the raced delete must have run on a pool worker"; + ASSERT_TRUE(replacement_token); + OperationForTest op(*backend); + const auto survivor = (*op).head(raced_key, Retry::once()); + ASSERT_TRUE(survivor); + EXPECT_EQ(PersistedEtag::capture(survivor->etag).value, *replacement_token); expectOnlyReplacedBlobSurvives(*backend, *store, rep); } @@ -707,11 +822,11 @@ void expectBodiesPresentOnlyFor(Backend & backend, const Pool & store, const std EXPECT_EQ(blobAbsent(backend, store.layout(), blobHash(b)), !present.contains(b)) << "blob " << b; } -struct ApplyHookState +struct RecordHookState { std::atomic armed{false}; BlobRef target{}; - size_t applied = 0; + size_t recorded = 0; }; } @@ -761,9 +876,12 @@ TEST(CASGCRedeleteConcurrency, TwoWorkerFailuresAreEachCountedAndTheRoundThrowsO EXPECT_EQ(progress.redeleted, kBlobs - 2); EXPECT_EQ(progress.redelete_failed, 2u); ASSERT_EQ(rows->rows.size(), 1u); - ASSERT_TRUE(rows->lastRowHas("jobs_failed")) << "the phase metrics must survive the round's exception"; - EXPECT_EQ(rows->total("jobs_scheduled"), kBlobs); - EXPECT_EQ(rows->total("jobs_failed"), 2u); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_jobs_failed")) << "the phase metrics must survive the round's exception"; + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), kBlobs); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 2u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 0u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); expectBodiesPresentOnlyFor(*backend, *store, {kFaultedBlob, kReplacedBlob}); const RoundReport next = runRegularRoundReclaiming(gc); @@ -773,6 +891,142 @@ TEST(CASGCRedeleteConcurrency, TwoWorkerFailuresAreEachCountedAndTheRoundThrowsO EXPECT_TRUE(allBlobsAbsent(*backend, *store)); } +TEST(CASGCRedeleteConcurrency, FailuresAreReportedAfterTheirWaveAndRethrowLowestBlobIndex) +{ + for (const uint64_t concurrency : {1, 4}) + { + SCOPED_TRACE(concurrency); + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + auto store = openPoolWithIoConcurrency(backend, concurrency, 1, 0, {}, std::nullopt, 2); + Gc gc(store, kGc); + publishThenDrop(*backend, store, gc); + ASSERT_NO_FATAL_FAILURE(driveUntilAllGraduated(*backend, gc, store, kFaultedBlob)); + const auto preview = previewPendingDeletes(gc); + ASSERT_EQ(preview.size(), kBlobs); + + /// Identical messages require the GC diagnostic to identify the operation itself. + const auto read_error = std::make_exception_ptr(std::runtime_error("injected job fault")); + const auto delete_error = std::make_exception_ptr(std::runtime_error("injected job fault")); + backend->blob_keys = allBlobKeys(*store); + backend->read_blob_hook = [&](const String & key) + { + if (key == preview.back().key) + std::rethrow_exception(read_error); + }; + backend->delete_chunk_hook = [&](const std::vector & removals, TransportAccess &) + { + if (removals.front().key == preview.front().key) + { + EXPECT_EQ(removals.size(), 2u); + std::rethrow_exception(delete_error); + } + }; + + ScopedLogCapture capture("CasGc", "warning"); + auto rows = std::make_shared(); + gc.setPhaseSink(recordPendingDeletes(rows)); + const auto failed_before = failedCounter(); + RoundReport progress; + std::exception_ptr error; + try + { + gc.runRegularRound({}, /*allow_steal*/ true, UniversePolicy::Authoritative, &progress); + } + catch (...) + { + error = std::current_exception(); + } + backend->read_blob_hook = {}; + backend->delete_chunk_hook = {}; + + const String log = capture.captured(); + const size_t read_warning = log.find("CAS gc: read blob job failed"); + const size_t delete_warning = log.find("CAS gc: delete chunk job failed"); + ASSERT_NE(read_warning, String::npos); + ASSERT_NE(delete_warning, String::npos); + EXPECT_LT(log.rfind("Test blob read finished"), read_warning); + EXPECT_LT(read_warning, log.find("Test delete chunk started")); + EXPECT_LT(log.rfind("Test delete chunk finished"), delete_warning); + EXPECT_EQ(capture.count("CAS gc: read blob job failed"), 1u); + EXPECT_EQ(capture.count("CAS gc: delete chunk job failed"), 2u); + EXPECT_EQ(error, delete_error) << "the lower blob index wins even though its wave is logged later"; + EXPECT_EQ(progress.redeleted, 3u); + EXPECT_EQ(progress.redelete_failed, 3u); + EXPECT_EQ(failedCounter() - failed_before, 3u); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 1u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 1u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 0u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); + EXPECT_EQ(rows->total("delete_chunks"), 3u); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), concurrency == 1 ? 0u : kBlobs); + EXPECT_EQ(rows->total("delete_chunk_jobs_scheduled"), concurrency == 1 ? 0u : 3u); + EXPECT_EQ(previewPendingDeletes(gc).size(), kBlobs); + for (size_t i = 0; i < preview.size(); ++i) + { + OperationForTest op(*backend); + EXPECT_EQ((*op).head(preview[i].key, Retry::once()).has_value(), i < 2 || i == 5); + } + store->renewWatermarkOnce(); + EXPECT_EQ(runRegularRoundReclaiming(gc).redeleted, kBlobs); + EXPECT_TRUE(allBlobsAbsent(*backend, *store)); + } +} + +TEST(CASGCRedeleteConcurrency, EnqueueFailuresIdentifyWaveAndUnscheduledBlobCount) +{ + struct Case + { + bool delete_wave; + size_t refused_job; + size_t deleted; + size_t unscheduled; + size_t read_blob_jobs_scheduled; + size_t delete_chunk_jobs_scheduled; + }; + for (const Case test : {Case{false, 0, 0, 6, 0, 0}, Case{false, 3, 3, 3, 3, 0}, Case{true, 0, 0, 6, 6, 0}, Case{true, 1, 4, 2, 6, 1}}) + { + SCOPED_TRACE(test.delete_wave); + SCOPED_TRACE(test.refused_job); + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + const size_t refused_index = (test.delete_wave ? kBlobs : 0) + test.refused_job; + auto store = openPoolWithIoConcurrency(backend, 4, 1, 0, {}, refused_index, 4); + Gc gc(store, kGc); + publishThenDrop(*backend, store, gc); + ASSERT_NO_FATAL_FAILURE(driveUntilAllGraduated(*backend, gc, store, kFaultedBlob)); + + ScopedLogCapture capture("CasGc", "warning"); + auto rows = std::make_shared(); + gc.setPhaseSink(recordPendingDeletes(rows)); + const auto failed_before = failedCounter(); + RoundReport progress; + expectThrowsCode(DB::ErrorCodes::CANNOT_SCHEDULE_TASK, [&] + { + gc.runRegularRound({}, /*allow_steal*/ true, UniversePolicy::Authoritative, &progress); + }); + EXPECT_EQ(capture.count(test.delete_wave + ? "CAS gc: scheduling delete chunk job failed" : "CAS gc: scheduling read blob job failed"), 1u); + EXPECT_NE(capture.captured().find(std::to_string(test.unscheduled) + (test.delete_wave + ? " blob deletions were not submitted" : " blob reads were not submitted")), String::npos); + EXPECT_EQ(progress.redeleted, test.deleted); + EXPECT_EQ(progress.redelete_failed, 1u); + EXPECT_EQ(failedCounter() - failed_before, 1u); + ASSERT_TRUE(rows->lastRowHas("read_blob_jobs_scheduled")); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_jobs_scheduled")); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), test.read_blob_jobs_scheduled); + EXPECT_EQ(rows->total("delete_chunk_jobs_scheduled"), test.delete_chunk_jobs_scheduled); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 0u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), test.delete_wave ? 0u : 1u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), test.delete_wave ? 1u : 0u); + EXPECT_EQ(previewPendingDeletes(gc).size(), kBlobs); + store->renewWatermarkOnce(); + EXPECT_EQ(runRegularRoundReclaiming(gc).redeleted, kBlobs); + EXPECT_TRUE(allBlobsAbsent(*backend, *store)); + } +} + TEST(CASGCRedeleteConcurrency, WorkerHeadFaultIsCountedAndRetriedNextRound) { auto backend = std::make_shared(); @@ -794,8 +1048,11 @@ TEST(CASGCRedeleteConcurrency, WorkerHeadFaultIsCountedAndRetriedNextRound) EXPECT_EQ(progress.redeleted, kBlobs - 1); EXPECT_EQ(progress.redelete_failed, 1u); ASSERT_EQ(rows->rows.size(), 1u); - EXPECT_EQ(rows->total("jobs_scheduled"), kBlobs); - EXPECT_EQ(rows->total("jobs_failed"), 1u); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), kBlobs); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 1u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 0u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 0u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); expectBodiesPresentOnlyFor(*backend, *store, {kFaultedBlob}); const RoundReport next = runRegularRoundReclaiming(gc); @@ -850,9 +1107,12 @@ TEST(CASGCRedeleteConcurrency, SchedulingFailureAttemptsNothingAndTheNextRoundRe EXPECT_EQ(progress.redelete_failed, 1u); EXPECT_EQ(failedCounter() - failed_before, 1u); ASSERT_EQ(rows->rows.size(), 1u); - ASSERT_TRUE(rows->lastRowHas("jobs_scheduled")); - EXPECT_EQ(rows->total("jobs_scheduled"), 0u); - EXPECT_EQ(rows->total("jobs_failed"), 1u); + ASSERT_TRUE(rows->lastRowHas("read_blob_jobs_scheduled")); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), 0u); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 0u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 1u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); expectBodiesPresentOnlyFor(*backend, *store, {1, 2, 3, 4, 5, 6}); const RoundReport next = runRegularRoundReclaiming(gc); @@ -890,8 +1150,11 @@ TEST(CASGCRedeleteConcurrency, EnqueueRefusalYieldsToLowerIndexWorkerFailure) EXPECT_EQ(progress.redelete_failed, 2u); EXPECT_EQ(failedCounter() - failed_before, 2u); - EXPECT_EQ(rows->total("jobs_scheduled"), 3u); - EXPECT_EQ(rows->total("jobs_failed"), 2u); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), 3u); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 1u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 1u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); EXPECT_NE(message.find(first_preview_key), String::npos); EXPECT_EQ(message.find("Injected CAS GC re-delete enqueue refusal"), String::npos); EXPECT_TRUE(backend->allFired()); @@ -943,8 +1206,13 @@ TEST(CASGCRedeleteConcurrency, SingletonWorkerFailureIsCountedWithoutScheduling) EXPECT_TRUE(backend->allFired()); EXPECT_EQ(progress.redelete_failed, 1u); EXPECT_EQ(failedCounter() - failed_before, 1u); - EXPECT_EQ(rows->total("jobs_scheduled"), 0u); - EXPECT_EQ(rows->total("jobs_failed"), 1u); + EXPECT_EQ(rows->total("read_blob_jobs_scheduled"), 0u); + ASSERT_TRUE(rows->lastRowHas("delete_chunk_jobs_scheduled")); + EXPECT_EQ(rows->total("delete_chunk_jobs_scheduled"), 0u); + EXPECT_EQ(rows->total("read_blob_jobs_failed"), 0u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 1u); + EXPECT_EQ(rows->total("read_blob_job_schedule_failures"), 0u); + EXPECT_EQ(rows->total("delete_chunk_job_schedule_failures"), 0u); } TEST(CASGCRedeleteConcurrency, ShardedPoolDeletesEveryShard) @@ -990,22 +1258,22 @@ TEST(CASGCRedeleteConcurrency, ShardedPoolDeletesEveryShard) EXPECT_EQ(shards_with_rows, expected_shards); } -TEST(CASGCRedeleteConcurrency, ApplyPhaseFaultLeavesEveryEntryForTheNextRound) +TEST(CASGCRedeleteConcurrency, RecordPhaseFaultLeavesEveryEntryForTheNextRound) { - auto state = std::make_shared(); + auto state = std::make_shared(); auto backend = std::make_shared(); PoolConfig config{.pool_prefix = "p", .server_root_id = "test"}; config.gc_io_concurrency = 4; - config.gc_redelete_apply_hook_for_test = [state](const BlobRef & ref) + config.gc_redelete_record_hook_for_test = [state](const BlobRef & ref) { if (!state->armed.load()) return; if (ref == state->target) { state->armed.store(false); - throw std::runtime_error("injected apply fault"); + throw std::runtime_error("injected record fault"); } - ++state->applied; + ++state->recorded; }; auto store = Pool::open(backend, std::move(config)); Gc gc(store, kGc); @@ -1027,10 +1295,10 @@ TEST(CASGCRedeleteConcurrency, ApplyPhaseFaultLeavesEveryEntryForTheNextRound) } store->renewWatermarkOnce(); - EXPECT_NE(message.find("injected apply fault"), String::npos) << message; - EXPECT_EQ(progress.redeleted, state->applied); - EXPECT_EQ(failedCounter() - failed_before, 0u) << "a failed apply is not a failed delete"; - EXPECT_TRUE(allBlobsAbsent(*backend, *store)) << "every delete finished before the apply walk started"; + EXPECT_NE(message.find("injected record fault"), String::npos) << message; + EXPECT_EQ(progress.redeleted, state->recorded); + EXPECT_EQ(failedCounter() - failed_before, 0u) << "a failed record is not a failed delete"; + EXPECT_TRUE(allBlobsAbsent(*backend, *store)) << "every delete finished before the record walk started"; const RoundReport next = runRegularRoundReclaiming(gc); EXPECT_EQ(next.redeleted, kBlobs); @@ -1105,6 +1373,138 @@ TEST(CASGCRedeleteConcurrency, BlobDeleteEventsMatchAppliedEntriesInAFailedRound EXPECT_EQ(next_round[blobKeyOf(*store, b)], b == kFaultedBlob ? "deleted" : "absent") << "blob " << b; } +TEST(CASGCRedeleteConcurrency, ExhaustedMultiKeyRetryKeepsPublishedPendingState) +{ + constexpr size_t chunk_size = 3; + auto events = std::make_shared(); + auto backend = std::make_shared(); + backend->pinConditionalBatchDeleteSupport(true); + /// Run chunks inline so expiring one request's virtual deadline cannot expire a sibling in flight. + auto store = openPoolWithIoConcurrency(backend, 1, 1, 0, events->sink(), std::nullopt, chunk_size); + const auto clock = VirtualRetryClock::installOn(store); + Gc gc(store, kGc); + ASSERT_NO_FATAL_FAILURE(publishThenDrop(*backend, store, gc)); + ASSERT_NO_FATAL_FAILURE(driveUntilAllGraduated(*backend, gc, store, kFaultedBlob)); + const auto pending = previewPendingDeletes(gc); + ASSERT_EQ(pending.size(), 2 * chunk_size); + OperationForTest op(*backend); + const auto & layout = store->layout(); + const auto read_state = [&] { return decodeGcState((*op).read(layout.gcStateKey(), Retry::once()).value().bytes); }; + const auto before = read_state(); + const auto runs_before = runsForShard(*backend, layout, 0); + ASSERT_FALSE(runs_before.empty()); + + size_t physical_attempts = 0; + std::optional replacement_token; + backend->delete_chunk_hook = [&](const std::vector & removals, TransportAccess & access) + { + ++physical_attempts; + ASSERT_EQ(removals.size(), chunk_size); + if (removals.front().key != pending[0].key) + return; + for (size_t i = 0; i < chunk_size; ++i) + { + EXPECT_EQ(removals[i].key, pending[i].key); + EXPECT_EQ(removals[i].expected_token, pending[i].token.value); + } + if (access.attemptNo() == 1) + { + /// A was deleted when the response was lost. A writer replaced B, and C is still eligible. + ASSERT_EQ(backend->remove(removals[0].key, removals[0].expected_token, access), Backend::RawRemoval::Removed); + const auto original = backend->read(removals[1].key, access); + ASSERT_TRUE(original); + const auto written = backend->write(removals[1].key, original->bytes, removals[1].expected_token, access); + ASSERT_TRUE(written.has_value()); + replacement_token = *written; + } + else + { + clock->advance(store->poolConfig().cas_request_budget.request_window_ms); + } + throw Poco::TimeoutException("lost conditional delete response"); + }; + + auto rows = std::make_shared(); + gc.setPhaseSink(recordPendingDeletes(rows)); + const auto failed_before = failedCounter(); + RoundReport progress; + expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "policy deadline after 2 attempt(s)", [&] + { + gc.runRegularRound({}, /*allow_steal*/ true, UniversePolicy::Authoritative, &progress); + }); + backend->delete_chunk_hook = {}; + + ASSERT_TRUE(replacement_token); + EXPECT_EQ(physical_attempts, 3u); + ASSERT_EQ(rows->rows.size(), 1u); + EXPECT_EQ(rows->total("delete_chunks"), 2u); + EXPECT_EQ(rows->total("delete_chunk_jobs_failed"), 1u); + EXPECT_EQ(progress.redelete_failed, chunk_size); + EXPECT_EQ(failedCounter() - failed_before, chunk_size); + EXPECT_EQ(progress.redeleted, chunk_size); + /// GC copies per-outcome totals into the report after writing the outcome logs. This round threw before that step. + EXPECT_EQ(progress.deleted, 0u); + EXPECT_EQ(progress.absent, 0u); + EXPECT_EQ(progress.replaced, 0u); + + const auto after = read_state(); + EXPECT_EQ(after.round, before.round); + EXPECT_EQ(after.snap_generation, before.snap_generation); + EXPECT_EQ(after.snap_attempt, before.snap_attempt); + EXPECT_EQ(runsForShard(*backend, layout, 0), runs_before); + const auto still_pending = previewPendingDeletes(gc); + ASSERT_EQ(still_pending.size(), pending.size()); + const auto [reported, count] = events->take(); + EXPECT_EQ(count, chunk_size); + std::map expected_events; + for (size_t i = 0; i < pending.size(); ++i) + { + const auto & entry = pending[i]; + EXPECT_EQ(still_pending[i].key, entry.key); + EXPECT_EQ(still_pending[i].token.value, entry.token.value); + const auto body = (*op).head(entry.key, Retry::once()); + EXPECT_EQ(body.has_value(), i == 1 || i == 2) << entry.key; + EXPECT_EQ((*op).head(layout.blobMetaKey(entry.ref), Retry::once()).has_value(), i < chunk_size) << entry.key; + if (i >= chunk_size) + expected_events.emplace(entry.key, "deleted"); + } + EXPECT_EQ(reported, expected_events); + + rows->rows.clear(); + store->renewWatermarkOnce(); + const RoundReport next = runRegularRoundReclaiming(gc); + EXPECT_EQ(next.redeleted, pending.size()); + EXPECT_EQ(next.absent, chunk_size + 1); + EXPECT_EQ(next.replaced, 1u); + EXPECT_EQ(next.deleted, 1u); + EXPECT_EQ(next.redelete_failed, 0u); + EXPECT_EQ(rows->total("delete_chunks"), 1u); + EXPECT_EQ(read_state().round, before.round + 1); + EXPECT_TRUE(previewPendingDeletes(gc).empty()); + expected_events.clear(); + for (size_t i = 0; i < pending.size(); ++i) + { + const auto & entry = pending[i]; + const auto body = (*op).head(entry.key, Retry::once()); + if (i == 1) + { + ASSERT_TRUE(body); + EXPECT_EQ(PersistedEtag::capture(body->etag).value, *replacement_token); + } + else + { + EXPECT_FALSE(body) << entry.key; + } + EXPECT_EQ((*op).head(layout.blobMetaKey(entry.ref), Retry::once()).has_value(), i == 1) << entry.key; + expected_events.emplace(entry.key, "absent"); + } + expected_events.at(pending[1].key) = "replaced"; + expected_events.at(pending[2].key) = "deleted"; + const auto [recovered, recovered_count] = events->take(); + EXPECT_EQ(recovered_count, pending.size()); + EXPECT_EQ(recovered, expected_events); +} + TEST(CASGCRedeleteConcurrency, OutcomeBudgetCapsAuditRowsButNotDeletes) { constexpr uint64_t budget = 3; @@ -1164,3 +1564,89 @@ TEST(CASGCRedeleteConcurrency, RoundsWithoutRedeletesWriteNoOutcomeLog) ADD_FAILURE() << "a round without redeletes or spares wrote the outcome log " << key << " with " << rows.size() << " row(s)"; EXPECT_TRUE(logs.empty()); } + +namespace +{ + +Gc::RedeleteBlobState liveRedeleteBlobState(CasOperation & op, const String & key) +{ + Gc::RedeleteBlobState redelete_blob_state; + redelete_blob_state.blob_key = key; + redelete_blob_state.live_token = std::get(op.create(key, "v", Retry::once())).etag; + return redelete_blob_state; +} + +Gc::RedeleteBlobState settledRedeleteBlobState(const String & key, const Removal removal_result) +{ + Gc::RedeleteBlobState redelete_blob_state; + redelete_blob_state.blob_key = key; + redelete_blob_state.removal_result = removal_result; + return redelete_blob_state; +} + +std::vector> chunkBlobStateIndices(const std::vector & chunks) +{ + std::vector> blob_state_indices; + for (const auto & chunk : chunks) + blob_state_indices.push_back(chunk.blob_state_indices); + return blob_state_indices; +} + +} + +TEST(CASGCConditionalDeleteChunks, OnlyLiveTokensWithoutErrorsAreChunked) +{ + auto backend = std::make_shared(); + OperationForTest op(backend); + std::vector redelete_blob_states; + redelete_blob_states.push_back(liveRedeleteBlobState(*op, "a")); + redelete_blob_states.push_back(settledRedeleteBlobState("gone", Removal::Gone)); + redelete_blob_states.push_back(liveRedeleteBlobState(*op, "b")); + redelete_blob_states.push_back(settledRedeleteBlobState("replaced", Removal::Mismatch)); + redelete_blob_states.push_back(liveRedeleteBlobState(*op, "failed")); + redelete_blob_states.back().error = std::make_exception_ptr(std::runtime_error("head failed")); + redelete_blob_states.emplace_back(); + redelete_blob_states.push_back(liveRedeleteBlobState(*op, "c")); + + const auto chunks = Gc::prepareConditionalDeleteChunks(redelete_blob_states, 10); + + ASSERT_EQ(chunkBlobStateIndices(chunks), (std::vector>{{0, 2, 6}})); + const auto & conditional_removals = chunks.front().conditional_removals; + ASSERT_EQ(conditional_removals.size(), 3u); + for (size_t pos = 0; pos < conditional_removals.size(); ++pos) + { + const auto & redelete_blob_state = redelete_blob_states[chunks.front().blob_state_indices[pos]]; + EXPECT_EQ(conditional_removals[pos].key, redelete_blob_state.blob_key); + EXPECT_EQ(conditional_removals[pos].expected_token, *redelete_blob_state.live_token); + } +} + +TEST(CASGCConditionalDeleteChunks, ChunksAreBoundedByChunkKeys) +{ + auto backend = std::make_shared(); + OperationForTest op(backend); + std::vector redelete_blob_states; + for (const String key : {"a", "b", "c", "d", "e"}) + redelete_blob_states.push_back(liveRedeleteBlobState(*op, key)); + + EXPECT_EQ( + chunkBlobStateIndices(Gc::prepareConditionalDeleteChunks(redelete_blob_states, 2)), + (std::vector>{{0, 1}, {2, 3}, {4}})); + EXPECT_EQ( + chunkBlobStateIndices(Gc::prepareConditionalDeleteChunks(redelete_blob_states, 1)), + (std::vector>{{0}, {1}, {2}, {3}, {4}})); + EXPECT_EQ( + chunkBlobStateIndices(Gc::prepareConditionalDeleteChunks(redelete_blob_states, 5)), + (std::vector>{{0, 1, 2, 3, 4}})); +} + +TEST(CASGCConditionalDeleteChunks, NothingEligibleGivesNoChunks) +{ + std::vector redelete_blob_states; + redelete_blob_states.push_back(settledRedeleteBlobState("gone", Removal::Gone)); + redelete_blob_states.push_back(settledRedeleteBlobState("replaced", Removal::Mismatch)); + redelete_blob_states.emplace_back(); + + EXPECT_TRUE(Gc::prepareConditionalDeleteChunks(redelete_blob_states, 2).empty()); + EXPECT_TRUE(Gc::prepareConditionalDeleteChunks({}, 2).empty()); +} diff --git a/src/Disks/tests/gtest_cas_probe.cpp b/src/Disks/tests/gtest_cas_probe.cpp index 1c189ffb59244..67df2bcfb98ca 100644 --- a/src/Disks/tests/gtest_cas_probe.cpp +++ b/src/Disks/tests/gtest_cas_probe.cpp @@ -7,6 +7,20 @@ #include #include #include +#include + +#include "config.h" +#if USE_AWS_S3 +#include +#endif + +#include + +namespace DB::ErrorCodes +{ +extern const int NETWORK_ERROR; +extern const int NOT_IMPLEMENTED; +} using namespace DB::Cas; @@ -33,6 +47,163 @@ TEST(CASProbe, PassesOnEnforcingBackend) EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()); // probe cleans up after itself } +namespace +{ + +class IgnoresConditionalBatchTokensBackend : public InMemoryBackend +{ +public: + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + std::vector results; + results.reserve(removals.size()); + for (const RawConditionalRemove & removal : removals) + { + const auto current = InMemoryBackend::head(removal.key, access); + results.push_back(current ? InMemoryBackend::remove(removal.key, current->value, access) : RawRemoval::Gone); + } + return results; + } +}; + +class NoOpConditionalBatchDeleteBackend : public InMemoryBackend +{ +public: + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + std::vector results; + results.reserve(removals.size()); + for (const RawConditionalRemove & removal : removals) + { + const auto current = InMemoryBackend::head(removal.key, access); + if (!current) + results.push_back(RawRemoval::Gone); + else if (current->value != removal.expected_token) + results.push_back(RawRemoval::Mismatch); + else + results.push_back(RawRemoval::Removed); + } + return results; + } +}; + +class ReportsAbsentKeyAsRemovedBackend : public InMemoryBackend +{ +public: + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + auto results = InMemoryBackend::removeManyByConditional(removals, access); + EXPECT_EQ(results, (std::vector{RawRemoval::Removed, RawRemoval::Mismatch, RawRemoval::Removed, RawRemoval::Gone})); + for (auto & result : results) + if (result == RawRemoval::Gone) + result = RawRemoval::Removed; + return results; + } +}; + +class FailingConditionalBatchBackend : public InMemoryBackend +{ +public: + explicit FailingConditionalBatchBackend(const std::exception_ptr & failure_) : failure(failure_) + { + } + + std::vector removeManyByConditional(const std::vector &, TransportAccess &) override + { + ++batch_attempts; + std::rethrow_exception(failure); + } + + size_t batch_attempts = 0; + +private: + std::exception_ptr failure; +}; + +} + +TEST(CASProbe, ReportsConditionalBatchCapability) +{ + InMemoryBackend enforcing; + auto enforcing_requests = makeRequests(enforcing); + auto enforcing_op = enforcing_requests.admit(); + EXPECT_TRUE(runCapabilityProbe(enforcing_op, "p/.cas_probe").conditional_batch_delete_supported); + + IgnoresConditionalBatchTokensBackend ignores; + auto ignores_requests = makeRequests(ignores); + auto ignores_op = ignores_requests.admit(); + EXPECT_FALSE(runCapabilityProbe(ignores_op, "p/.cas_probe").conditional_batch_delete_supported); +} + +TEST(CASProbe, RejectsConditionalBatchSuccessWithoutDeletion) +{ + NoOpConditionalBatchDeleteBackend backend; + auto requests = makeRequests(backend); + auto op = requests.admit(); + + EXPECT_FALSE(runCapabilityProbe(op, "p/.cas_probe").conditional_batch_delete_supported); + EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()); +} + +TEST(CASProbe, RequiresGoneForAnAlreadyDeletedKeyInTheBatch) +{ + ReportsAbsentKeyAsRemovedBackend backend; + auto requests = makeRequests(backend); + auto op = requests.admit(); + + EXPECT_FALSE(runCapabilityProbe(op, "p/.cas_probe").conditional_batch_delete_supported); +} + +TEST(CASProbe, ExplicitlyUnsupportedConditionalBatchReturnsFalseAndCleansUp) +{ + FailingConditionalBatchBackend backend(std::make_exception_ptr( + DB::Exception(DB::ErrorCodes::NOT_IMPLEMENTED, "conditional batch delete unsupported"))); + auto requests = makeRequests(backend); + auto op = requests.admit(); + + EXPECT_FALSE(runCapabilityProbe(op, "p/.cas_probe").conditional_batch_delete_supported); + EXPECT_EQ(backend.batch_attempts, 1u); + EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()); +} + +#if USE_AWS_S3 +TEST(CASProbe, ConditionalBatchAccessDeniedPropagatesAndCleansUp) +{ + FailingConditionalBatchBackend backend(std::make_exception_ptr( + DB::S3Exception("conditional batch delete denied", Aws::S3::S3Errors::ACCESS_DENIED))); + auto requests = makeRequests(backend); + auto op = requests.admit(); + + try + { + runCapabilityProbe(op, "p/.cas_probe"); + ADD_FAILURE() << "expected S3 access denial"; + } + catch (const DB::S3Exception & e) + { + EXPECT_EQ(e.getS3ErrorCode(), Aws::S3::S3Errors::ACCESS_DENIED); + } + EXPECT_EQ(backend.batch_attempts, 1u); + EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()); +} +#endif + +TEST(CASProbe, ConditionalBatchTimeoutPropagatesAfterRetriesAndCleansUp) +{ + DB::Cas::tests::FakeClock clock; + auto backend = std::make_shared(std::make_exception_ptr(Poco::TimeoutException())); + CasRequests requests(backend, Fence::open(), clock.nowFn(), clock.sleepFn(), nullptr, kDefaultRequestWindowMs); + auto op = requests.admit(); + + DB::Cas::tests::expectThrowsCodeWithMessage(DB::ErrorCodes::NETWORK_ERROR, "removeManyByConditional", + [&] { runCapabilityProbe(op, "p/.cas_probe"); }); + EXPECT_GT(backend->batch_attempts, 1u); + EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()); +} + TEST(CASProbe, FailsClosedOnNonEnforcingDelete) { auto b = std::make_shared(); @@ -100,6 +271,20 @@ TEST(CASProbe, ConcurrentMountsDoNotCollide) EXPECT_TRUE(probe_op.read("p/_probe/token", Retry::once()).has_value()); } +TEST(CASProbe, PoolPinsConditionalBatchCapabilityOnlyAfterProbe) +{ + auto backend = std::make_shared(); + EXPECT_EQ(backend->conditionalBatchDeleteKeyLimit(), 1u); + + auto store = Pool::open(backend, PoolConfig{.pool_prefix = "p", .server_root_id = "test"}); + EXPECT_EQ(store->poolBackendPtr()->conditionalBatchDeleteKeyLimit(), kBulkDeleteMaxKeys); + + auto skipped_backend = std::make_shared(); + auto skipped = Pool::open(skipped_backend, PoolConfig{ + .pool_prefix = "skip", .server_root_id = "test", .skip_access_check = true}); + EXPECT_EQ(skipped->poolBackendPtr()->conditionalBatchDeleteKeyLimit(), 1u); +} + /// RFC cas-s3-timeout-retry-control: a Native-mode mount over an object storage that does not support /// the SingleAttempt retry profile must never silently proceed under the disk's default (~500-attempt) /// transparent retry policy — see Backend::checkConditionalWriteSingleAttemptSupport. This calls the @@ -235,7 +420,7 @@ TEST(CASProbe, ReorderedProbePassesOnAllThreeDialects) auto op = requests.admit(); EXPECT_NO_THROW(runCapabilityProbe(op, "p/.cas_probe")) << "dialect " << static_cast(dialect); EXPECT_TRUE(op.list("p/.cas_probe", "", 10, Retry::once()).keys.empty()) << "dialect " << static_cast(dialect); - EXPECT_EQ(b.write_reached, 4) << "dialect " << static_cast(dialect); - EXPECT_EQ(b.remove_reached, 2) << "dialect " << static_cast(dialect); + EXPECT_EQ(b.write_reached, 8) << "dialect " << static_cast(dialect); + EXPECT_EQ(b.remove_reached, 3) << "dialect " << static_cast(dialect); } } diff --git a/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp b/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp index d5bbf95a8dbab..5f8a563748905 100644 --- a/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp +++ b/src/Disks/tests/gtest_cas_s3_bulk_delete_fallback.cpp @@ -4,8 +4,10 @@ #if USE_AWS_S3 +#include #include #include +#include #include #include #include @@ -13,6 +15,7 @@ #include #include #include +#include #include #include @@ -36,6 +39,7 @@ namespace DB::ErrorCodes { +extern const int LOGICAL_ERROR; extern const int NOT_IMPLEMENTED; } @@ -80,10 +84,13 @@ class ScriptedS3Server std::lock_guard lock(owner.mutex); owner.methods_seen.push_back(request.getMethod()); } - /// `DeleteObjects` carries a request body (the XML `` payload); leaving it unread on a - /// keep-alive connection makes Poco parse those leftover bytes as the start of the NEXT - /// request once this handler returns, corrupting the very next `DeleteObject` this test expects. - request.stream().ignore(std::numeric_limits::max()); + const std::string body( + std::istreambuf_iterator(request.stream()), + std::istreambuf_iterator()); + { + std::lock_guard lock(owner.mutex); + owner.bodies_seen.push_back(body); + } owner.responder(request, response); } }; @@ -108,6 +115,7 @@ class ScriptedS3Server Responder responder; mutable std::mutex mutex; std::vector methods_seen; + std::vector bodies_seen; public: explicit ScriptedS3Server(Responder responder_) @@ -127,13 +135,19 @@ class ScriptedS3Server std::lock_guard lock(mutex); return static_cast(std::count(methods_seen.begin(), methods_seen.end(), method)); } + + std::string lastBody() const + { + std::lock_guard lock(mutex); + return bodies_seen.empty() ? std::string{} : bodies_seen.back(); + } }; void sendXml(Poco::Net::HTTPServerResponse & response, Poco::Net::HTTPResponse::HTTPStatus status, const std::string & body) { response.setContentType("application/xml"); response.setContentLength(body.size()); - response.setStatus(status); + response.setStatusAndReason(status); auto & out = response.send(); out << body; out.flush(); @@ -199,15 +213,31 @@ void sendDeleteObjectsResult(Poco::Net::HTTPServerResponse & response, const std sendXml(response, Poco::Net::HTTPResponse::HTTP_OK, body); } +void sendConditionalDeleteObjectsResult(Poco::Net::HTTPServerResponse & response) +{ + const std::string body = + "" + "" + "first" + "secondPreconditionFailedstale" + ""; + sendXml(response, Poco::Net::HTTPResponse::HTTP_OK, body); +} + std::shared_ptr makeStorageForTest( - const std::string & endpoint, const DB::S3Capabilities & capabilities, std::optional objects_chunk_size_to_delete = {}) + const std::string & endpoint, + const DB::S3Capabilities & capabilities, + std::optional objects_chunk_size_to_delete = {}, + const DB::S3ObjectStorage::S3CredentialsRefreshCallback & credentials_refresh_callback = {}, + unsigned int max_retries = 0) { - DB::RemoteHostFilter remote_host_filter; + /// The configuration retains this reference when the storage clones its single-attempt client. + static const DB::RemoteHostFilter remote_host_filter; DB::S3::PocoHTTPClientConfiguration cfg = DB::S3::ClientFactory::instance().createClientConfiguration( "us-east-1", remote_host_filter, /* s3_max_redirects = */ 100, - DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = max_retries}, /* s3_slow_all_threads_after_network_error = */ false, /* s3_slow_all_threads_after_retryable_error = */ false, /* enable_s3_requests_logging = */ false, @@ -238,7 +268,7 @@ std::shared_ptr makeStorageForTest( return std::make_shared( std::move(client), std::move(settings), DB::S3::URI(endpoint + "/test-bucket/"), capabilities, - DB::ObjectStorageKeyGeneratorPtr{}, "disk"); + DB::ObjectStorageKeyGeneratorPtr{}, "disk", true, credentials_refresh_callback); } DB::ContextPtr contextForTest() @@ -294,6 +324,233 @@ TEST(S3BulkDeleteFallback, PerKeyErrorsWithinASuccessfulBatchAreUnchanged) EXPECT_EQ(server.countMethod("DELETE"), 0u); } +TEST(CASS3ConditionalBatchDelete, ReturnsPerKeyOutcomesInInputOrder) +{ + (void)contextForTest(); + + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) + { + ASSERT_EQ(request.getMethod(), "POST"); + sendConditionalDeleteObjectsResult(response); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + const auto results = storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("first"), "etag-first"}, {DB::StoredObject("second"), "etag-second"}}, + DB::ObjectStorageControlRequest{}); + + ASSERT_EQ(results.size(), 2u); + EXPECT_EQ(results[0].outcome, DB::ConditionalRemoveOutcome::Removed); + EXPECT_EQ(results[1].outcome, DB::ConditionalRemoveOutcome::TokenMismatch); + EXPECT_EQ(server.countMethod("POST"), 1u); + EXPECT_NE(server.lastBody().find("first"), std::string::npos); + EXPECT_NE(server.lastBody().find("etag-first"), std::string::npos); + EXPECT_NE(server.lastBody().find("second"), std::string::npos); + EXPECT_NE(server.lastBody().find("etag-second"), std::string::npos); +} + +TEST(CASS3ConditionalBatchDelete, UnsupportedConditionalRequestDoesNotDisableOrdinaryBatch) +{ + (void)contextForTest(); + + std::atomic attempts{0}; + ScriptedS3Server server([&](const Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) + { + EXPECT_EQ(request.getMethod(), "POST"); + if (attempts++ == 0) + sendBatchNotImplemented(response); + else + sendDeleteObjectsResult(response, {}); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + expectNotImplemented([&] + { + storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("first"), "etag-first"}, {DB::StoredObject("second"), "etag-second"}}, + DB::ObjectStorageControlRequest{}); + }); + EXPECT_EQ(storage->batchDeleteKeyLimit(), 1000u); + EXPECT_NO_THROW(storage->removeObjectsIfExistUnderProfile( + {DB::StoredObject("first"), DB::StoredObject("second")}, DB::ObjectStorageControlRequest{})); + EXPECT_EQ(server.countMethod("POST"), 2u); + EXPECT_EQ(server.countMethod("DELETE"), 0u); +} + +TEST(CASS3ConditionalBatchDelete, RefreshesExpiredCredentialsBeforeReturningFailure) +{ + (void)contextForTest(); + + std::atomic requests{0}; + ScriptedS3Server server([&](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + if (requests.fetch_add(1) == 0) + { + sendSingleDeleteError(response, Poco::Net::HTTPResponse::HTTP_FORBIDDEN, "ExpiredToken", "expired"); + return; + } + sendConditionalDeleteObjectsResult(response); + }); + + std::atomic refreshes{0}; + auto storage = makeStorageForTest( + server.getUrl(), DB::S3Capabilities{}, {}, + [&]() -> std::unique_ptr + { + ++refreshes; + DB::RemoteHostFilter remote_host_filter; + DB::S3::PocoHTTPClientConfiguration cfg = DB::S3::ClientFactory::instance().createClientConfiguration( + "us-east-1", remote_host_filter, 100, + DB::S3::PocoHTTPClientConfiguration::RetryStrategy{.max_retries = 0}, + false, false, false, true, {}, {}); + cfg.endpointOverride = server.getUrl(); + cfg.s3_use_adaptive_timeouts = false; + return DB::S3::ClientFactory::instance().create( + cfg, + DB::S3::ClientSettings{ + .use_virtual_addressing = false, + .disable_checksum = false, + .gcs_issue_compose_request = false, + .is_s3express_bucket = false, + }, + "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "", {}, {}, DB::S3::CredentialsConfiguration{}); + }); + + const auto results = storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("first"), "etag-first"}, {DB::StoredObject("second"), "etag-second"}}, + DB::ObjectStorageControlRequest{}); + + ASSERT_EQ(results.size(), 2u); + EXPECT_EQ(refreshes.load(), 1u); + EXPECT_EQ(requests.load(), 2u); +} + +TEST(CASS3ConditionalBatchDelete, DoesNotTreatMissingBucketAsMissingObject) +{ + (void)contextForTest(); + + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + sendDeleteObjectsResult(response, {{"first", "NoSuchBucket"}, {"second", "PreconditionFailed"}}); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + try + { + storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("first"), "etag-first"}, {DB::StoredObject("second"), "etag-second"}}, + DB::ObjectStorageControlRequest{}); + FAIL() << "expected NoSuchBucket to remain a hard error"; + } + catch (const DB::S3Exception & e) + { + EXPECT_EQ(e.getExceptionName(), "NoSuchBucket"); + } +} + +TEST(CASS3ConditionalBatchDelete, ReturnsDeleteMarkerOnlyForTheMarkedKey) +{ + (void)contextForTest(); + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + sendXml(response, Poco::Net::HTTPResponse::HTTP_OK, + "" + "markedtrue" + "plain" + "absentNoSuchKey" + ""); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + const auto results = storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("absent"), "etag-a"}, {DB::StoredObject("plain"), "etag-p"}, {DB::StoredObject("marked"), "etag-m"}}, + DB::ObjectStorageControlRequest{}); + + ASSERT_EQ(results.size(), 3u); + EXPECT_EQ(results[0].outcome, DB::ConditionalRemoveOutcome::NotFound); + EXPECT_FALSE(results[0].created_delete_marker); + EXPECT_EQ(results[1].outcome, DB::ConditionalRemoveOutcome::Removed); + EXPECT_FALSE(results[1].created_delete_marker); + EXPECT_EQ(results[2].outcome, DB::ConditionalRemoveOutcome::Removed); + EXPECT_TRUE(results[2].created_delete_marker); +} + +namespace +{ + +void expectLogicalError(const std::function & operation, const char * expected_message) +{ +#if defined(DEBUG_OR_SANITIZER_BUILD) + EXPECT_DEATH(operation(), expected_message); +#else + try + { + operation(); + FAIL() << "expected LOGICAL_ERROR: " << expected_message; + } + catch (const DB::Exception & e) + { + EXPECT_EQ(e.code(), DB::ErrorCodes::LOGICAL_ERROR) << e.message(); + EXPECT_NE(e.message().find(expected_message), std::string::npos) << e.message(); + } +#endif +} + +void expectMalformedConditionalResponse(const std::string & entries, const char * expected_message) +{ + /// Construct the server in the death-test child rather than forking its running threads. + expectLogicalError([&] + { + (void)contextForTest(); + ScriptedS3Server server([&](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + sendXml(response, Poco::Net::HTTPResponse::HTTP_OK, + "" + entries + ""); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("first"), "etag-first"}, {DB::StoredObject("second"), "etag-second"}}, + DB::ObjectStorageControlRequest{}); + }, expected_message); +} + +} + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASS3ConditionalBatchDeleteDeathTest, RejectsDuplicateResponseKey) +#else +TEST(CASS3ConditionalBatchDelete, RejectsDuplicateResponseKey) +#endif +{ + expectMalformedConditionalResponse( + "firstfirst" + "secondPreconditionFailedstale", + "unexpected or duplicate key first"); +} + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASS3ConditionalBatchDeleteDeathTest, RejectsMissingResponseKey) +#else +TEST(CASS3ConditionalBatchDelete, RejectsMissingResponseKey) +#endif +{ + expectMalformedConditionalResponse("first", "response omitted keys"); +} + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASS3ConditionalBatchDeleteDeathTest, RejectsUnexpectedResponseKey) +#else +TEST(CASS3ConditionalBatchDelete, RejectsUnexpectedResponseKey) +#endif +{ + for (const std::string entry : {"foreign", "foreignNoSuchKey"}) + { + SCOPED_TRACE(entry); + expectMalformedConditionalResponse("first" + entry, "unexpected or duplicate key foreign"); + } +} + TEST(S3BulkDeleteFallback, UnsupportedBatchReplyRecordsCapabilityFalseAndThrowsNotImplemented) { (void)contextForTest(); @@ -456,12 +713,157 @@ void expectRefusalNamed(const std::function & remove, const std::string { EXPECT_EQ(e.getExceptionName(), name) << e.message(); } + catch (const DB::Exception & e) + { + ADD_FAILURE() << "expected an S3Exception named " << name << ", got " << e.message(); + } } const DB::StoredObjects kTwoObjects{DB::StoredObject("key-a"), DB::StoredObject("key-b")}; } +TEST(CASS3ConditionalBatchDelete, SingleAttemptSendsCallerAttemptNumberWithoutRetrying) +{ + (void)contextForTest(); + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) + { + EXPECT_EQ(request.getMethod(), "POST"); + EXPECT_EQ(request.get("clickhouse-request", ""), "attempt=7"); + sendBatchInternalError(response); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}, {}, {}, /*max_retries=*/2); + + expectRefusalNamed([&] + { + storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("first"), "etag-first"}, {DB::StoredObject("second"), "etag-second"}}, + {.profile = DB::ObjectStorageRetryProfile::SingleAttempt, .attempt_number = 7}); + }, "InternalError"); + EXPECT_EQ(server.countMethod("POST"), 1u); + EXPECT_EQ(server.countMethod("DELETE"), 0u); +} + +#if defined(DEBUG_OR_SANITIZER_BUILD) +TEST(CASConditionalBatchDeleteBackendDeathTest, RejectsMalformedRawTokenBeforeSendingBatch) +#else +TEST(CASConditionalBatchDeleteBackend, RejectsMalformedRawTokenBeforeSendingBatch) +#endif +{ + using namespace DB::Cas; + + /// Inject the wildcard after admission, at the raw boundary where a backend decorator can supply a bad token. + class MalformedTokenBackend final : public InMemoryBackend + { + public: + explicit MalformedTokenBackend(ObjectStorageBackend & native_) : native(native_) + { + } + + std::vector removeManyByConditional( + const std::vector & removals, TransportAccess & access) override + { + auto malformed = removals; + malformed.back().expected_token = "*"; + return native.removeManyByConditional(malformed, access); + } + + private: + ObjectStorageBackend & native; + }; + + expectLogicalError([] + { + (void)contextForTest(); + ScriptedS3Server server([](const Poco::Net::HTTPServerRequest &, Poco::Net::HTTPServerResponse & response) + { + ADD_FAILURE() << "malformed batch reached S3"; + sendConditionalDeleteObjectsResult(response); + }); + ObjectStorageBackend native(makeStorageForTest(server.getUrl(), DB::S3Capabilities{}), ObjectStorageBackend::Mode::Native); + auto backend = std::make_shared(native); + backend->pinConditionalBatchDeleteSupport(true); + auto requests = tests::openRequestsForTest(backend); + auto op = requests.admit(); + const auto first = std::get(op.create("first", "a", Retry::once())).etag; + const auto second = std::get(op.create("second", "b", Retry::once())).etag; + op.removeManyByConditional({{"first", first}, {"second", second}}, Retry::once()); + }, "malformed conditional removal token for second"); +} + +TEST(CASS3ConditionalBatchDelete, InvalidRequestAndArgumentRemainErrors) +{ + (void)contextForTest(); + for (const std::string name : {"InvalidRequest", "InvalidArgument"}) + { + SCOPED_TRACE(name); + ScriptedS3Server server([&](const Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) + { + EXPECT_EQ(request.getMethod(), "POST"); + sendSingleDeleteError(response, Poco::Net::HTTPResponse::HTTP_BAD_REQUEST, name, "invalid conditional delete input"); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}); + + expectRefusalNamed([&] + { + storage->removeObjectsIfTokensMatch( + {{DB::StoredObject("first"), "etag-first"}, {DB::StoredObject("second"), "etag-second"}}, + DB::ObjectStorageControlRequest{}); + }, name); + EXPECT_EQ(storage->batchDeleteKeyLimit(), 1000u); + EXPECT_EQ(server.countMethod("POST"), 1u); + EXPECT_EQ(server.countMethod("DELETE"), 0u); + } +} + +TEST(CASConditionalBatchDeleteBackend, RuntimeFailureDoesNotDowngradePinnedCapability) +{ + using namespace DB::Cas; + (void)contextForTest(); + std::atomic delete_attempts{0}; + ScriptedS3Server server([&](const Poco::Net::HTTPServerRequest & request, Poco::Net::HTTPServerResponse & response) + { + if (request.getMethod() == "HEAD") + { + response.setContentLength(1); + response.set("ETag", "\"head-etag\""); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_OK); + response.send(); + return; + } + EXPECT_EQ(request.getMethod(), "POST"); + if (delete_attempts++ < 2) + sendBatchNotImplemented(response); + else + sendConditionalDeleteObjectsResult(response); + }); + auto storage = makeStorageForTest(server.getUrl(), DB::S3Capabilities{}, 7); + auto backend = std::make_shared(storage, ObjectStorageBackend::Mode::Native); + auto requests = tests::openRequestsForTest(backend); + auto op = requests.admit(); + EXPECT_EQ(backend->conditionalBatchDeleteKeyLimit(), 1u); + backend->pinConditionalBatchDeleteSupport(true); + ASSERT_EQ(backend->conditionalBatchDeleteKeyLimit(), 7u); + + const auto first = op.head("first", Retry::once()); + const auto second = op.head("second", Retry::once()); + ASSERT_TRUE(first && second); + const std::vector removals{{"first", first->etag}, {"second", second->etag}}; + expectNotImplemented([&] { op.removeManyByConditional(removals, Retry::once()); }); + ASSERT_EQ(backend->conditionalBatchDeleteKeyLimit(), 7u); + + expectNotImplemented([&] + { + storage->removeObjectsIfExistUnderProfile(kTwoObjects, DB::ObjectStorageControlRequest{}); + }); + EXPECT_EQ(backend->bulkDeleteKeyLimit(), 1u); + ASSERT_EQ(backend->conditionalBatchDeleteKeyLimit(), 7u); + EXPECT_EQ(op.removeManyByConditional(removals, Retry::once()), (std::vector{Removal::Removed, Removal::Mismatch})); + EXPECT_EQ(backend->bulkDeleteKeyLimit(), 1u); + EXPECT_EQ(server.countMethod("POST"), 3u); + EXPECT_EQ(server.countMethod("DELETE"), 0u); +} + TEST(CASS3BatchDelete, RefusedDeleteObjectsKeepsTheStoresErrorName) { (void)contextForTest(); diff --git a/tests/integration/test_cas_gc_conditional_batch_delete/__init__.py b/tests/integration/test_cas_gc_conditional_batch_delete/__init__.py new file mode 100644 index 0000000000000..e69de29bb2d1d diff --git a/tests/integration/test_cas_gc_conditional_batch_delete/configs/storage_conf.xml b/tests/integration/test_cas_gc_conditional_batch_delete/configs/storage_conf.xml new file mode 100644 index 0000000000000..66f2ffef21a0c --- /dev/null +++ b/tests/integration/test_cas_gc_conditional_batch_delete/configs/storage_conf.xml @@ -0,0 +1,55 @@ + + + + + object_storage + s3 + cas + 30 + 10000 + itest-cas-gc-conditional-batch + http://rustfs1:11121/test/cas_gc_conditional_batch/ + clickhouse + clickhouse + 1 + 3600 + 1 + + 1 + 2 + + + object_storage + s3 + cas + 30 + 10000 + itest-cas-gc-conditional-singleton + http://rustfs1:11121/test/cas_gc_conditional_singleton/ + clickhouse + clickhouse + 1 + 3600 + 1 + 1 + 1 + + + + + +
+ disk_cas_gc_conditional_batch +
+
+
+ + +
+ disk_cas_gc_conditional_singleton +
+
+
+
+
+
diff --git a/tests/integration/test_cas_gc_conditional_batch_delete/test.py b/tests/integration/test_cas_gc_conditional_batch_delete/test.py new file mode 100644 index 0000000000000..d618da4f75b4c --- /dev/null +++ b/tests/integration/test_cas_gc_conditional_batch_delete/test.py @@ -0,0 +1,139 @@ +import pytest + +from helpers.cluster import ClickHouseCluster + +cluster = ClickHouseCluster(__file__) + +NUM_INSERTS = 6 +MAX_GC_ROUNDS = 12 + + +@pytest.fixture(scope="module", autouse=True) +def start_cluster(): + cluster.add_instance( + "node", + main_configs=["configs/storage_conf.xml"], + with_rustfs=True, + stay_alive=True, + ) + try: + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def list_blob_bodies(pool_prefix): + return { + o.object_name + for o in cluster.rustfs_client.list_objects( + cluster.rustfs_bucket, f"{pool_prefix}/blobs/", recursive=True + ) + if not o.object_name.endswith(".meta") + } + + +def latest_gc_round(node, disk_name): + node.query("SYSTEM FLUSH LOGS") + return int( + node.query( + f"SELECT max(round) FROM system.cas_gc_log WHERE disk_name = '{disk_name}'" + ).strip() + ) + + +def gc_pending_delete_rows(node, disk_name, after_round): + node.query("SYSTEM FLUSH LOGS") + rows = ( + node.query( + "SELECT f.round, p.phase_metrics['redeleted'], p.phase_metrics['deleted'], " + "p.phase_metrics['delete_chunks'], p.ProfileEvents['DiskS3DeleteObjects'], " + "p.phase_metrics['read_blob_jobs_failed'], " + "p.phase_metrics['delete_chunk_jobs_failed'], " + "p.phase_metrics['read_blob_job_schedule_failures'], " + "p.phase_metrics['delete_chunk_job_schedule_failures'] " + "FROM system.cas_gc_log AS p INNER JOIN " + "(SELECT round_id, round FROM system.cas_gc_log " + f"WHERE disk_name = '{disk_name}' AND event_type = 'Finish' AND outcome = 'Success' " + f"AND round > {after_round}) AS f USING round_id " + f"WHERE p.disk_name = '{disk_name}' AND p.event_type = 'Phase' " + "AND p.phase = 'pending_deletes' AND p.phase_metrics['redeleted'] > 0 " + "ORDER BY p.event_time_microseconds" + ) + .strip() + .splitlines() + ) + return [tuple(int(x) for x in row.split("\t")) for row in rows] + + +def reclaim_blob_bodies(pool_prefix, chunk_keys): + node = cluster.instances["node"] + disk_name = f"disk_{pool_prefix}" + table = f"{pool_prefix}_redelete" + node.query(f"SYSTEM CAS GC STOP '{disk_name}'") + assert not list_blob_bodies(pool_prefix) + node.query( + f"CREATE TABLE {table} (k UInt64, v String) ENGINE = MergeTree ORDER BY k " + f"SETTINGS storage_policy = '{pool_prefix}'" + ) + node.query(f"SYSTEM STOP MERGES {table}") + for i in range(NUM_INSERTS): + node.query( + f"INSERT INTO {table} SELECT number + {i * 1000}, " + f"concat(toString(number), '-{i}') FROM numbers(1000) " + "SETTINGS max_insert_block_size = 1000" + ) + + created_blobs = list_blob_bodies(pool_prefix) + assert len(created_blobs) > 2 + first_round = latest_gc_round(node, disk_name) + node.query(f"DROP TABLE {table} SYNC") + + rows = [] + remaining = created_blobs + for _ in range(MAX_GC_ROUNDS): + node.query(f"SYSTEM CAS GC RUN '{disk_name}'") + rows = gc_pending_delete_rows(node, disk_name, first_round) + remaining = created_blobs & list_blob_bodies(pool_prefix) + if not remaining and rows: + break + + assert not remaining, f"GC left blob bodies in RustFS: {sorted(remaining)}" + assert rows, f"{disk_name}: pending_deletes never reported completed blob redeletes" + total_deleted = total_requests = 0 + for ( + round_no, + redeleted, + deleted, + delete_chunks, + s3_delete_requests, + read_blob_jobs_failed, + delete_chunk_jobs_failed, + read_blob_job_schedule_failures, + delete_chunk_job_schedule_failures, + ) in rows: + assert read_blob_jobs_failed == 0 + assert delete_chunk_jobs_failed == 0 + assert read_blob_job_schedule_failures == 0 + assert delete_chunk_job_schedule_failures == 0 + # The isolated workload neither replaces nor removes blob bodies outside GC. + assert redeleted == deleted, f"{disk_name}, round {round_no}: {rows}" + expected_chunks = (deleted + chunk_keys - 1) // chunk_keys + assert delete_chunks == expected_chunks, f"{disk_name}, round {round_no}: {rows}" + assert s3_delete_requests == delete_chunks, f"{disk_name}, round {round_no}: {rows}" + total_deleted += deleted + total_requests += s3_delete_requests + assert total_deleted == len(created_blobs) + return total_deleted, total_requests + + +def test_pending_blob_deletes_use_conditional_batches(): + singleton_deleted, singleton_requests = reclaim_blob_bodies( + "cas_gc_conditional_singleton", 1 + ) + batch_deleted, batch_requests = reclaim_blob_bodies("cas_gc_conditional_batch", 2) + + # With the real mount probe gating multi-key chunks, a passed comparison proves batching + # was enabled and reached RustFS. Both disks reclaim the same size workload. + assert singleton_deleted == batch_deleted + assert batch_requests < singleton_requests