Skip to content

Backup: the operator runs backups and restores itself through SQL on clickhouse-backup sidecars, multi-period schedules - #2148

Open
Slach wants to merge 8 commits into
0.28.0from
backup-config-file
Open

Slach wants to merge 8 commits into
0.28.0from
backup-config-file

Conversation

@Slach

@Slach Slach commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

fix #2146

What changes

The operator itself runs backups and restores. There is no orchestrator script, no Job, no CronJob and no backup Pod: the operator drives the clickhouse-backup sidecar of every host through SQL on the integration tables of that host's ClickHouse, as the Altinity usual backup workflow does.

  • Transport: sidecar only. Every Pod of a backup-enabled cluster gets a clickhouse-backup server sidecar. Its configuration, generated by the operator, always turns on api.create_integration_tables, so system.backup_actions and system.backup_list read http://127.0.0.1:7171. The sidecar runs with TZ=UTC, and restore_distributed_cluster defaults to {cluster}. backup.mode and pod mode are removed.

  • Credentials. The operator connects to each host by its FQDN, port, scheme and the cluster TLS, with the ClickHouse credentials of its own configuration (clickhouse.access or its Secret). They must be set explicitly: the built-in defaults do not count, and without them a backup or restore waits with reason NoClickHouseAccess.

  • Commands. A command is a row inserted into system.backup_actions of the host after now('UTC') of that host (async_insert = 0, http_receive_timeout/http_send_timeout from httpReceiveTimeout), and polled every 5 seconds, matched by its exact command text and start time, until it finishes. A busy sidecar (another operation is currently running) is retried with exponential backoff from 5 seconds up to 5 minutes. After an operator restart the leader resumes every run from its status: a running command is followed, a finished one is settled.

  • Timeouts. spec.timeouts of a ClickHouseBackup, ClickHouseRestore and ClickHouseBackupSchedule, defaults in the operator configuration backup.timeouts and in code, as Go durations: actionTimeout (12h), unreachableTimeout (10m), httpReceiveTimeout (4h), errorResponsesMax (5). Errors and empty polls are counted apart; the logs carry the response code.

  • Backup. create_remote --rbac on one replica per shard (replicaSelection: FirstPerShard, preferredReplica, fallback to another replica reported by OrchestratorReplicaFallback) or on every replica (AllReplicas). A shard is backed up once any of its replicas made the backup. A failed backup deletes only its local copy, once, and reports a remote leftover (OrchestratorFailedBackupLeft, OrchestratorFailedBackupRemoteLeft) with the SQL to delete it by hand.

  • Storage paths. Before any command, path and object_disks_path of every writer of a backup, with the macros of each host expanded, must be unique (AllReplicas: every replica; embedded backups and CAS disks included). Otherwise the backup or schedule is Aborted with StoragePathConflict, e.g. storage path 's3://backups/backup/main/0' is not unique: replicas ... both write a backup of one name there. Unknown macros give an OrchestratorStoragePathAdvice warning.

  • Backup names. spec.backup.naming / clusters[].backup.naming, a template such as shard{shardNum}-{YYYYMMDDHHIISS} with {shard} (the shard macro), {shardNum} (shard_num, from 1), {cluster}, {chi}, {namespace}, {prefix} and the time tokens. Retention, the incremental base and restore parse names back with it, so backups named by the Altinity usual backup workflow on a shared remote path are recognized. Empty keeps the current names.

  • Schedules. ClickHouseBackupSchedule.spec.schedules[] (minute, hour, 6hours, daily, weekly, monthly, with time, day, keep, incremental, configOverrides) and spec.timeZone:

    • the operator runs the slots of the finest entry itself; the heaviest matching entry decides full vs incremental;
    • no CronJob, Job or ClickHouseBackup is created per run: a run lives in status.active while it runs and in status.recentRuns afterwards, bounded by successfulRunsHistoryLimit (3) and failedRunsHistoryLimit (1); startingDeadlineSeconds skips a slot missed by more; runs of one cluster never overlap;
    • a ClickHouseBackup with scheduleRef runs the schedule now: its incremental entry, retention after it, a base of the next incremental backup.
  • Retention follows the Altinity usual backup workflow: each entry keeps its own keep newest backups (GFS); the newest backup is always kept; run-now backups live within the window of the scheduled ones; an incomplete backup is deleted only once a newer complete one exists and system.backup_actions shows no create_remote or upload of it on its shard; nothing is deleted during a restore; incremental bases are rebased by clickhouse-backup (forced backups_to_keep_remote: 0, rebase_*: true).

  • Schedule changes. spec.retention.onScheduleChange: Hold (default) records a change of the entries or the time zone in status.retention.previous[] with its expires. The backups the schedule kept before it and the current one would delete are held, listed in status.clusters[].heldBackups with expires: backup time + keep × the period of the longest old entry that kept it. Expired holds are deleted (OrchestratorRetentionHeldExpired), an expired change leaves previous[] (RetentionHeldExpired).

    • keepBackups: [{name, until}] pins backups: with until until then; without it a backup held by a change until its expiry, any other backup while it is listed. Pins protect incomplete backups too.
    • releaseHeld: <changeID> releases a change and all earlier ones, Prune applies a change at once.
  • Config overrides. schedules[].configOverrides and ClickHouseBackup.spec.configOverrides: up to 16 KEY=VALUE clickhouse-backup settings for one backup, sent as create_remote --env=.... Keys the operator pins (TZ included), API_*, paths, buckets, endpoints, the embedded backup disk and secrets are refused.

  • Restore, as in the Altinity usual backup workflow:

    1. download --hardlink-exists-files on every host (an incomplete local copy is resumed);
    2. restore --schema=1 --ignore-dependencies on every replica, the first replica of the first shard with RESTORE_SCHEMA_ON_CLUSTER={cluster}; skipped for partitions;
    3. spec.replicatedCopyToDetached (default true): the other replicas of each shard run restore --data=1 --ignore-dependencies --replicated-copy-to-detached at the same time, so the parts of replicated tables are already local; with false they download only the schema;
    4. restore --data=1 --ignore-dependencies on one replica per shard, the next answering replica when the first does not;
    5. delete local on every host after success; a failed restore keeps its downloads.

    Filters go on every command: tables, partitions, skipTables, skipTableEngines, databaseMapping, rbac. overwrite: true without tables adds --rm, which recreates the databases of the backup as a whole; with tables, or without overwrite, only the tables of the backup are recreated (clickhouse-backup drops the existing ones) and no database is dropped. A restore of access entities holds the backups and schedules of its own clusters only; restores of different clusters run at the same time.

  • Reruns: spec.taskID on all three kinds, with status.taskID, taskIDsStarted, taskIDsCompleted. A completed backup runs again under a new name (the old one in status.previousBackupNames), an unfinished one under its own name after its earlier copies are deleted. A failed restore runs again: spec.resumeDownload (default true) resumes its earlier downloads, false deletes them and starts over. An aborted schedule is validated again. A taskID set while a run is active is ignored with a TaskIDIgnored warning.

  • Deletion. Finalizers on ClickHouseBackup, ClickHouseRestore and ClickHouseBackupSchedule. Deleting an unfinished backup kills only its own running commands (kill "<command>") and deletes its local and remote copies; a completed one keeps its remote backup for retention. A restore and a schedule run are waited for, and a deleted schedule starts no new slot. Unreachable hosts are given up after unreachableTimeout, with a DeletionLeftover warning and the SQL to finish by hand.

  • Validation. Backup enabled without configSecretRef, a missing or invalid storage Secret, or an invalid backup block stops the CHI reconcile: status Aborted, reason BackupConfigInvalid, Warning event BackupStorageConfigInvalid.

  • Status and events. status.clusters[].run of a ClickHouseBackup / ClickHouseRestore records the step, the hosts and every command sent with its status. Phases Pending, Running, Completed, Failed, Aborted. The operator posts every event itself (Orchestrator* per step and command) on the ClickHouseBackup, ClickHouseBackupSchedule and ClickHouseRestore.

  • Removed: backup.mode and pod mode, backup.orchestrator, the bash and Python orchestrator scripts and their ConfigMaps, orchestratorConfigMapRef, backup.path of the operator configuration, jobName / cronJobName, concurrencyPolicy, backup verification (spec.verify, the Verified condition and its metric), the restore overwrite guard, and the operator RBAC rights for Jobs, CronJobs, Pod writes, ServiceAccounts, Roles and RoleBindings. The operator gains update/patch on its three backup kinds for the finalizers.

  • Manifests are regenerated with the current yq, which writes plain <<: merge keys in place of !!merge <<:; the documents are the same.

  • Docs: docs/backup.md (how the operator runs an operation, naming, timeouts, storage paths, migration from the Altinity usual backup workflow, schedule changes, reruns, deletion, config overrides, integration tables), docs/custom_resource_explained.md, docs/operator_configuration.md, docs/chb-examples/.

Tests

  • go build ./... && go test ./..., -race for pkg/controller/chbackup and pkg/model/chbackup: runs against fake sidecars (command text, polling, timeouts, resume after restart, busy backoff, replica fallback, copy to detached, finalizers, taskID reruns, path uniqueness, per-cluster gates), retention plans, schedule changes and their expiry, naming, slots, incremental base.
  • helm template of the regenerated chart.
  • Native e2e on orbstack:
    • test_010087: backup, schedule, retention and restore of two clusters through real sidecars;
    • test_010088: recovery of a new CHI from another one's backup;
    • test_010089: retention across a schedule change: held backups expiring by the old schedule, keepBackups pins without until, with until in the past and in the future.

Related, in clickhouse-backup:

deploy/operatorhub is not regenerated (done at release). dev/run_code_generator.sh does not build with the current Go (#2152); deepcopy was generated with deepcopy-gen v0.34.1.

🤖 Generated with Claude Code

@Slach
Slach force-pushed the backup-config-file branch from d0d0ec5 to d81d518 Compare November 12, 2026 10:51
@Slach Slach changed the title Backup: sidecar or per-shard Pod transport, multi-period schedules, orchestrator script as operator config Backup: sidecar or per-shard Pod transport, multi-period schedules, bash or Python orchestrator as operator config Nov 13, 2026
@Slach
Slach force-pushed the backup-config-file branch from ddc3980 to c3e4b29 Compare November 16, 2026 14:54
@Slach Slach changed the title Backup: sidecar or per-shard Pod transport, multi-period schedules, bash or Python orchestrator as operator config Backup: the operator runs backups and restores itself through SQL on clickhouse-backup sidecars, multi-period schedules Nov 17, 2026
@Slach
Slach force-pushed the backup-config-file branch from c3e4b29 to 3c76ca0 Compare November 17, 2026 13:26
… the sidecars

Operator-managed backup no longer creates Jobs, CronJobs or backup Pods and ships
no orchestrator script. The operator drives the clickhouse-backup sidecar of every
host through the integration tables of its ClickHouse - system.backup_actions and
system.backup_list - with the ClickHouse credentials of its own configuration, one
run per cluster, as the Altinity usual backup workflow does.

- Runs: a command is a row of system.backup_actions, inserted after now('UTC') of
  the host and polled every 5 seconds until it finishes; a lost command is settled
  (a resumed upload or download is followed, a complete backup counts as made); a
  busy sidecar is retried with exponential backoff from 5 seconds to 5 minutes.
- Status: status.clusters[].run of a backup or a restore, and status.active and
  status.recentRuns of a schedule, record each run - step, hosts, the commands sent
  with their status, the plan - and the leader resumes runs after a restart.
- Schedules run at the slots of their finest entry without a CronJob or a
  ClickHouseBackup per run; startingDeadlineSeconds skips a slot missed by more;
  successful/failedRunsHistoryLimit bound status.recentRuns.
- Credentials: backup runs only with a ClickHouse username and password the operator
  configuration sets itself, in clickhouse.access or its Secret; the defaults do not
  count (reason NoClickHouseAccess).
- A shard is backed up once any of its replicas made the backup, AllReplicas
  included; retention takes a backup as complete per shard.
- Restore follows the Altinity usual backup workflow: download
  --hardlink-exists-files on every host, restore --schema=1 on every replica - the
  first replica of the first shard with RESTORE_SCHEMA_ON_CLUSTER={cluster} -,
  restore --data=1 on one replica per shard, then delete local. Every restore
  command takes --ignore-dependencies; --rm, which recreates the databases of the
  backup whole, only for an overwrite without tables; the filters (tables,
  partitions, skipTables, skipTableEngines, databaseMapping, rbac) go on every
  command. The
  overwrite guard, the separate ON CLUSTER drop pass, the max_table_size_to_drop
  checks and the dependency on the ZooKeeper configuration of the CHI are gone.
- The sidecar pins API_CREATE_INTEGRATION_TABLES and API_INTEGRATION_TABLES_HOST;
  the operator reaches each host by its FQDN, port, scheme and cluster TLS.
- Events of the runs (Orchestrator*), conditions and reasons (Started, Failed,
  FailedUnchanged, NoClickHouseAccess) are posted by the operator.
- Removed: backup.mode and pod mode, backup.orchestrator, the bash and Python
  orchestrator scripts and their ConfigMaps, orchestratorConfigMapRef, jobName and
  cronJobName, concurrencyPolicy, backup verification (spec.verify, the Verified
  condition and its reasons, the verification metric), backup.path of the operator
  configuration, and the RBAC rights for Jobs, CronJobs, Pods and RBAC objects.
- Names: {shard} of a naming template is the shard macro, {shardNum} the shard_num
  from 1; restore keeps the naming prefix.
- The sidecar runs with TZ=UTC; restore_distributed_cluster defaults to {cluster}.
- spec.timeouts of backups, restores and schedules, with defaults in the operator
  configuration (backup.timeouts) and in code: actionTimeout (12h),
  unreachableTimeout (10m), httpReceiveTimeout (4h), errorResponsesMax (5), as Go
  durations. Errors and empty polls are counted apart, with the response code.
- A failed backup deletes only its local copy, once; a failed restore keeps its
  downloads, deleted only after success.
- Retention: keepBackups pins {name, until}; backups held by a schedule change, and
  pins without until of such backups, expire by the schedule before the change
  (backup time + keep x period of its longest entry that kept it), listed in
  heldBackups with expires; an incomplete backup is deleted only when no
  create_remote or upload of it runs on its shard per system.backup_actions.
- Storage paths: path and object_disks_path, macros expanded per host, must be
  unique among the writers of one backup (AllReplicas: every replica), embedded
  backups included; otherwise the backup or schedule is Aborted with
  StoragePathConflict before any command.
- Finalizers: deleting an unfinished backup kills its commands and deletes its
  local and remote copies; a completed one keeps its remote backup; a restore and a
  schedule run are waited for.
- spec.taskID: a completed backup runs again under a new name, an unfinished one
  under its own; a failed restore runs again, resuming its downloads
  (spec.resumeDownload, default true) or starting over; an aborted schedule is
  validated again. A taskID during a run is ignored with a warning.
- Restore: spec.replicatedCopyToDetached (default true) - the other replicas of a
  shard restore the data of replicated tables to detached first, then one replica
  attaches it; false downloads only the schema on them.
- A restore of access entities holds the backups and schedules of its own clusters
  only; restores of different clusters run at the same time.
- Manifests regenerated with the current yq: plain <<: merge keys.
- CRDs, manifests, Helm chart, docs/backup.md and the examples follow; unit tests
  cover the runs on fake sidecars; e2e test_010087, test_010088 and test_010089
  (retention across a schedule change, keepBackups) cover real sidecars.
- ci: actions/setup-python@v7, which runs on Node.js 24.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alex-zaitsev

Copy link
Copy Markdown
Member

@Slach , for consistency sake, could you change the secret reference to valueFrom syntax? Like:

backup:
  enabled: "yes"
  storageConfig:
    valueFrom:
      secretKeyRef:
        name: backup-storage
        key: config.yml

It has been missed in secrets review.

Slach and others added 2 commits November 19, 2026 12:59
The storage Secret of the operator-managed backup is referenced with the
valueFrom/secretKeyRef syntax cluster secrets and settings use, and the
key of the Secret is the user's choice instead of a fixed config.yml.

The key is required: an empty key, hotReload or optional refuse the backup
block (BackupConfigInvalid, BackupStorageConfigInvalid event). Inheritance
and the explicit secretKeyRef.name "" opt-out work as before. The generated
configuration Secret and the sidecar mount keep config.yml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@denisok

denisok commented Nov 19, 2026

Copy link
Copy Markdown

claude AI review, maybe slop but JFYI:

Blockers

1. Deleting a failed backup CR can delete someone else's backup. A ClickHouseBackup refused in preflight with "backup X already exists in the storage" never sent a command. When that CR is deleted, the finalizer still sends delete remote X on every host and removes the pre-existing backup.
   - unfinishedHosts (pkg/controller/chbackup/finalizer.go:220) falls back to cluster.Hosts whenever a run exists but recorded no hosts, which is exactly the preflight-failure case.
   - The docs promise deletion only on "every host it sent a command to".
   - Confirmed with a repro test, kept outside the repo at /tmp/claude-1000/-home-stdden-Workspace-clickhouse-operator/d0327138-a99a-438b-adb1-d10ccc1ed6b9/scratchpad/review_repro_refused_backup_delete_test.go. Copied into pkg/controller/chbackup/, it fails with delete remote manual sent to both hosts.
2. Losing the leader lease disables backup until the pod restarts. The backup manager is now leader-elected (cmd/operator/app/thread_backup.go:153). On a lost lease, controller-runtime returns "leader election lost" from Start and expects the process to exit.
   - launchBackup (cmd/operator/app/main.go:182) only logs "Starting backup FAILED" and returns, while the CHI controller keeps the pod alive.
   - With one replica, a brief API-server outage (the default renew deadline is 10s) is enough to trigger it.
   - Afterwards no backup, restore or schedule runs, in-flight runs stay Running forever, and CRs with finalizers hang in Terminating.
   - Confirmed by reading both sides; not reproduced live.

Should fix

3. DST gap slots match no schedule entry (reviewer, confirmed by test). A slot at a wall time that does not exist is shifted by Go, so its tag matches no entry and the next run's retention deletes that backup. Example: daily plus weekly Sunday 02:30 in Europe/Berlin on 2026-03-29. See pkg/model/chbackup/schedule.go:125.
4. Adding a shard wipes backup history at the next scheduled run. Backups made before the shard existed are not "complete on every shard", so retention deletes all of them older than the newest one, whatever keep says (retention.go:223, backup_orchestrator_backup.go:476). The docs describe this as intended; I think it is too destructive for a routine scale-out.
5. Storage path check has false negatives (reviewer, confirmed by test). The storage identity is compared as a raw string, so https://minio:9000 and https://minio:9000/, or an explicit versus unset endpoint, count as different storages. Two clusters can then share a path and prune each other's backups (pathcheck.go:371). A storage path is also never compared with an object disk path.
6. Aborted CHI is not retried after the storage Secret is fixed (reviewer, confirmed by test). isAbortedOnInvalidBackupConfig (pkg/controller/chi/controller-backup-config.go:430) checks only status.errors[0], and the "FAILED to reconcile" line is prepended above the tagged error. This happens with the shipped status.fields.errors: true.
7. Generated Secret name collides and is overwritten without an ownership check (reviewer, confirmed by test). CHI a-b with cluster c and CHI a with cluster b-c both map to a-b-c-backup-config. A user Secret with that name has its data replaced. The apply function already exists on origin/0.28.0, so this may predate these commits.
8. Storage Secret content leaks through YAML errors (reviewer, confirmed by test). A decode error quotes up to 10 characters of the offending value into CHI status, an Event and the log (backupconfig.go:154).
9. mutated is not set when a resumed run adopts a command. In submit (backup_orchestrator.go:390) the adopt branch returns before Mutated is set. A restore that restarted between the insert and the status write can then fail as FailedUnchanged, which releases the schedule hold a changed cluster should keep. The window is narrow.

Lower priority

- Config overrides use a denylist. S3_ACL=public-read and S3_DISABLE_CERT_VERIFICATION=true pass validation (type_backup_overrides.go:49), so anyone who can create a CHB can set them. An allowlist would be safer.
- Leases RBAC is cluster-wide (reviewer). In the ClusterRole variants, create/get/update on all Leases is granted where one named Lease is needed.
- Data-restore fallback may double-apply. If the submit on the first replica fails after the command reached the sidecar, runData hands the same restore to the next replica (backup_orchestrator_restore.go:287). Plausible only.
- Fall-back hour name collision (reviewer). Hourly and minute entries produce the same backup name twice, so the second run fails with "already exists".
- Unbounded deadline seconds (reviewer, plausible). activeDeadlineSeconds and the schedule's startingDeadlineSeconds have no upper bound and overflow time.Duration above about 9.2e9.
- Nits.
  - In runs.go:268 the final notification fires before done closes.
  - A reconcile can start a run before the registry has the manager's context.
  - The comment on parseClickHouseTime names the wrong function.
  - The comment at type_backup.go:65 is stale.

@denisok

denisok commented Nov 19, 2026 •

Copy link
Copy Markdown

API:

  • type. Lets do it explicit and required? type: full, type:incremental. That way ppl wouldn’t be confused.
  • period  maybe hourly . Just adding every:X give us hourly (if absent) or 6 or what ever
  • day : weekly: 0..6, 0 being Sunday; monthly: 1..31. This doesn’t look good. Day like Mon, Tue and etc is better. For month backup it is weird, don’t know yet (first, last maybe)…
  • keep is tight to schedule. Currently retention checks backup times against it’s entries as I understand. Logically separate it will be better, so Retention looks on it’s configuration and existing backups instead of slots:
retention:
  keepLast: 3
  keepHourly: 24
  keepDaily: 7
  keepWeekly: 4
  keepMonthly: 6

also slot matching is fragile because of DST time changes.

Missed runs. Say operator was down, when backup will be done? another point to the bucketed retention above is better control that nothing will be deleted if there was a miss.

types:

  • tables and skipTables should be []string.
  • configOverrides should be map?

Jitter, we start backup on all replicas without delay now as I understand?

One-off backups, no pruning strategy for them? ttl or some keep until?

  schedules:
    - period: hourly
      every:1
      type: incremental
    - period: daily
      time: "02:30"
      type: incremental
    - period: weekly
      time: "02:30"
      day: Monday
      type: full
    - period: monthly
      time: "02:30"
      day: first #or last
      type: full
  retention:
    keepLast: 3
    keepHourly: 24
    keepDaily: 7
    keepWeekly: 4
    keepMonthly: 6

@Slach

Slach commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

@denisok thanks for review

Thanks, I went through both comments point by point against the code.

AI review

Real, will be fixed before merge:

  • Finalizer can delete someone else's backup. A backup refused in preflight ("already exists in the storage") records no hosts, so unfinishedHosts falls back to every host of the cluster and delete remote removes the pre-existing backup. It will delete only on hosts a command was actually sent to.
  • Lost leader lease disables backups until the pod restarts. launchBackup only logs the error. It will re-enter leader election with backoff.
  • DST. A spring-forward daily slot (02:30 → named …033000) matches no entry and is pruned by the next run; a fall-back hour gives two slots one name and the second run fails with "already exists". Slot classification becomes zone-aware, and a repeated wall-clock slot runs once.
  • Generated <chi>-<cluster>-backup-config Secret name collision (a-b/c vs a/b-c), plus an ownership check before overwriting.
  • Storage Secret content leaking through YAML decode errors.
  • Mutated not set when a resumed run adopts a command.
  • Overflow of unbounded activeDeadlineSeconds / startingDeadlineSeconds (schema maximum added).
  • Cluster-wide leases RBAC → namespaced Role only.
  • Storage path check: endpoint normalization (https://minio:9000 vs https://minio:9000/).
  • configOverrides: the denylist gets S3_ACL, S3_DISABLE_CERT_VERIFICATION and their GCS/Azure/COS/FTP/SFTP analogues.
  • The nits.

Not a bug:

  • Aborted CHI after the Secret is fixed. An aborted CHI skips reconcile, so the [BackupConfigInvalid] marker stays first in status.errors, and the config sync re-enqueues the CHI once the Secret is usable.
  • Adding a shard prunes older backups. This is intended and documented: those backups cannot restore the new shard. It needs a newer complete backup first, and retention.keepBackups pins protect them. I'll make the warning more prominent and add a test.

API

Changing now, while unreleased:

  • tables / skipTables become []string, rendered comma-joined like partitions already is.
  • skipTables and skipTableEngines ([]string) on backups and schedules, so "skip Kafka" no longer needs configOverrides. The same key in configOverrides is refused.

Keeping, with the reasons:

  • Per-entry keep, no keepLast/keepHourly/... block. That block is a different policy, not a safer implementation of this one:

    • it changes which backups existing users keep;
    • it loses the per-entry incremental / configOverrides and the hold on schedule change;
    • it does not fix the DST name collision, which happens before retention runs.

    The DST problems are fixed in slot matching instead.

  • No every: X. {period: hour, incremental: true, keep: N} already means "hourly incrementals, a full one every N". The incremental chain is bounded by the entry's keep.

  • No type: full|incremental. The kind of a run is derived: the heaviest entry matching the slot decides, and an incremental run still falls back to full when the chain reaches keep or has no base. A required per-entry type would be wrong on every overlapping slot.

  • Numeric day. Weekly 0..6 with 0 = Sunday, as cron and CronJob. Monthly 1..31, where a day past the month's end means its last day, so 31 is "last" (cron cannot express that). Cloud maps MONDAY → 1 trivially. I'll document it more clearly.

  • No text schedule format (skedule-like) in the CRD. These CRs will be generated by Cloud, so a typo should fail at kubectl apply through the schema, not later as an Aborted event. A text syntax fits better one level up, in the UI/API, emitting these structured entries.

  • configOverrides stays a list. It is one --env flag per item, validated per item.

  • Missed runs. As CronJob: only the latest missed slot runs, late, within startingDeadlineSeconds. Retention is count-based over existing backups, so a miss never deletes anything earlier; it only keeps older backups longer. Will document.

  • Jitter. By default one replica per shard runs (FirstPerShard), all shards together for a consistent snapshot, as CronJob without jitter. Will document.

  • One-off backups are kept until deleted on purpose. scheduleRef (lives within the schedule's retention) and retention.keepBackups[].until cover expiry today. A TTL can come later as an opt-in.

Agreed that consistency with the Altinity usual backup workflow is the baseline, not the goal. The PR already goes beyond it, e.g. held backups survive a retention change until they expire by the old schedule, and the DST fixes above are cases the old workflow never handled.

Slach and others added 5 commits November 20, 2026 15:07
- The finalizer of a backup that sent no command, as one refused in preflight
  for a backup of its name the storage holds already, deletes nothing: only the
  hosts a command was inserted into, or may have been, are cleaned up.
- A backup manager that loses the leader Lease is built again and elected again
  with backoff, until the operator stops; the run registry settles the runs of
  the ended term, drops those it interrupted so the next term resumes them, and
  refuses runs outside a term. A run ends before it tells its resource.
- A daily, weekly or monthly slot whose time does not exist the day the clocks
  go forward is classified, by orchestration and retention alike, as the slot
  of its entries at the time the zone moves it to.
- A wall-clock time the zone repeats as the clocks go back is a slot once, its
  first occurrence, so the second run no longer collides with the first name.
- The command a resumed run adopts marks the run mutated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…um, namespaced leases, endpoint normalization

- The generated configuration Secret is updated only when it carries the
  operator's labels for the CHI; a foreign one of the same name is left alone
  and aborts the reconcile with BackupConfigInvalid naming it.
- YAML errors of the storage Secret name the line at most, never the content.
- startingDeadlineSeconds of a schedule and activeDeadlineSeconds of backups,
  schedules and restores are capped at 2147483647 in validation and the CRDs.
- Leases are granted by a namespaced Role in the operator namespace, not by
  the ClusterRole, in every install flavor and in the Helm chart.
- The storage path check compares endpoints and addresses normalized.
- configOverrides refuses settings that weaken the security of the storage.
- Comment fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…backups, docs

tables and skipTables of ClickHouseBackup, ClickHouseBackupSchedule and
ClickHouseRestore are lists of patterns, one per item, joined by commas as
partitions are, each checked by the pattern regexp without a comma, at most 64
items and 1024 characters joined. Backups and schedules take skipTables and
skipTableEngines, rendered as --env=CLICKHOUSE_SKIP_TABLES (after the default
system skip list) and --env=CLICKHOUSE_SKIP_TABLE_ENGINES on create_remote, and
configOverrides refuse both keys. CRD templates, examples, e2e manifests and
docs updated, with docs of missed slots, jitter, one-off backups, the
incremental entry, the day of an entry and adding a shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the latest due slot of a schedule runs; the slots missed before it, while the operator did
not run or a run of the cluster was in progress or held, were dropped silently. A
ScheduledRunSkipped warning now tells how many slots of the cluster were missed since its last
one, from which, and which slot runs instead, counted up to 10000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The announcement that a deleted restore or schedule waits for its runs was marked made before it
was emitted: a status write that failed on a conflict, or a reconcile that changed no status,
dropped it for good. It is marked made only as it is emitted now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backup: operator-run backup and restore through clickhouse-backup sidecars, multi-period schedules

3 participants