Repository navigation
Conversation
d0d0ec5 to
d81d518
Compare
ddc3980 to
c3e4b29
Compare
c3e4b29 to
3c76ca0
Compare
… the sidecars
Operator-managed backup no longer creates Jobs, CronJobs or backup Pods and ships
no orchestrator script. The operator drives the clickhouse-backup sidecar of every
host through the integration tables of its ClickHouse - system.backup_actions and
system.backup_list - with the ClickHouse credentials of its own configuration, one
run per cluster, as the Altinity usual backup workflow does.
- Runs: a command is a row of system.backup_actions, inserted after now('UTC') of
the host and polled every 5 seconds until it finishes; a lost command is settled
(a resumed upload or download is followed, a complete backup counts as made); a
busy sidecar is retried with exponential backoff from 5 seconds to 5 minutes.
- Status: status.clusters[].run of a backup or a restore, and status.active and
status.recentRuns of a schedule, record each run - step, hosts, the commands sent
with their status, the plan - and the leader resumes runs after a restart.
- Schedules run at the slots of their finest entry without a CronJob or a
ClickHouseBackup per run; startingDeadlineSeconds skips a slot missed by more;
successful/failedRunsHistoryLimit bound status.recentRuns.
- Credentials: backup runs only with a ClickHouse username and password the operator
configuration sets itself, in clickhouse.access or its Secret; the defaults do not
count (reason NoClickHouseAccess).
- A shard is backed up once any of its replicas made the backup, AllReplicas
included; retention takes a backup as complete per shard.
- Restore follows the Altinity usual backup workflow: download
--hardlink-exists-files on every host, restore --schema=1 on every replica - the
first replica of the first shard with RESTORE_SCHEMA_ON_CLUSTER={cluster} -,
restore --data=1 on one replica per shard, then delete local. Every restore
command takes --ignore-dependencies; --rm, which recreates the databases of the
backup whole, only for an overwrite without tables; the filters (tables,
partitions, skipTables, skipTableEngines, databaseMapping, rbac) go on every
command. The
overwrite guard, the separate ON CLUSTER drop pass, the max_table_size_to_drop
checks and the dependency on the ZooKeeper configuration of the CHI are gone.
- The sidecar pins API_CREATE_INTEGRATION_TABLES and API_INTEGRATION_TABLES_HOST;
the operator reaches each host by its FQDN, port, scheme and cluster TLS.
- Events of the runs (Orchestrator*), conditions and reasons (Started, Failed,
FailedUnchanged, NoClickHouseAccess) are posted by the operator.
- Removed: backup.mode and pod mode, backup.orchestrator, the bash and Python
orchestrator scripts and their ConfigMaps, orchestratorConfigMapRef, jobName and
cronJobName, concurrencyPolicy, backup verification (spec.verify, the Verified
condition and its reasons, the verification metric), backup.path of the operator
configuration, and the RBAC rights for Jobs, CronJobs, Pods and RBAC objects.
- Names: {shard} of a naming template is the shard macro, {shardNum} the shard_num
from 1; restore keeps the naming prefix.
- The sidecar runs with TZ=UTC; restore_distributed_cluster defaults to {cluster}.
- spec.timeouts of backups, restores and schedules, with defaults in the operator
configuration (backup.timeouts) and in code: actionTimeout (12h),
unreachableTimeout (10m), httpReceiveTimeout (4h), errorResponsesMax (5), as Go
durations. Errors and empty polls are counted apart, with the response code.
- A failed backup deletes only its local copy, once; a failed restore keeps its
downloads, deleted only after success.
- Retention: keepBackups pins {name, until}; backups held by a schedule change, and
pins without until of such backups, expire by the schedule before the change
(backup time + keep x period of its longest entry that kept it), listed in
heldBackups with expires; an incomplete backup is deleted only when no
create_remote or upload of it runs on its shard per system.backup_actions.
- Storage paths: path and object_disks_path, macros expanded per host, must be
unique among the writers of one backup (AllReplicas: every replica), embedded
backups included; otherwise the backup or schedule is Aborted with
StoragePathConflict before any command.
- Finalizers: deleting an unfinished backup kills its commands and deletes its
local and remote copies; a completed one keeps its remote backup; a restore and a
schedule run are waited for.
- spec.taskID: a completed backup runs again under a new name, an unfinished one
under its own; a failed restore runs again, resuming its downloads
(spec.resumeDownload, default true) or starting over; an aborted schedule is
validated again. A taskID during a run is ignored with a warning.
- Restore: spec.replicatedCopyToDetached (default true) - the other replicas of a
shard restore the data of replicated tables to detached first, then one replica
attaches it; false downloads only the schema on them.
- A restore of access entities holds the backups and schedules of its own clusters
only; restores of different clusters run at the same time.
- Manifests regenerated with the current yq: plain <<: merge keys.
- CRDs, manifests, Helm chart, docs/backup.md and the examples follow; unit tests
cover the runs on fake sidecars; e2e test_010087, test_010088 and test_010089
(retention across a schedule change, keepBackups) cover real sidecars.
- ci: actions/setup-python@v7, which runs on Node.js 24.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
eaf9557 to
a9e00ff
Compare
|
@Slach , for consistency sake, could you change the secret reference to valueFrom syntax? Like: It has been missed in secrets review. |
The storage Secret of the operator-managed backup is referenced with the valueFrom/secretKeyRef syntax cluster secrets and settings use, and the key of the Secret is the user's choice instead of a fixed config.yml. The key is required: an empty key, hotReload or optional refuse the backup block (BackupConfigInvalid, BackupStorageConfigInvalid event). Inheritance and the explicit secretKeyRef.name "" opt-out work as before. The generated configuration Secret and the sidecar mount keep config.yml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
claude AI review, maybe slop but JFYI: |
|
API:
also slot matching is fragile because of DST time changes. Missed runs. Say operator was down, when backup will be done? another point to the bucketed retention above is better control that nothing will be deleted if there was a miss. types:
Jitter, we start backup on all replicas without delay now as I understand? One-off backups, no pruning strategy for them? ttl or some keep until? |
|
@denisok thanks for review Thanks, I went through both comments point by point against the code. AI reviewReal, will be fixed before merge:
Not a bug:
APIChanging now, while unreleased:
Keeping, with the reasons:
Agreed that consistency with the Altinity usual backup workflow is the baseline, not the goal. The PR already goes beyond it, e.g. held backups survive a retention change until they expire by the old schedule, and the DST fixes above are cases the old workflow never handled. |
- The finalizer of a backup that sent no command, as one refused in preflight for a backup of its name the storage holds already, deletes nothing: only the hosts a command was inserted into, or may have been, are cleaned up. - A backup manager that loses the leader Lease is built again and elected again with backoff, until the operator stops; the run registry settles the runs of the ended term, drops those it interrupted so the next term resumes them, and refuses runs outside a term. A run ends before it tells its resource. - A daily, weekly or monthly slot whose time does not exist the day the clocks go forward is classified, by orchestration and retention alike, as the slot of its entries at the time the zone moves it to. - A wall-clock time the zone repeats as the clocks go back is a slot once, its first occurrence, so the second run no longer collides with the first name. - The command a resumed run adopts marks the run mutated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…um, namespaced leases, endpoint normalization - The generated configuration Secret is updated only when it carries the operator's labels for the CHI; a foreign one of the same name is left alone and aborts the reconcile with BackupConfigInvalid naming it. - YAML errors of the storage Secret name the line at most, never the content. - startingDeadlineSeconds of a schedule and activeDeadlineSeconds of backups, schedules and restores are capped at 2147483647 in validation and the CRDs. - Leases are granted by a namespaced Role in the operator namespace, not by the ClusterRole, in every install flavor and in the Helm chart. - The storage path check compares endpoints and addresses normalized. - configOverrides refuses settings that weaken the security of the storage. - Comment fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…backups, docs tables and skipTables of ClickHouseBackup, ClickHouseBackupSchedule and ClickHouseRestore are lists of patterns, one per item, joined by commas as partitions are, each checked by the pattern regexp without a comma, at most 64 items and 1024 characters joined. Backups and schedules take skipTables and skipTableEngines, rendered as --env=CLICKHOUSE_SKIP_TABLES (after the default system skip list) and --env=CLICKHOUSE_SKIP_TABLE_ENGINES on create_remote, and configOverrides refuse both keys. CRD templates, examples, e2e manifests and docs updated, with docs of missed slots, jitter, one-off backups, the incremental entry, the day of an entry and adding a shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the latest due slot of a schedule runs; the slots missed before it, while the operator did not run or a run of the cluster was in progress or held, were dropped silently. A ScheduledRunSkipped warning now tells how many slots of the cluster were missed since its last one, from which, and which slot runs instead, counted up to 10000. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The announcement that a deleted restore or schedule waits for its runs was marked made before it was emitted: a status write that failed on a conflict, or a reconcile that changed no status, dropped it for good. It is marked made only as it is emitted now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix #2146
What changes
The operator itself runs backups and restores. There is no orchestrator script, no Job, no CronJob and no backup Pod: the operator drives the
clickhouse-backupsidecar of every host through SQL on the integration tables of that host's ClickHouse, as the Altinity usual backup workflow does.Transport: sidecar only. Every Pod of a backup-enabled cluster gets a
clickhouse-backup serversidecar. Its configuration, generated by the operator, always turns onapi.create_integration_tables, sosystem.backup_actionsandsystem.backup_listreadhttp://127.0.0.1:7171. The sidecar runs withTZ=UTC, andrestore_distributed_clusterdefaults to{cluster}.backup.modeand pod mode are removed.Credentials. The operator connects to each host by its FQDN, port, scheme and the cluster TLS, with the ClickHouse credentials of its own configuration (
clickhouse.accessor its Secret). They must be set explicitly: the built-in defaults do not count, and without them a backup or restore waits with reasonNoClickHouseAccess.Commands. A command is a row inserted into
system.backup_actionsof the host afternow('UTC')of that host (async_insert = 0,http_receive_timeout/http_send_timeoutfromhttpReceiveTimeout), and polled every 5 seconds, matched by its exact command text and start time, until it finishes. A busy sidecar (another operation is currently running) is retried with exponential backoff from 5 seconds up to 5 minutes. After an operator restart the leader resumes every run from its status: a running command is followed, a finished one is settled.Timeouts.
spec.timeoutsof aClickHouseBackup,ClickHouseRestoreandClickHouseBackupSchedule, defaults in the operator configurationbackup.timeoutsand in code, as Go durations:actionTimeout(12h),unreachableTimeout(10m),httpReceiveTimeout(4h),errorResponsesMax(5). Errors and empty polls are counted apart; the logs carry the response code.Backup.
create_remote --rbacon one replica per shard (replicaSelection: FirstPerShard,preferredReplica, fallback to another replica reported byOrchestratorReplicaFallback) or on every replica (AllReplicas). A shard is backed up once any of its replicas made the backup. A failed backup deletes only its local copy, once, and reports a remote leftover (OrchestratorFailedBackupLeft,OrchestratorFailedBackupRemoteLeft) with the SQL to delete it by hand.Storage paths. Before any command,
pathandobject_disks_pathof every writer of a backup, with the macros of each host expanded, must be unique (AllReplicas: every replica; embedded backups and CAS disks included). Otherwise the backup or schedule isAbortedwithStoragePathConflict, e.g.storage path 's3://backups/backup/main/0' is not unique: replicas ... both write a backup of one name there. Unknown macros give anOrchestratorStoragePathAdvicewarning.Backup names.
spec.backup.naming/clusters[].backup.naming, a template such asshard{shardNum}-{YYYYMMDDHHIISS}with{shard}(the shard macro),{shardNum}(shard_num, from 1),{cluster},{chi},{namespace},{prefix}and the time tokens. Retention, the incremental base and restore parse names back with it, so backups named by the Altinity usual backup workflow on a shared remote path are recognized. Empty keeps the current names.Schedules.
ClickHouseBackupSchedule.spec.schedules[](minute,hour,6hours,daily,weekly,monthly, withtime,day,keep,incremental,configOverrides) andspec.timeZone:ClickHouseBackupis created per run: a run lives instatus.activewhile it runs and instatus.recentRunsafterwards, bounded bysuccessfulRunsHistoryLimit(3) andfailedRunsHistoryLimit(1);startingDeadlineSecondsskips a slot missed by more; runs of one cluster never overlap;ClickHouseBackupwithscheduleRefruns the schedule now: its incremental entry, retention after it, a base of the next incremental backup.Retention follows the Altinity usual backup workflow: each entry keeps its own
keepnewest backups (GFS); the newest backup is always kept; run-now backups live within the window of the scheduled ones; an incomplete backup is deleted only once a newer complete one exists andsystem.backup_actionsshows nocreate_remoteoruploadof it on its shard; nothing is deleted during a restore; incremental bases are rebased byclickhouse-backup(forcedbackups_to_keep_remote: 0,rebase_*: true).Schedule changes.
spec.retention.onScheduleChange: Hold(default) records a change of the entries or the time zone instatus.retention.previous[]with itsexpires. The backups the schedule kept before it and the current one would delete are held, listed instatus.clusters[].heldBackupswithexpires: backup time +keep× the period of the longest old entry that kept it. Expired holds are deleted (OrchestratorRetentionHeldExpired), an expired change leavesprevious[](RetentionHeldExpired).keepBackups: [{name, until}]pins backups: withuntiluntil then; without it a backup held by a change until its expiry, any other backup while it is listed. Pins protect incomplete backups too.releaseHeld: <changeID>releases a change and all earlier ones,Pruneapplies a change at once.Config overrides.
schedules[].configOverridesandClickHouseBackup.spec.configOverrides: up to 16KEY=VALUEclickhouse-backup settings for one backup, sent ascreate_remote --env=.... Keys the operator pins (TZincluded),API_*, paths, buckets, endpoints, the embedded backup disk and secrets are refused.Restore, as in the Altinity usual backup workflow:
download --hardlink-exists-fileson every host (an incomplete local copy is resumed);restore --schema=1 --ignore-dependencieson every replica, the first replica of the first shard withRESTORE_SCHEMA_ON_CLUSTER={cluster}; skipped forpartitions;spec.replicatedCopyToDetached(default true): the other replicas of each shard runrestore --data=1 --ignore-dependencies --replicated-copy-to-detachedat the same time, so the parts of replicated tables are already local; with false they download only the schema;restore --data=1 --ignore-dependencieson one replica per shard, the next answering replica when the first does not;delete localon every host after success; a failed restore keeps its downloads.Filters go on every command:
tables,partitions,skipTables,skipTableEngines,databaseMapping,rbac.overwrite: truewithouttablesadds--rm, which recreates the databases of the backup as a whole; withtables, or withoutoverwrite, only the tables of the backup are recreated (clickhouse-backupdrops the existing ones) and no database is dropped. A restore of access entities holds the backups and schedules of its own clusters only; restores of different clusters run at the same time.Reruns:
spec.taskIDon all three kinds, withstatus.taskID,taskIDsStarted,taskIDsCompleted. A completed backup runs again under a new name (the old one instatus.previousBackupNames), an unfinished one under its own name after its earlier copies are deleted. A failed restore runs again:spec.resumeDownload(default true) resumes its earlier downloads, false deletes them and starts over. An aborted schedule is validated again. AtaskIDset while a run is active is ignored with aTaskIDIgnoredwarning.Deletion. Finalizers on
ClickHouseBackup,ClickHouseRestoreandClickHouseBackupSchedule. Deleting an unfinished backup kills only its own running commands (kill "<command>") and deletes its local and remote copies; a completed one keeps its remote backup for retention. A restore and a schedule run are waited for, and a deleted schedule starts no new slot. Unreachable hosts are given up afterunreachableTimeout, with aDeletionLeftoverwarning and the SQL to finish by hand.Validation. Backup enabled without
configSecretRef, a missing or invalid storage Secret, or an invalidbackupblock stops the CHI reconcile: statusAborted, reasonBackupConfigInvalid, Warning eventBackupStorageConfigInvalid.Status and events.
status.clusters[].runof aClickHouseBackup/ClickHouseRestorerecords the step, the hosts and every command sent with its status. PhasesPending,Running,Completed,Failed,Aborted. The operator posts every event itself (Orchestrator*per step and command) on theClickHouseBackup,ClickHouseBackupScheduleandClickHouseRestore.Removed:
backup.modeand pod mode,backup.orchestrator, the bash and Python orchestrator scripts and their ConfigMaps,orchestratorConfigMapRef,backup.pathof the operator configuration,jobName/cronJobName,concurrencyPolicy, backup verification (spec.verify, theVerifiedcondition and its metric), the restore overwrite guard, and the operator RBAC rights for Jobs, CronJobs, Pod writes, ServiceAccounts, Roles and RoleBindings. The operator gainsupdate/patchon its three backup kinds for the finalizers.Manifests are regenerated with the current yq, which writes plain
<<:merge keys in place of!!merge <<:; the documents are the same.Docs:
docs/backup.md(how the operator runs an operation, naming, timeouts, storage paths, migration from the Altinity usual backup workflow, schedule changes, reruns, deletion, config overrides, integration tables),docs/custom_resource_explained.md,docs/operator_configuration.md,docs/chb-examples/.Tests
go build ./... && go test ./...,-raceforpkg/controller/chbackupandpkg/model/chbackup: runs against fake sidecars (command text, polling, timeouts, resume after restart, busy backoff, replica fallback, copy to detached, finalizers, taskID reruns, path uniqueness, per-cluster gates), retention plans, schedule changes and their expiry, naming, slots, incremental base.helm templateof the regenerated chart.test_010087: backup, schedule, retention and restore of two clusters through real sidecars;test_010088: recovery of a new CHI from another one's backup;test_010089: retention across a schedule change: held backups expiring by the old schedule,keepBackupspins withoutuntil, withuntilin the past and in the future.Related, in
clickhouse-backup:/metricson its own address, so the API can stay on127.0.0.1;start/finishofsystem.backup_actionswithout a time zone (worked around here withTZ=UTC);system.backup_actions;DROP ... ON CLUSTERunderRESTORE_SCHEMA_ON_CLUSTERandmax_table_size_to_drop.deploy/operatorhubis not regenerated (done at release).dev/run_code_generator.shdoes not build with the current Go (#2152); deepcopy was generated with deepcopy-gen v0.34.1.🤖 Generated with Claude Code