Skip to content

[module-sdk] feat: propagate hook allowFailure to bindings - #144

Open
fuldaxxx wants to merge 1 commit into
mainfrom
fix/hook-allow-failure-fanout
Open

fuldaxxx wants to merge 1 commit into
mainfrom
fix/hook-allow-failure-fanout

Conversation

@fuldaxxx

@fuldaxxx fuldaxxx commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Description

  • Hook-level AllowFailure on pkg.HookConfig and pkg.ApplicationHookConfig is now propagated to every Kubernetes and Schedule binding in the hook config passed to addon-operator. This is the same fan-out the embedded go_hook API in addon-operator performs.
  • A per-binding AllowFailure that is set explicitly, including false, takes precedence over the hook-level value.
  • The wire-format ScheduleConfig gets an allowFailure field. Schedule bindings start honoring it once addon-operator copies the field into ScheduleConfigV1 (follow-up PR). Older addon-operator versions ignore the unknown field, because the config is decoded without DisallowUnknownFields.
  • OnStartup and before/after Helm bindings still cannot be allowed to fail. This is unchanged and now documented on the field.

Why do we need it, and what problem does it solve?

Top-level AllowFailure was silently dropped. remapModuleHookConfig and remapApplicationHookConfig never read it, and shell-operator only understands allowFailure per binding. So a hook declared with AllowFailure: true actually ran with allowFailure=false. This affects every module-sdk release from v0.10.4 up to v0.14.0. There is no compile or validation error. The existing test only covers the per-binding field.

The consequence is severe. When such a hook fails on a Kubernetes Synchronization task, addon-operator requeues the task forever (queue.Fail). The binding is never marked done, so the module stays in WaitForSynchronization, and ModuleRun repeats at the head of the main queue. Convergence of all other modules stops until the Deckhouse leader is restarted. This happened in production with a hook in the observability module: a PatchWithMerge returned NotFound for an object deleted between snapshot and patch.

The embedded go_hook API, which module-sdk replaces, fans the hook-level flag out to all bindings (newHookConfigFromGoConfig). Hooks migrated one-to-one therefore lost the semantics silently. Fan-out in the SDK restores the expected behavior. For Kubernetes bindings it works with every already released addon-operator, because they already read the per-binding field. Deprecating the field with a validation error was rejected: it would break hooks that compile today and would diverge from the legacy API semantics.

Hooks that already declare AllowFailure: true (e.g. log-shipper generate_config) will start tolerating errors as their authors intended. Errors are logged and counted in ModuleHookAllowedErrorsTotal instead of being retried.

Signed-off-by: Ruslan Gorbunov <ruslan.gorbunov@flant.com>
@fuldaxxx
fuldaxxx requested a review from ldmonster November 18, 2026 07:58
@fuldaxxx fuldaxxx self-assigned this Nov 18, 2026
@fuldaxxx
fuldaxxx marked this pull request as ready for review November 18, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant