Skip to content

Antalya 26.3 Backport of #131963 - Clear --jwt and the other credential options from the command line of clickhouse-client and clickhouse-keeper-client - #2930

Draft
blau-ai wants to merge 2 commits into
antalya-26.3from
backports/antalya-26.3/131963
Draft

blau-ai wants to merge 2 commits into
antalya-26.3from
backports/antalya-26.3/131963

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

clickhouse-client now clears the values of --jwt, --ssh-key-passphrase, --one-time-password and --oauth-client-secret from its command line, as it already did for --password, so they are no longer visible in ps and /proc/<pid>/cmdline. clickhouse-keeper-client now does the same for --password and --identity. Closes ClickHouse#131191. (ClickHouse#131963 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

…ptions-from-command-line

Clear `--jwt` and the other credential options from the command line of `clickhouse-client` and `clickhouse-keeper-client`
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [c13bea4]

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for 17d6e42

Summary: 7 failing checks. 1 is caused by this PR (4 stateless jobs, all the same new test). 5 are infra or unrelated to this PR. DCO is a process check.

Check Verdict
Stateless tests: amd_debug, parallel · amd_ubsan, parallel · amd_asan, distributed plan, 4/4 · amd_debug, distributed plan, s3 storage 🔴 PR-caused: new test 05813_clear_credentials_from_command_line fails in every job
Integration tests (amd_binary, 1/5) and (amd_asan, db disk, old analyzer, 7/8) ⚪ Infra: the cluster never started
Stress test (amd_ubsan) ⚪ Not related: Hybrid table couldn't attach on restart
Regression release swarms ⚪ Not related: timing-dependent assertion in a swarms scenario
Grype Scan (server) ⚪ Infra: the GitHub job was cancelled; the summary says 0 high/critical vulnerabilities
DCO ⚪ Process: no Signed-off-by on the backport merge commit (normal for backport PRs)

🔴 05813_clear_credentials_from_command_line: PR-caused (test isn't adapted to antalya-26.3)

Output from amd_debug, parallel (the other three jobs are identical):

05813_clear_credentials_from_command_line.sh: line 59: /proc/2140354/cmdline: No such file or directory
... (5 times)
stdout:
client, separate values
client, values after =
client, values after spaced =
client, --password without a value
keeper-client
keeper-client, values after :
--password:
--identity:

Cause: all four clickhouse-client cases pass --oauth-client-secret SECRET. That option does not exist on antalya-26.3. programs/client/Client.cpp (lines ~804–828) defines --password, --ssh-key-passphrase, --jwt, --one-time-password, --login, --oauth-credentials, --oauth-url, --oauth-client-id and --oauth-audience, but not --oauth-client-secret, which upstream added in a separate PR that wasn't backported. Option parsing fails on the unknown option, so the client exits before it connects to the listener. Its stderr goes to /dev/null in the test, so the failure is silent, and /proc/<pid>/cmdline is gone by the time the test reads it.

The first keeper-client case is collateral damage. The test counts accepted connections globally (CONNECTIONS vs wc -l $ACCEPTED_FILE). Clients 1–4 never connected, so case 5 waits for 5 connections while only keeper-client's reconnect attempts arrive. That ran for about 9 minutes (03:19 → 03:28) until keeper-client died. Case 6 then hit the count right away, which is why only the last block printed.

Suggested fix (test-only, the C++ change is fine):

  1. In 05813_clear_credentials_from_command_line.sh, remove --oauth-client-secret SECRET, --oauth-client-secret=SECRET and --oauth-client-secret = SECRET from the three client invocations. You can also drop oauth-client-secret from the grep -E pattern.
  2. In 05813_clear_credentials_from_command_line.reference, delete the three --oauth-client-secret / --oauth-client-secret= lines.
  3. Optional: keep "--oauth-client-secret"sv in secret_options in clearPasswordFromCommandLine.cpp. It's harmless and lowers the diff against upstream. Remove it only if you want the list to match the options 26.3 actually has.
  4. Optional hardening: redirect the client's stderr to a file instead of /dev/null, so the next mismatch like this shows up in the test output instead of as a missing /proc/<pid>.

I can push this fix to the PR branch if you want; just ask.

⚪ Integration tests: infra

The failures come from 8 whole modules (test_database_iceberg, test_storage_mongodb, test_storage_s3_queue/test_0,test_6, test_checking_s3_blobs_paranoid, test_distributed_inter_server_secret, test_grpc_protocol, test_named_collections_encrypted2), each failing in its cluster.start() fixture:

File: helpers/cluster.py:3513 - in start
    retry(...)(run_and_check, images_pull_cmd, timeout=600)
E   Failed: Timeout (>900.0s) from pytest-timeout.

(The other modules failed the same way while waiting for ZooKeeper in wait_zookeeper_nodes_to_start or Mongo in wait_mongo_to_start.) Docker image pulls and helper containers timed out before ClickHouse ran any query, so this is runner/registry infra. Safe to re-run.

⚪ Stress test (amd_ubsan): not related to this PR

Code: 695. Load job 'load table test_1.t3' failed: Code: 60. DB::Exception: Table test_1.local_hot does not exist:
Cannot attach table `test_1`.`t3` ... ENGINE = Hybrid(remote('localhost:9000','test_1','local_hot'), ...)

On restart the server couldn't load an Antalya Hybrid table whose underlying local_hot table had been dropped during the stress run. That's server table-loading code, and this PR only changes client-side argv scrubbing. I found no existing issue for it; worth tracking separately if it shows up on other 26.3 PRs.

⚪ Regression release swarms: not related to this PR

One scenario failed: /swarms/feature/feature/object storage cluster profile events with overloaded node:

assert int(sent_to_non_matched_replica) > 0, error()

This is a timing-dependent check on how object-storage-cluster tasks get distributed, and nothing in this PR touches it. 1520 of 1521 scenarios passed.


Next step: apply the test fix above. The PR-caused failures should then clear; the rest only needs a re-run. Evidence comes from the praktika result_pr.json for this SHA and the job logs of run 44625839776. I didn't run the test locally; this diagnosis is from the CI logs and the 26.3 option definitions.

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai fix fails caused by this PR here

…antalya-26.3 clickhouse-client

clickhouse-client in this branch has no --oauth-client-secret option, so every
client invocation in the test exited with an unrecognized-option error before
connecting, and reading /proc/<pid>/cmdline failed.
@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed a fix: c13bea4

Caused by this PR: 05813_clear_credentials_from_command_line

Failed in 4 stateless jobs (amd_debug, amd_asan distributed plan, amd_ubsan, amd_debug s3 storage), each time with:

05813_clear_credentials_from_command_line.sh: line 59: /proc/2140354/cmdline: No such file or directory

Root cause: the upstream test passes --oauth-client-secret to clickhouse-client, but that option does not exist in antalya-26.3. The client here only has --oauth-url, --oauth-client-id and --oauth-audience (programs/client/Client.cpp:824-828). Because of that, all four client cases exited with an unrecognized-option error before they connected to the dummy listener. By the time the test read /proc/<pid>/cmdline, the process was gone. The two keeper-client cases passed.

Fix (test-only): I removed --oauth-client-secret from the three client invocations and from the .reference. The C++ change is unchanged. Keeping --oauth-client-secret in the clearPasswordFromCommandLine list does no harm and keeps the code the same as upstream.

Validation: I ran the fixed test against this PR's CI amd_release binary on the builder (--reason "Verify 05813 passes once --oauth-client-secret (absent in antalya-26.3 client) is removed from the test"):

05813_clear_credentials_from_command_line:                              [ OK ] 1.18 sec.
1 tests passed. 0 tests skipped.

Not caused by this PR

Check Evidence
Integration tests (amd_binary 1/5, amd_asan db disk old analyzer 7/8) Whole modules (test_grpc_protocol, test_storage_mongodb, test_database_iceberg, test_storage_s3_queue, …) fail in cluster.start(): the docker image pull times out (helpers/cluster.py:3513, run_and_check(images_pull_cmd, timeout=600)). This is an infra problem.
Stress test (amd_ubsan) Cannot start clickhouse-server: the Hybrid table test_1.t3 can't attach on restart because test_1.local_hot does not exist (UNKNOWN_TABLE). This is a Hybrid-engine table-load-order problem and has nothing to do with clearing credentials from argv.
Regression release swarms, DCO, Grype Not related to the code in this PR. I did not look deeper into the swarms failure.

CI on the new head still needs to confirm the fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants