Repository navigation
Conversation
Since ClickHouse#110068 a pooled connection is validated with a zero-timeout poll and not with a `Ping`. The poll sees a closed socket, but not a server that stopped without closing it (for example, `SIGSTOP`). For a cluster table function such as `fileCluster`, `s3Cluster` or a swarm read, such a replica stays in the pipeline and the query waits for `receive_timeout` (300 seconds by default) before it fails with `SOCKET_TIMEOUT`. Before that PR the `Ping` skipped the replica after `sync_request_timeout`. `ReadFromCluster::initializePipeline` now passes `ping_pooled_connection` to `getMany` for every shard that is not local. A local shard is excluded because `ReplicasAsShards` gives this server's own address a real TCP pool, and pinging that address does not reveal a dead peer. The ping is limited by `sync_request_timeout`. A failed ping raises `SOCKET_TIMEOUT` inside the synchronous `ConnectionEstablisher::run`, and that path already drops the socket and reconnects once. A connection that is not established yet is still checked by the handshake, so no extra round trip is added. The asynchronous path (hedged connections and `tryGetEntry` with an async callback) keeps the poll, because a blocking ping would stall the fiber. Other callers are unchanged and keep the poll-based validation. Adds the `unresponsive_pooled_connection` failpoint, which makes `Connection::ping` fail, and the stateless test `05812_ping_pooled_connection_on_cluster_read`. Issue: #2896 Co-authored-by: Cursor <cursoragent@cursor.com>
`Connection::ping` is private, so `ConnectionEstablisher::run` failed to compile in b3f14c5. Call `Connection::checkConnected`, the public wrapper around `ping`. The branch that calls it is already known to be connected, so the call only runs the ping. Co-authored-by: Cursor <cursoragent@cursor.com>
A cluster read pings a pooled connection to a remote shard before using it. If the peer kept the socket open and sent no `Pong` within `sync_request_timeout`, the establisher reconnected once and waited up to `handshake_timeout_ms` for a `Hello` that a stopped process never sends. `Connection::checkConnectedThrowOnTimeout` lets a `SOCKET_TIMEOUT` out of `Connection::ping`. `ConnectionEstablisher::run` then skips the replica and does not reconnect. A socket the server has already closed still returns false from `ping`, so it still reconnects once. `checkConnected` keeps its old contract for the client resync path. Adds the `pooled_connection_ping_timeout` failpoint and two checks to `05812_ping_pooled_connection_on_cluster_read`: a timed-out ping does not increase `DistributedConnectionReconnectCount`, and it does increase `DistributedConnectionFailTry`. Related: #2896 Co-authored-by: Cursor <cursoragent@cursor.com>
A pooled remote connection that does not answer `Ping` within `sync_request_timeout` was skipped. A live server behind a dropped flow was then dropped from the cluster read, and the query could fail with `ALL_CONNECTION_TRIES_FAILED` even though a new connection would succeed. The establisher now reconnects once after a `SOCKET_TIMEOUT` from the ping, the same way it already does for a dead socket. The handshake of that retry is capped by `sync_request_timeout`, so a stopped server does not wait out `handshake_timeout_ms`. The cap only shortens `handshake_timeout` on that attempt and never raises it. The `pooled_connection_ping_timeout` failpoint test now expects `DistributedConnectionReconnectCount` to increase, and both remote and local shards to be usable after the reconnect. Related: #2896 Co-authored-by: Cursor <cursoragent@cursor.com>
…e-cluster-replicas
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a839c6dc57
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| attempt_timeouts.handshake_timeout = ConnectionTimeouts::saturate( | ||
| attempt_timeouts.handshake_timeout, attempt_timeouts.sync_request_timeout); |
There was a problem hiding this comment.
Cap zero-valued handshake timeouts
When a cluster read uses handshake_timeout_ms=0 and a pooled peer times out on Ping, saturate(0, sync_request_timeout) leaves the retry's handshake timeout at zero. Connection::connect then installs that zero value as the socket send/receive timeout, so a paused peer that accepts TCP but never sends Hello can block indefinitely instead of being skipped after sync_request_timeout, defeating the new bounded-reconnect behavior. Treat a zero handshake timeout as unbounded here and replace it with sync_request_timeout; otherwise take the minimum.
Useful? React with 👍 / 👎.
The reconnect after a `Ping` timeout caps the handshake timeout by `sync_request_timeout` with `ConnectionTimeouts::saturate`. `saturate` keeps any value smaller than the limit, so a zero `handshake_timeout_ms` stayed zero. A zero socket timeout never fires, so a peer that accepts TCP and never sends `Hello` could block the retry indefinitely instead of stopping at `sync_request_timeout`. Treat a zero handshake timeout as no limit and replace it with `sync_request_timeout`. Any other value is still capped to the minimum of the two. Related: #2912 Co-authored-by: Cursor <cursoragent@cursor.com>
`TimeoutSetter` with `limit_max_timeout` keeps the current socket timeout when it is not longer than the requested one. A zero socket timeout means no limit, but it is not longer than any finite value, so it was kept. `Connection::ping` uses this setter with `sync_request_timeout`, so a pooled connection with `receive_timeout = 0` could wait for a `Pong` forever when the peer accepted the connection and never answered. Treat a zero send or receive timeout as no limit and install the requested timeout in its place. A finite timeout that is already shorter is still kept. The destructor restores the original zero timeout, so the query itself stays unlimited. Related: #2912 Co-authored-by: Cursor <cursoragent@cursor.com>
The comment in `Connection::ping` said a timeout skips the replica. The cluster-read caller does not skip it on the first timeout: it reconnects once with the handshake capped by `sync_request_timeout`, and the replica is skipped only if that reconnect fails. Describe the actual flow. No code change. Related: #2912 Co-authored-by: Cursor <cursoragent@cursor.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8be0a0f986
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// closing the socket otherwise stays in the pipeline and the query waits for | ||
| /// `receive_timeout`. No `Pong` within `sync_request_timeout` drops that socket and opens | ||
| /// one new connection. The handshake on that connection is capped by `sync_request_timeout`. | ||
| const bool ping_pooled_connection = !shard_info.isLocal(); |
There was a problem hiding this comment.
Avoid serial preflight pings across remote shards
For a cluster function with many remote shards and pooled connections, this loop invokes getMany synchronously for each shard, and enabling ping_pooled_connection makes every invocation complete a Ping/Pong round trip before the next shard is considered. Healthy queries therefore pay the sum of all shard RTTs before the pipeline can start, while several slow peers can each consume the five-second sync_request_timeout; previously the resulting RemoteSource pipes performed their network work concurrently. Ping or acquire these connections concurrently so startup latency is bounded by the slowest shard rather than multiplied by the shard count.
Useful? React with 👍 / 👎.
This is a very annoying type of bug we are starting to face everywhere. Upstream features our antalya features depend on keep changing and silently break.. |
…e-cluster-replicas
Solved #2896
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Ping unresponsive cluster replicas
Documentation entry for user-facing changes
PR ClickHouse#110068 had removed ping for replicas before each query.
But in Swarm cinfiguration is expected that Swarm node can be donw in any moment.
PR returns ping back for cluster requests.
CI/CD Options
Exclude tests:
Regression jobs to run: