Skip to content

Antalya 26.6 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2909

Merged
zvonand merged 3 commits into
antalya-26.6from
backports/antalya-26.6/128217
Nov 20, 2026
Merged

zvonand merged 3 commits into
antalya-26.6from
backports/antalya-26.6/128217

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed PARTITION BY of file, url and object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1: PARTITION BY intDiv(rowNumberInAllBlocks(), N) restarted its counter on every block and wrote rows into the wrong files, and now64() produced one file per block instead of one per INSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

alexey-milovidov and others added 2 commits November 18, 2026 15:51
…tateful-function-spans-chunks

Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block

Conflicts committed as-is; the next commit resolves them:
- src/Storages/IPartitionStrategy.cpp (both modified)
- src/Storages/IPartitionStrategy.h (both modified)
Dropped the upstream `getPathForRead`/`getPathForWrite` hunks, which don't exist on antalya-26.6, and kept a one-argument `computePartitionKey(Chunk)` for existing callers.
@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Port-AL: cherry-pick conflicts resolved by AI

Conflicted files:

  • src/Storages/IPartitionStrategy.cpp (both modified)
  • src/Storages/IPartitionStrategy.h (both modified)

Commits:

  • b554bdd: cherry-pick with the conflicts committed as-is

  • d551664: resolution

  • IPartitionStrategy.h conflict: The derived-struct declarations conflicted. antalya-26.6 has no getPathForRead/getPathForWrite on the partition strategies, because path generation lives in ObjectStorageFilePathGenerator. I kept only getPartitionKeyActions() and the computePartitionKey(Block &) override, and dropped the Chunk override.

  • IPartitionStrategy.cpp conflict: The upstream side carried the whole Hive getPathForRead/getPathForWrite implementations, which antalya-26.6 doesn't have. I removed them and left only HiveStylePartitionStrategy::getPartitionKeyActions(). The Wildcard getPartitionKeyActions() had already merged cleanly.

  • Kept the old one-argument call: I added a non-virtual one-argument computePartitionKey(const Chunk &) to the base class that calls getPartitionKeyActions(). DeltaLakePartitionedSink.cpp still calls that form, so nothing else needed touching. PartitionedSink uses the new overload that takes precomputed actions, so a stateful function keeps its state across the chunks of one INSERT.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [02439a6]

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for 02439a6

4 red checks. None are caused by this PR: 0 PR-caused, 2 stress-test crashes in unrelated code, 2 infra/process.

The PR changes only IPartitionStrategy.{h,cpp}, PartitionedSink.{h,cpp} and the new test 05459_partition_by_stateful_function_spans_chunks. None of the crashes go through partitioned writes (PARTITION BY on file or object storage). All stateless, integration and regression suites passed, including the new test.

Check What failed Verdict
Stress test (amd_debug) Abort on a LIMIT -3 BY query, and the server could not restart because of an Alias→Alias table Not this PR (a known upstream issue plus a random stress crash)
Stress test (amd_asan_ubsan) Logical error: 'Not-ready Set ...' for function 'nullIn' Not this PR (a known upstream bug family)
GrypeScanServer / Grype Scan Job ran for 6h5m and hit the runner time limit Infra
DCO Commits have no Signed-off-by Process. Every antalya-26.6 backport fails it (#2900, #2848)

Stress test (amd_debug): 3 of 4 sub-checks failed

  1. Received signal 6 while running a plain negative LIMIT BY over numbers():
    (query: SELECT ... FROM numbers(15) AS `__table1` ORDER BY `__table1`.`number` % 3 ASC, `__table1`.`number` ASC LIMIT _CAST(-3, 'Int64') BY `__table1`.`number` % 3) Received signal Aborted (6)
    ... DB::executeQueryImpl ... executeQuery.cpp:2247 ... TCPHandler::runImpl
    
    The query comes from the negative-LIMIT BY tests (03837_negative_limit_by, 04108_negative_limit_by_edge_cases, …). It reads no storage and writes nothing, so PartitionedSink is not involved. The abort happens inside executeQueryImpl and no Logical error line was logged before it, which points to a debug-only consistency assertion in the negative LIMIT BY feature.
  2. Cannot start clickhouse-server after the stress run:
    Load job 'load table test_ycrjgnmt.outer_alias_05213' failed: Code: 36. DB::Exception: Alias table cannot refer to another Alias table ... ENGINE = Alias('test_ycrjgnmt', 'inner_alias_05213')
    
    This table was left behind by 05213_storage_alias_chain_join_snapshot.sql. Upstream tracks it as Server cannot start after restart with an Alias → Alias chain in metadata: CREATE-time validation enforced at ATTACH breaks stress tests ClickHouse/ClickHouse#116207: the check that blocks an Alias→Alias chain on CREATE is also enforced on ATTACH, so the server cannot start after a restart. That issue is closed upstream, but the fix doesn't seem to be in antalya-26.6 yet. Either way it has nothing to do with this PR.
  3. "Check failed" is the summary of the two failures above.

Stress test (amd_asan_ubsan): 1 of 5 sub-checks failed

<Fatal> : Logical error: 'Not-ready Set is passed as the second argument for function 'nullIn''.
(query: SELECT count() = 33333 FROM null_in_subquery WHERE i IN (SELECT i FROM null_in_subquery WHERE dt != 1025)) Received signal Aborted (6)

The query comes from 01231_operator_null_in_2.sql (a SELECT with IN (subquery)). This is the open upstream "Not-ready Set" analyzer/planner bug family: ClickHouse#118722, ClickHouse#72739, ClickHouse#121039. It has nothing to do with partitioned writes.

GrypeScanServer / Grype Scan

The job ran for 6h5m, which is the GitHub Actions job time limit. That is a hang or timeout, not a reported vulnerability. On the sibling backports #2900 and #2848 the same GrypeScanServer job passed in 5–9 minutes. Safe to re-run.

DCO

The commits come from the backport bot (b554bdd, d551664) and a merge (02439a6), and none have Signed-off-by. Other antalya-26.6 backports (#2900, #2848) fail DCO the same way. Normally this is ignored or overridden by a maintainer. Fixing it would mean rewriting history, which needs a force-push, so I didn't touch it.

Caveat and suggested next steps

@zvonand
zvonand marked this pull request as ready for review November 20, 2026 13:30
@zvonand
zvonand merged commit ffa9bea into antalya-26.6 Nov 20, 2026
306 of 313 checks passed
@zvonand zvonand added the verified Approved for release label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rowNumberInAllBlocks() resets per chunk in S3/file PARTITION BY after #92844

3 participants