Skip to content

Antalya 26.3 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2907

Draft
blau-ai wants to merge 3 commits into
antalya-26.3from
backports/antalya-26.3/128217
Draft

blau-ai wants to merge 3 commits into
antalya-26.3from
backports/antalya-26.3/128217

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed PARTITION BY of file, url and object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1: PARTITION BY intDiv(rowNumberInAllBlocks(), N) restarted its counter on every block and wrote rows into the wrong files, and now64() produced one file per block instead of one per INSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

alexey-milovidov and others added 2 commits November 18, 2026 15:49
…tateful-function-spans-chunks

Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block

Conflicts committed as-is; the next commit resolves them:
- src/Storages/IPartitionStrategy.cpp (both modified)
- src/Storages/IPartitionStrategy.h (both modified)
Kept the PR's `getPartitionKeyActions` design; dropped the upstream-only `getPathForRead`/`getPathForWrite` and kept antalya's `computePartitionKey(Block&)`; updated the DeltaLake sink caller.
@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Port-AL: cherry-pick conflicts resolved by AI

Conflicted files:

  • src/Storages/IPartitionStrategy.cpp (both modified)
  • src/Storages/IPartitionStrategy.h (both modified)

Commits:

  • a1cd63b: cherry-pick with the conflicts committed as-is

  • b4d8dee: resolution

  • IPartitionStrategy.h: the Wildcard and Hive strategies now declare getPartitionKeyActions() and keep antalya's computePartitionKey(Block &) override. The computePartitionKey(const Chunk &) overrides are gone. I did not take getPathForRead/getPathForWrite from the PR side, because antalya-26.3 has no such virtuals in the base struct.

  • IPartitionStrategy.cpp: the conflict was in the Hive strategy. I removed the whole upstream-only block of HiveStylePartitionStrategy::getPathForRead and getPathForWrite, plus the old computePartitionKey(const Chunk &). What remains is getPartitionKeyActions() next to the existing computePartitionKey(Block &).

  • DeltaLakePartitionedSink.cpp (outside the conflict, but it would no longer build): it called the removed computePartitionKey(chunk). It now calls computePartitionKey(chunk, partition_strategy->getPartitionKeyActions()). Its partition columns are plain columns, so the actions stay cached and behaviour is unchanged.

  • PartitionedSink.cpp/.h and the new test merged cleanly.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [664fc50]

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai inspect test fails and fix fails caused by this PR

@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for b4d8deea5d4

Short version: none of the red checks come from this PR, so I didn't push anything. The new test 05459_partition_by_stateful_function_spans_chunks passed ([ OK ] in both amd_debug parallel and amd_ubsan parallel). For comparison I used #2898, another backport to the same base (30af90bd) that ran CI around the same time.

Check Verdict Evidence
Integration tests, all 13 shards (2184 failing tests) Infra 2142 of the 2184 fail inside cluster.start() → helpers/cluster.py:3513, where images_pull_cmd times out (subprocess ... communicate(timeout=600)). The Docker image pull timed out, so whole modules failed before any test ran: test_database_replicated_settings (780), test_backup_restore_new, test_storage_delta, test_mysql57_database_engine, test_grpc_protocol, …
↳ the other 42: test_database_glue/test.py (36), test_storage_s3/test_sts.py (6) Already failing on the base The same tests fail on #2898.
Stress test (amd_ubsan): hung check / possible deadlock on shutdown Unrelated The only stuck query thread in gdb.log is on the read path: StorageObjectStorageSource::lazyInitialize → createReader → KeysIterator::next → S3ObjectStorage::getObjectMetadata → S3::getObjectInfo (waiting on an S3 HEAD request). No PartitionedSink or IPartitionStrategy frames anywhere. This PR only changes the sink/write side. Stress amd_debug and amd_release passed.
Stateless (amd_ubsan, sequential): 04306_delta_lake_merge_missing_table_state (got \N \N, expected 1 2) Flaky / unrelated It's a DeltaLakeLocal read test that uses the datalake_simulate_missing_table_state failpoint. The PR's only DeltaLake change is in DeltaLakePartitionedSink::consume (write). The same test passed in amd_debug sequential, asan sequential and s3-storage sequential on this PR, and in ubsan sequential on #2898.
Stateless (amd_debug, parallel): 00172_hits_joins Infra / timeout Timeout! Processes left in process group after 600 s. It's a hits-dataset join and has nothing to do with partitioned writes. The job itself is green.
SQLLogic test Already failing on the base Hits the 3h timeout on #2898 as well.
Grype scans (keeper, server-alpine) Already failing on the base Same "4 high/critical vulnerabilities" on #2898.
DCO Already failing on the base Fails on #2898 too. Backport commits carry no sign-off.

What remains

  • Re-run the integration jobs (they don't need code changes). With almost every test lost to Docker pull timeouts, those shards say nothing about this PR either way. The tests that exercise this code (test_storage_s3, test_storage_delta, test_export_*_to_iceberg, test_export_merge_tree_part_to_object_storage) only show results once the pull works.
  • I didn't run anything locally because there was no fix to validate. The verdicts above come from the CI reports, and a green integration re-run on the PR should confirm them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants