Skip to content

Antalya 26.8 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2910

Merged
zvonand merged 3 commits into
antalya-26.8from
backports/antalya-26.8/128217
Nov 20, 2026
Merged

zvonand merged 3 commits into
antalya-26.8from
backports/antalya-26.8/128217

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed PARTITION BY of file, url and object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1: PARTITION BY intDiv(rowNumberInAllBlocks(), N) restarted its counter on every block and wrote rows into the wrong files, and now64() produced one file per block instead of one per INSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

alexey-milovidov and others added 2 commits November 18, 2026 15:52
…tateful-function-spans-chunks

Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block

Conflicts committed as-is; the next commit resolves them:
- src/Storages/IPartitionStrategy.cpp (both modified)
- src/Storages/IPartitionStrategy.h (both modified)
Kept antalya's `computePartitionKey(Block&)` overrides, added the new `getPartitionKeyActions()` overrides, and left out upstream's `getPathForRead`/`getPathForWrite` context lines.
@blau-ai

blau-ai commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

Port-AL: cherry-pick conflicts resolved by AI

Conflicted files:

  • src/Storages/IPartitionStrategy.cpp (both modified)
  • src/Storages/IPartitionStrategy.h (both modified)

Commits:

  • 477745a: cherry-pick with the conflicts committed as-is

  • 4ca0a99: resolution

  • IPartitionStrategy.h: the Wildcard and HiveStyle strategies now declare getPartitionKeyActions() const override plus antalya's existing computePartitionKey(Block &) override. I removed the old computePartitionKey(const Chunk &) override.

  • IPartitionStrategy.cpp: the HiveStyle conflict left the old computePartitionKey(const Chunk &) header facing upstream's new getPartitionKeyActions definition. I kept only the getPartitionKeyActions signature.

  • Dropped the getPathForRead/getPathForWrite code that sat inside the conflict hunk. It came from newer upstream refactors and is not in antalya-26.8's header, so it isn't part of this PR's change.

  • The non-conflicting parts of the PR applied cleanly. These are the new IPartitionStrategy::computePartitionKey(chunk, actions) and the PartitionedSink caching of partition_key_actions.

  • StorageObjectStorage.cpp:907 still calls computePartitionKey(Block &), so that overload had to stay.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [1fcf913]

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for 1fcf913

Verdict: 4 red checks. None are caused by this PR. 2 are known failures already present on the antalya-26.8 line, 1 is a known regression-suite failure, and 1 is DCO process noise. No fixes are needed in the PR.

Check Failure Classification
Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) 03770_min_columns_to_activate_adaptive_write_buffer Pre-existing
Integration tests (amd_asan_ubsan, db disk, old analyzer, 3/8) test_s3_cluster/test.py::test_graceful_shutdown Pre-existing / flaky
RegressionTestsRelease / Swarms /swarms/feature/node failure/cpu overload, … cpu overload all swarm nodes Pre-existing
DCO Commits lack Signed-off-by Process, not code

For reference, this PR only touches IPartitionStrategy and PartitionedSink, which are used by partitioned INSERT INTO file()/s3()/…. The new test 05459_partition_by_stateful_function_spans_chunks passed in every lane it ran in.


1. 03770_min_columns_to_activate_adaptive_write_buffer (CAS S3 storage, ASan)

Code: 241. DB::Exception: Query memory limit exceeded: would use 50.24 MiB (attempt to allocate chunk of 1.00 MiB), maximum: 50.00 MiB. (MEMORY_LIMIT_EXCEEDED)
(query: insert into metric_log_1 select * from generateRandom() limit 1 settings max_memory_usage = '50Mi' ...)
... WriteBufferFromS3::WriteBufferFromS3 <- S3ObjectStorage::writeObject <- Cas::ObjectStorageBackend::publish <- Cas::PartWriteTxn::ensureBlobPresent <- Cas::fanOutBlobUploads

2. test_s3_cluster::test_graceful_shutdown

test_s3_cluster/test.py:2171 - in test_graceful_shutdown
    assert errors == 0
E   assert 3 == 0

3. Regression release swarms

✘ [ Fail ] '/swarms/feature/node failure/cpu overload' (5m 3s)
✘ [ Fail ] '/swarms/feature/node failure/cpu overload all swarm nodes' (5m 8s)

4. DCO

None of the 3 commits (477745aac3 upstream merge, 4ca0a999dc conflict resolution, 1fcf9136f9 base merge) has a Signed-off-by trailer. DCO is red the same way on the other backport PRs (#2905, #2913). This is the usual state for backports that carry upstream merge commits.

  • Next step: ignore it as on other backports, or sign off the commits. Signing off would require rewriting history, so I have not done it.

Bottom line: the backport looks clean from a CI standpoint. All remaining reds reproduce on unrelated antalya-26.8 PRs. I made no changes to the branch.

@zvonand
zvonand marked this pull request as ready for review November 20, 2026 13:31
@zvonand
zvonand merged commit b2b5725 into antalya-26.8 Nov 20, 2026
309 of 317 checks passed
@zvonand zvonand added the verified Approved for release label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants