Repository navigation
Antalya 26.3 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2907
Conversation
…tateful-function-spans-chunks Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block Conflicts committed as-is; the next commit resolves them: - src/Storages/IPartitionStrategy.cpp (both modified) - src/Storages/IPartitionStrategy.h (both modified)
Kept the PR's `getPartitionKeyActions` design; dropped the upstream-only `getPathForRead`/`getPathForWrite` and kept antalya's `computePartitionKey(Block&)`; updated the DeltaLake sink caller.
|
Port-AL: cherry-pick conflicts resolved by AI Conflicted files:
Commits:
|
|
@blau-ai inspect test fails and fix fails caused by this PR |
CI triage for
|
| Check | Verdict | Evidence |
|---|---|---|
| Integration tests, all 13 shards (2184 failing tests) | Infra | 2142 of the 2184 fail inside cluster.start() → helpers/cluster.py:3513, where images_pull_cmd times out (subprocess ... communicate(timeout=600)). The Docker image pull timed out, so whole modules failed before any test ran: test_database_replicated_settings (780), test_backup_restore_new, test_storage_delta, test_mysql57_database_engine, test_grpc_protocol, … |
↳ the other 42: test_database_glue/test.py (36), test_storage_s3/test_sts.py (6) |
Already failing on the base | The same tests fail on #2898. |
| Stress test (amd_ubsan): hung check / possible deadlock on shutdown | Unrelated | The only stuck query thread in gdb.log is on the read path: StorageObjectStorageSource::lazyInitialize → createReader → KeysIterator::next → S3ObjectStorage::getObjectMetadata → S3::getObjectInfo (waiting on an S3 HEAD request). No PartitionedSink or IPartitionStrategy frames anywhere. This PR only changes the sink/write side. Stress amd_debug and amd_release passed. |
Stateless (amd_ubsan, sequential): 04306_delta_lake_merge_missing_table_state (got \N \N, expected 1 2) |
Flaky / unrelated | It's a DeltaLakeLocal read test that uses the datalake_simulate_missing_table_state failpoint. The PR's only DeltaLake change is in DeltaLakePartitionedSink::consume (write). The same test passed in amd_debug sequential, asan sequential and s3-storage sequential on this PR, and in ubsan sequential on #2898. |
Stateless (amd_debug, parallel): 00172_hits_joins |
Infra / timeout | Timeout! Processes left in process group after 600 s. It's a hits-dataset join and has nothing to do with partitioned writes. The job itself is green. |
| SQLLogic test | Already failing on the base | Hits the 3h timeout on #2898 as well. |
| Grype scans (keeper, server-alpine) | Already failing on the base | Same "4 high/critical vulnerabilities" on #2898. |
| DCO | Already failing on the base | Fails on #2898 too. Backport commits carry no sign-off. |
What remains
- Re-run the integration jobs (they don't need code changes). With almost every test lost to Docker pull timeouts, those shards say nothing about this PR either way. The tests that exercise this code (
test_storage_s3,test_storage_delta,test_export_*_to_iceberg,test_export_merge_tree_part_to_object_storage) only show results once the pull works. - I didn't run anything locally because there was no fix to validate. The verdicts above come from the CI reports, and a green integration re-run on the PR should confirm them.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fixed
PARTITION BYoffile,urland object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1:PARTITION BY intDiv(rowNumberInAllBlocks(), N)restarted its counter on every block and wrote rows into the wrong files, andnow64()produced one file per block instead of one perINSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: