Repository navigation
Move the engine pin to 76e6bf79426 and follow its text for a folded range with an exclusive bound - #770
Move the engine pin to 76e6bf79426 and follow its text for a folded range with an exclusive bound#770el-marvin wants to merge 3 commits into
Conversation
…n the predicate tests
…f a folded range as two comparisons
el-marvin
left a comment
There was a problem hiding this comment.
Changes requested.
I reviewed dfca0f4 against the merge base 32b9950 (round 1, so there is no earlier delta). The branch holds the test change fd7028f, a merge of v2.0-cyanoptera at 32b9950 that is identical to what git merge-tree produces, and the pin move dfca0f4. The net change is the external/duckdb pointer, 925bcdf3575 to 76e6bf79426, and five lines of tests/fast/arrow/scan/test_predicate_translation.py. I built the head and the merge base as Release builds the same way, and also used the released 2.0.0. The premise holds: Release run 43145686025 on v2.0-cyanoptera fails exactly the four cases this change updates (4 failed, 18152 passed, 129 skipped, 164 xfailed). That run builds the sdist against the engine branch tip 331f91a84f8, where engine commit f2090084542 (duckdb/duckdb#28448) prints a folded range with an exclusive bound as two comparisons. Following the engine's text is the right layer. The text belongs to the engine, it is now the true one, and the fragment the walker hands the Python source is byte for byte the same on both engines. The four expectations, the pointer and the claim that no bindings source must follow all hold.
What keeps this from merging is the record the pin move leaves. Engine commit 5b8ee21b05e makes the HUGEINT to DOUBLE conversion that the NumPy result path calls once per value about twice as expensive. fetchnumpy() and df() of HUGEINT columns, and of DECIMAL columns with a width of 19 to 38 (which every SUM over a DECIMAL returns), get 27 to 49 percent slower, and the description does not say so.
Findings
1. The pin move makes fetchnumpy and df of HUGEINT and wide DECIMAL columns 27 to 49 percent slower, through engine commit 5b8ee21b05e, and the description does not record it.
Severity: should fix. Category: performance. Evidence: ran. Fix: containment. Origin: engine. Base: introduced.
Ask: put the figures below in the description, report the regression to the engine with this reproducer (duckdb/duckdb#28429 is the change), and leave the question of waiting for an engine fix to the maintainer; no bindings change is wanted.
The NumPy conversion of a HUGEINT value, and of a DECIMAL stored as a hugeint before it divides by the scale, is the engine's cast (src/numpy/array_wrapper.cpp:354-358):
double IntegralConvert::ConvertValue(hugeint_t val, NumpyAppendData &append_data) {
(void)append_data;
double result;
Hugeint::TryCast(val, result);
return result;
}Commit 5b8ee21b05e ("Round HUGEINT and UHUGEINT casts to DOUBLE and FLOAT once") replaced two floating point operations in CastBigintToFloating (external/duckdb/src/common/types/hugeint.cpp:452) with a negation through uhugeint_t::operator- and a call to Uhugeint::TryCast, both out of line in uhugeint.cpp:
const bool negative = input.upper < 0;
uhugeint_t magnitude(static_cast<uint64_t>(input.upper), input.lower);
if (negative) {
magnitude = -magnitude;
}
Uhugeint::TryCast(magnitude, result);The correction itself is right: on the base, fetchnumpy() of '85070591730234625310576617597232480257'::HUGEINT gives 8.507059173023462e+37 and the head gives the nearest double, 8.507059173023463e+37. It is the cost that changed. Ten million values, one thread, best of five fetches, two alternating runs of each build, with the user space instructions of the calling thread for the best fetch (round-1/probes/bench_hugeint_numpy.py):
| column | base | head | instructions base, head |
|---|---|---|---|
| HUGEINT, values over the whole range | 110.9 ms | 158.0 ms (+42%) | 542 M, 942 M |
| HUGEINT, magnitudes below 2**53, both signs | 113.2 ms | 143.9 ms (+27%) | 537 M, 1024 M |
| DECIMAL(38,2), whole range | 97.2 ms | 144.5 ms (+49%) | 566 M, 966 M |
| DECIMAL(38,2), below 2**53 | 101.1 ms | 134.3 ms (+33%) | 561 M, 1048 M |
df() goes through the same conversion: DECIMAL(38,2) over the whole range takes 99.9 ms on the base and 139.0 ms on the head (+39%), 576 M against 977 M instructions (round-1/probes/bench_hugeint_df.py).
Two controls place it in the cast. A BIGINT column through the same NumPy path is unchanged: 79.9 ms against 75.5 ms, 156 M against 155 M instructions. The engine's own cast inside a query, SELECT sum(v::DOUBLE) over ten million HUGEINT values with no NumPy conversion, goes from 36.4 ms to 77.2 ms and from 355 M to 842 M instructions (round-1/probes/bench_controls.py). A variant of the head with only the old body of CastBigintToFloating restored gets the base's instruction counts back exactly: 537 M for HUGEINT below 2**53, 566 M for DECIMAL(38,2) over the whole range, 355 M for the SQL cast, at 96, 97 and 34 ms (round-1/probes/bench_variant.txt). Nothing else in the 59 commits contributes.
The same cast reached duckdb-python main with its pin move to 23221edfd577 (pull request 761), and engine main has not changed CastBigintToFloating since, so a fix belongs upstream for both lines. A client side conversion would be a second mechanism beside the engine's cast and could disagree with it, so I do not ask for one. My recommendation is to land the pin with the regression on record and an engine issue filed: the pin brings correctness fixes users want (below), and the 2.0 Release workflow stays red without it. Waiting for a faster correctly rounded cast is also defensible for a patch release line. That call is the maintainer's.
What the 59 engine commits ask of the bindings
I read every commit in 925bcdf3575..76e6bf79426 (56 commits and 3 merges) and the diff of each that touches a header under src/include or a behaviour the bindings rely on. The grouping is in round-1/engine-commit-categories.txt. In short:
- Headers the bindings include. The four the description names are exactly the changed ones among the bindings' direct
#includelines.bound_between_expression.hppgainsLowerComparisonType(bool)andUpperComparisonType(bool). The calls atsrc/arrow/predicate_translation.cpp:275-276pass aBoundFunctionExpression, which has no conversion tobool, so they keep their overload.add.hpprespells the TIMESTAMP specializations astimestamp_sec_t,timestamp_ms_tandtimestamp_us_t.timestamp_tandtimestamp_us_tare the same alias (timestamp_base.hpp:87-88), soAddOperator::Operation<interval_t, interval_t, interval_t>(src/native/python_objects.cpp:92-93) andInterval::Add(timestamp_t, interval_t)(:368) resolve as before.interval.hppgains TIMESTAMP_NS overloads and alsointerval_t::SameFields, from duckdb/duckdb#28331.settings.hppadds two HTTP retry settings beside theSettings::Get<EnableExternalAccessSetting>call atsrc/python_replacement_scan.cpp:481. Nothing to follow; the head compiles unchanged. - Engine functions the bindings call from their own code. Only the HUGEINT to DOUBLE cast above. It changes values (a correction) and cost (finding 1). UHUGEINT does not reach it from NumPy, because
src/numpy/raw_array_wrapper.cpp:70refuses the type before conversion. - Expression text and filter pushdown. f2090084542 is followed by the four expectations. 46347b94abe lets a filter through a TUPLE cast; the walker refuses it, so it stays in the engine. On both builds,
CAST(st AS TUPLE(INTEGER, VARCHAR)) = row(1, 'x')is offered and refused, with the same rows. 96b554036f9, fed8ee4a81d, 9c993a17c41 and the struct_extract folding in 923d9c5ed04 change nothing the walker reads. - Binder, parser, casts and functions. These are SQL semantics only. One reaches the bind seam. Since 0c5a115259b, an untyped parameter that meets an argument of type ANY or an incomplete type makes the planner defer the statement. So
con._bind("SELECT typeof(?) AS l")without values now reports['unknown']of type UNKNOWN where the base reported['l']. Every caller induckdb/_relation.pyeither passes the values, which type the parameter, or refuses parameters, so no relation built from runnable SQL sees the placeholder. The same commit does not slowexecutemany: 20,000 rows with a parameter intoconcatorformattake the same instructions on both builds (29.4 to 29.5 G on the head, 29.4 to 30.1 G on the base), because such statements already bound again at every execution on the base. A plainINSERT INTO t VALUES (?, ?)is equal too, at 10.9 G. - Storage, readers, runtime, extensions, C API, build. Nothing to follow. The new
maximum_files_for_statsoption defaults to 0 and is reachable through SQL;duckdb.read_parquetkeeps its fixed keyword list. No test or message in the bindings names the extension repository gate of 999a462d7eb. The deprecated C API fix of 675dabd347a is off the Arrow path the ADBC driver uses. bb4f82c8fa8 adds/Gwto MSVC release flags, which the Windows wheels pick up throughcmake/duckdb_loader.cmake:257. The CI wheel jobs cover that. - Generated lists.
duckdb_settings()grows from 164 to 166 rows andduckdb_functions()from 3302 to 3323.tests/fast/expression/test_expression_generated.pychecks the keyword, aggregate and function family modules against the engine, and passes on the head.
The two engine changes main's pin move had to follow, duckdb/duckdb#28324 and duckdb/duckdb#28576, are absent at 76e6bf79426 (no CastIsInvertibleForEquality, no stability declaration in glob.cpp), as the description says.
Smaller things
- The description calls 76e6bf79426 "the branch tip on 2026-11-12". It was the tip from 08:47 to 09:45 UTC. By the time dfca0f4 was written (10:30 UTC), the branch had moved to 234b108bd94, which bumps the iceberg and glue extensions and adds their autoload entries. The Python build does not need that, so the pin is fine, but the sentence should say "the branch tip at 08:47 UTC".
- The header list omits that
interval.hppalso gainsinterval_t::SameFields(duckdb/duckdb#28331). Nothing in the bindings calls it. - Release notes may want the Python visible corrections the pin brings, all verified on base and head: HUGEINT and wide DECIMAL values convert to the nearest double;
(2.0 ** 81)::VARCHARprinted 4.835703278458517e+24, twice the value, and now prints 2.4178516392292583e+24;'12:00:00 garbage'::TIMEwas 12:00 and now raises a ConversionException;sum(DISTINCT x) EXPORT_STATEreturned a state and now raises a BinderException; casting -2**127 as a DOUBLE to HUGEINT now succeeds; the EXPLAIN text of a FILTER over a registered Arrow table now reads(n > 1) AND (n < 8)instead ofn BETWEEN 1 AND 8. - No fast test pins the rounding of a HUGEINT or wide DECIMAL value in
fetchnumpy(). One midpoint case, such as the value above, would keep the NumPy path correct if the engine rewrites the cast for speed. SHAPESnow covers three of the four inclusivity combinations. Addingn > 1 AND n <= 8, which gives((n > 1) AND (n <= 8))with fragment(("n" > 1) AND ("n" <= 8)), and the folded inclusive pairn >= 1 AND n <= 8, which still gives(n BETWEEN 1 AND 8), would pin every combination. Both outputs are as I printed them on the head.
Outside findings not taken
- Codex (xhigh, briefed on the engine range) reported no finding. It said HUGEINT and UHUGEINT values reach the changed cast through
fetchnumpy(). For UHUGEINT that is not reachable: the type is refused atsrc/numpy/raw_array_wrapper.cpp:70on both builds. It did not measure cost, which is where finding 1 came from. - The code-review skill was not run. The diff is one pointer and five test lines, and its reviewers read the diff, not the submodule, so I gave the outside opinion to Codex with a brief on the 59 engine commits instead.
Prerequisites in the engine
CastBigintToFloating in src/common/types/hugeint.cpp (and CastUhugeintToFloating in uhugeint.cpp) at 76e6bf79426, identical on engine main. A faster correctly rounded conversion is possible: a native 128 bit conversion where the compiler has one, or a fast path for values whose upper word is 0 or -1. Either is an engine change. Recommendation: land the pin with the regression recorded, and file the engine issue, rather than holding the 2.0 line's pin.
What I verified
- Head dfca0f4, merge base 32b9950. The GitHub diff equals
git diff 32b9950..dfca0f4apart from abbreviation lengths. The merge 2ae8c66 has exactly the tree thatgit merge-treewrites for fd7028f and 32b9950 (empty diff). - The pointer is exactly 76e6bf79426c414cc5f0634adbe824dba941c98d. It is on the first parent line of the engine's
v2.0-cyanoptera(the merge of duckdb/duckdb#28733), its parent there is 331f91a84f8 (#28602) on top of the old pin 925bcdf3575, and it is not an ancestor of engine main. - Builds: head (Release,
build/debug, byprepare-tree.sh), merge base (same two stepuv sync, engine at 925bcdf3575), released 2.0.0. A variant of the head with only the old body ofCastBigintToFloatingin the engine, built the same way, to attribute finding 1; it prints the old rounding, 8.507059173023462e+37, and keeps the other new casts. - The four cases print the new text on the head, and the old text on the base and on 2.0.0 (
round-1/probes/probe_texts.*.txt). The head's test file on the base build: 4 failed, 197 passed, the four failing on the old text with equal fragments. The merge base's test file on the head build: 4 failed, 197 passed, the same four. Rows over a registered Arrow table, with a NULL and both bounds, are equal on both builds for all three exclusive combinations. - Full fast suite, eight workers, a 600 second timeout per test: head
18167 passed, 113 skipped, 164 xfailed in 616.06s, which matches the description. Merge base18167 passed, 113 skipped, 164 xfailed in 631.42s, with the same skipped and xfailed tests. Apart fromtest_predicate_translation.py, the merge base's tests are the head's, so the merge base's expectations on the new engine fail in exactly the four cases. That is what Release run 43145686025 shows on engine 331f91a84f8, which lacks only the multi-file statistics commits of #28733. - Behaviour probes on both builds:
round-1/probes/probe_behaviour.{base,head}.txt. - Benchmarks:
round-1/probes/bench_hugeint_numpy.txt,bench_hugeint_df.txt,bench_controls.txt,bench_variant.txt,bench_executemany.txt. - Outside opinions: Codex at xhigh, report in
round-1/codex-report.md; the code-review skill was not run, for the reason above. - CI on dfca0f4: Code-quality checks passed. Submodule sanity passed on its second run once the
submodule-bumplabel was on; the first run was cancelled. "Tests and builds on PR" was still running when I wrote this: the cp311 manylinux aarch64 wheel with its inline tests and the Seed wheels for macOS x86_64, manylinux aarch64 and Windows ARM64 had passed, nothing had failed, and the sdist build, the Full-API build with assertions and the other wheels were running or queued. No red check is this pull request's.
What held
- The bindings compile against 76e6bf79426 with no source change, and no overload added in the four headers captures an existing call.
- The walker's fragments for every
SHAPEScase are unchanged; only the engine's text of three inclusivity combinations and the two date rewrites moved. - No test in the fast suite other than the four depends on an engine text, setting list, function list or error message that the 59 commits change.
- The main line's stability default and lossless cast drop are not on this branch, so nothing of pull request 761's source changes is needed here.
- No bindings source changed, so the bindings' own hot paths are untouched. Of the engine code they call per value or per statement, only the HUGEINT cast moved, and the binder rule of 0c5a115259b, which leaves
executemanyat the same instruction count.
el-marvin
left a comment
There was a problem hiding this comment.
Changes requested.
I reviewed dfca0f4 again (round 2). The head is unchanged since round 1, so the delta is the description alone: the author answered round 1 with a paragraph on the slower HUGEINT to DOUBLE cast, a list of the corrections a Python user sees, the times of both engine tips, and interval_t::SameFields in the header list, and with no new commit. No code, pointer or test changed, so round 1's results stand (the premise, the four expectations, the fast suite, the compile against the new engine), and I did not rerun them. I checked every sentence of the new description against the engine history, against round 1's measurements, and with two new probes on the same head and merge base builds: one for the TIMESTAMP_NS claim, which round 1 had not run, and one benchmark to place the author's second set of figures, which round 1's figures do not explain. I did not ask for outside opinions this round: the delta is text, and the code they would read is the code Codex read in round 1.
Two things keep this from merging, both in the description and both in the record round 1 asked for. The description says the cost is reported to the engine, and no such report exists. And the range it gives for the slowdown, taken from round 1, holds only when every value's magnitude sits on one side of 2**64; values on both sides cost 1.4 to 2.2 times the instructions and 54 to 86 percent more time, and the author's own second figures are that case, not "the same shape". Round 1 caused the second one by calling its values "the whole range"; I correct that below. Each is one sentence to change. With both fixed I have nothing left that blocks merging; whether to land the pin with the cost on record or wait for a faster cast stays the maintainer's call, as in round 1.
Since the last review
- Round 1 finding 1, the slower HUGEINT to DOUBLE cast (should fix): partly addressed. The description now names 5b8ee21b05e as the backport of duckdb/duckdb#28429. The backport's change to
hugeint.cppanduhugeint.cppis byte for byte that of upstream 467d1ca5ee62, merged on 2026-11-06. The description says what the change corrects and what it costs, and carries round 1's figures as round 1 measured them: 1.7 to 1.9 times the instructions is 942 M over 542 M up to 1048 M over 561 M; df() of DECIMAL(38,2) is 99.9 against 139.0 ms; the BIGINT control is flat. It declines a client side conversion for round 1's reason and leaves the wait to the maintainer. "Round once instead of twice" is what the commit does. The base computedREAL_T(input.lower) + REAL_T(input.upper) * (REAL_T(NumericLimits<uint64_t>::Maximum()) + 1), which rounds the low half and then the sum. The head converts one 64 bit value with a sticky bit and scales it by an exact power of two. "On main as well since pull request 761" also holds: main's pin 23221edfd577 contains 467d1ca5ee62, and the pin before it, 71f47d410f6a, does not. Still open: the engine report (finding 1) and the range (finding 2). - A correction to round 1. Round 1's table and its df() figure describe
((i * 2654435761) % 4294967291)::HUGEINT * '19807040628566084398385987584'::HUGEINT + ...as "values over the whole range". Those values are all nonnegative, and all but one of the ten million are above 264. Its other rows keep every magnitude below 253. So round 1's 27 to 49 percent and 1.7 to 1.9 times describe values whose magnitudes all sit on one side of 2**64, not the whole range. Round 1 also missed that #28429's own description already records a cost. There,sum(x::DOUBLE)over 20 million rows went from about 0.07 to 0.12 s for positive values of similar size, and from 0.06 to 0.21 s for values of random sign and length. Both points change what I ask below. - The tip wording: addressed. 925bcdf3575 (09:49 UTC on the 11th) and 76e6bf79426 (08:47 UTC on the 12th) are the commit times, and each was the branch tip then. The new clause about the branch moving on has fallen behind already (smaller things).
interval_t::SameFields: addressed. The list names four headers:add.hpp,interval.hpp,bound_between_expression.hppandsettings.hpp. Those are exactly the changed headers among the#includelines ofsrcandadbc_driver_duckdb, andSameFieldsand the TIMESTAMP_NS overloads are all thatinterval.hppgains.- The list of corrections a Python user sees: addressed. Each of its thirteen pull request numbers is a commit in
925bcdf3575..76e6bf79426. The six that round 1 ran match round 1's probe output. I ran the new one, TIMESTAMP_NS plus INTERVAL, on both builds:typeofgives TIMESTAMP on the merge base and TIMESTAMP_NS on the head, for plus and for minus (round-2/probes/probe_ts_ns.{base,tree}.txt). One gloss among those taken from commit titles is wrong (smaller things). - The HUGEINT midpoint test for fetchnumpy() and the two SHAPES cases: addressed by decision. Round 1 offered them as suggestions. Landing them on main first, where the same NumPy conversion and predicate walker live, is a sound home, and nothing in this head depends on them. The reply also mentions an approval at dfca0f4. This pull request has none, so I do not rely on it.
Findings
1. The description says the cost "is reported to the engine against 5b8ee21b05e and the fix it belongs to", and no such report exists on duckdb/duckdb.
Severity: should fix. Category: docs. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Ask: replace the sentence with what the engine already has on record, or file the report and link it.
I searched through the read only GitHub client between 11:45 and 11:50 UTC on 2026-11-12:
- Issues on duckdb/duckdb opened by el-marvin: none. Its pull requests there are about other things (#28690, #28678, #28677, #28676, #28662, #28661, #28652, #28651).
- Issues and pull requests on duckdb/duckdb that el-marvin commented on, updated since the 12th: none. For evertlammerts, only #28676, the Arrow scan batch.
- 5b8ee21b05e finds only the two backport pull requests that list it (#28602, #28621). 467d1ca5ee62,
CastBigintToFloatingandCastUhugeintToFloatingfind #28429 itself and #12627 from 2024. "fetchnumpy HUGEINT" and "HUGEINT DOUBLE cast slower" find nothing. - #28429 has one comment, the thanks at its merge on 2026-11-06, and no review threads. The fork evertlammerts/duckdb has issues turned off and no pull request about the cast. The newest engine discussion that matches HUGEINT is #26380, about something else.
The reply says the report "is handed to the engine session", so it is planned but not filed. The true sentence is close at hand, because the engine merged this cost knowingly: #28429's description measures it, as quoted in the correction above, and says most of the extra time is in CountZeros<uint64_t>::Leading. A pointer to that paragraph is enough for the record, so I drop round 1's request for a separate engine report. If someone files an issue asking for a faster correctly rounded cast, link it here.
2. The description gives the slowdown as "1.7 to 1.9 times the instructions" and "27 to 49 percent more time" for HUGEINT and wide DECIMAL columns. That holds only when every magnitude sits on one side of 264. Values on both sides cost 54 to 86 percent more time, and the author's second figures, which the description calls "the same shape", are that case.**
Severity: should fix. Category: docs. Evidence: ran. Fix: root-cause. Origin: duckdb-python. Base: introduced.
Ask: give the range by the values it applies to, say which values the 98.6 to 183.1 ms run used, and say that a7e6fc10 is the engine the 2.0 line pinned before pull request 767.
The new cast's cost depends on the value. CastBigintToFloating (external/duckdb/src/common/types/hugeint.cpp:452) branches on the sign, and CastUhugeintToFloating (external/duckdb/src/common/types/uhugeint.cpp:412) branches on whether the magnitude fits 64 bits:
if (input.upper == 0) {
result = REAL_T(input.lower);
return true;
}
// The top 64 bits with every lower bit folded into the last one round to the mantissa as the whole value does.
const auto leading = CountZeros<uint64_t>::Leading(input.upper);Round 1 measured only values that take one side of that branch. I measured ten million values, one thread, best of five fetches, two alternating runs of each build, with the user space instructions of the calling thread (round-2/probes/bench_df_dist.py; results in bench_df_dist.txt and bench_df_dist2.txt):
| values | call | merge base | head | instructions, base and head |
|---|---|---|---|---|
| round 1's "whole range": nonnegative, all but one above 2**64 | df() of DECIMAL(38,2) | 106.7, 106.9 ms | 145.7, 147.1 ms (+37%) | 577 M, 977 M (1.7 times) |
| round 1's small: below 2**53, both signs | df() of DECIMAL(38,2) | 99.9, 105.5 ms | 145.6, 145.9 ms (+38 to +46%) | 573 M, 1059 M (1.85 times) |
| nonnegative, bit length 1 to 127, 5.2 M of 10 M below 2**64 | df() of DECIMAL(38,2) | 102.2, 100.3 ms | 157.2, 162.2 ms (+54 to +62%) | 576 M, 827 M (1.4 times) |
| the same magnitudes with random signs | df() of DECIMAL(38,2) | 109.5, 119.4 ms | 203.1, 207.6 ms (+74 to +85%) | 575 M, 1214 M (2.1 times) |
| the same magnitudes with random signs | fetchnumpy() of HUGEINT | 116.1, 108.0 ms | 208.1, 200.6 ms (+79 to +86%) | 540 M, 1163 M (2.15 times) |
The first row reproduces round 1 (99.9 against 139.0 ms, 576 M against 977 M instructions). The author's figures, 98.6 and 104.7 against 176.6 and 183.1 ms (+75 to +79%), with a BIGINT control as flat as round 1's, fall in the range of the last two rows. So both sets of figures describe the same cast cost under different values, not the same shape under the same values. In the third row the time rises more than the instruction count, which fits a branch that such values take both ways. I did not count branch misses. A sentence such as "1.7 to 1.9 times the instructions and 27 to 49 percent more time when the magnitudes all sit on one side of 2**64, and 54 to 86 percent more time when they fall on both sides" would carry it. The description took round 1's wording over faithfully; the narrow range is round 1's error, not the author's.
Smaller things
- The clause "the branch has since moved on to 234b108bd94 with bumps of the iceberg and glue extensions only" has fallen behind. The tip is now 1479d1b191d, "[v2.0-cyanoptera] Bump spatial (#28644)", committed at 10:24:55 UTC, which is also before dfca0f4 (10:30:15 UTC). After 76e6bf79426 the branch changes only extension configs and patches, two CI scripts and the autoload entries in
extension_entries.hpp. "Has since moved on with extension bumps only", without a hash, stays true. - "JSON arrow chains and dot calls on lambda parameters parse as before the PEG parser (duckdb/duckdb#28240, #28463)" is right for #28240 only. #28463 fixes binding, not parsing. A dot call on a lambda parameter that shares a column's name bound to the column, so
SELECT list_transform([-2], lambda y: y.abs()) FROM (VALUES (5)) t(y)returned[5]instead of[2](its commit message). - "Every one of the 59 is a backport of a merged fix" is not quite true. The 59 are 56 commits and 3 merges. Three of the 56 were written on the 2.0 branch during the backports and have no counterpart on main. 8d90a4c4c61 skips
variant_deep_nesting.testundershredded_vector. a7d7af90260 sets the string prefix of a negated BIGNUM, which main fixed in its limb rewrite. c8c6f650291 drops fallback statistics of a mismatching type. Round 1 let this pass. "Backports of merged fixes, with three adaptations for the 2.0 branch" would be exact. - The TIMESTAMP_NS item has a second effect a Python user sees.
fetchall()ofTIMESTAMP_NS '2024-01-01 00:00:00.123456789' + INTERVAL 1 DAYgives microsecond 123456 on the head, truncated by the conversion todatetime, where the merge base gave 123457, rounded by the cast to TIMESTAMP. The VARCHAR text keeps the nanoseconds.
Outside findings not taken
None. No outside opinion was obtained this round, because the delta is the description alone. Round 1's Codex review covered the same code and pointer.
Prerequisites in the engine
As in round 1: CastBigintToFloating (src/common/types/hugeint.cpp:452) and CastUhugeintToFloating (src/common/types/uhugeint.cpp:412) at 76e6bf79426. Their bodies are identical on engine main at 7ea23aff4d4 (10:22 UTC today). The 2.0 branch has not touched src/common/types since the pin. A faster correctly rounded conversion is an engine change. My recommendation is unchanged: land the pin with the cost on record, because it brings the corrections listed and removes the four failures that keep the 2.0 Release workflow red. Waiting stays defensible, and the choice is the maintainer's.
What I verified
- Head dfca0f4, the same as round 1, with the same three commits (fd7028f, 2ae8c66, dfca0f4). Merge base 32b9950. The pointer at the head is 76e6bf79426c414cc5f0634adbe824dba941c98d.
- Engine history from the bare mirror, which now holds both pins. 925bcdf3575 has commit time 2026-11-11 09:49:33 UTC and was the first parent tip until 331f91a84f8 at 15:09:59. 76e6bf79426 has commit time 2026-11-12 08:47:07 UTC. The range holds 59 commits. 5b8ee21b05e is in it, not in 925bcdf3575, and not on main. Its cast files equal 467d1ca5ee62's, which is on main. Pull request 767 (32b9950) moved the 2.0 line's pin from a7e6fc10 to 925bcdf3575. a7e6fc10 is 12 commits before 925bcdf3575, and none of those touch the cast files.
- The thirteen pull request numbers in the user list, matched to commits in the range by subject. Commit messages read for #28240, #28463, #28455, #28580 and #28279.
- Probes on the round 1 builds,
tree(head) andbase(merge base), both Release builds made the same way, each imported from its own virtual environment (an import runs a cmake build that finds nothing to do):round-2/probes/probe_ts_ns.pyandround-2/probes/bench_df_dist.py, run alternately byrun_df_dist.shandrun_df_dist2.sh. Every connection setsmemory_limitto 8GB, and threads to 8 (probe) or 1 (benchmark). - The engine report search listed in finding 1.
- CI on dfca0f4 at 11:53 UTC. Passed: Code-quality checks, Submodule sanity, the sdist build, Sanity cp311 manylinux aarch64, and the Seed wheels for cp312 macOS arm64 and x86_64, manylinux aarch64, Windows ARM64 and amd64. Still running: the Full-API build with assertions and the Seed wheel for cp312 manylinux x86_64. Queued: Sanity cp311 manylinux x86_64. Nothing has failed. The two x86_64 Linux wheel legs and the assertion build are not in yet.
What held
- Every other sentence of the description. The tip times, pull request 767, and the 59 commits. The four headers with what each gains, and the bindings compiling unchanged. The absence of #28324 and #28576 on the 2.0 branch. The predicate paragraph and its four cases. The cast paragraph's mechanism, its call site at
src/numpy/array_wrapper.cpp:354-358, the corrected value 8.507059173023463e+37, and the reason for no client side conversion. The seven corrections checked on both engines. The closing figures from round 1's runs. - The author's decision not to add a bindings side conversion and to keep the gitlink and test lines as reviewed. That is right, and it is what round 1 asked.
The 2.0 line's counterpart of the pin move and expectation change that #761 made on main, sized to what the 2.0 engine branch carries.
The engine pin moves from 925bcdf3575 (the tip of the engine's v2.0-cyanoptera branch on 2026-11-11 at 09:49 UTC, set by pull request 767) to 76e6bf79426, the branch tip on 2026-11-12 at 08:47 UTC, 59 commits; the branch has since moved on to 234b108bd94 with bumps of the iceberg and glue extensions only, which the Python build does not need. Every one of the 59 is a backport of a merged fix. The headers the bindings include that change are duckdb/planner/expression/bound_between_expression.hpp, which gains two overloads and keeps the ones the predicate translation calls, duckdb/common/operator/add.hpp and duckdb/common/types/interval.hpp, which add TIMESTAMP_NS arithmetic (duckdb/duckdb#28279) beside the interval and TIMESTAMP calls the bindings make, interval.hpp also gaining
interval_t::SameFields(duckdb/duckdb#28331), which nothing in the bindings calls, and duckdb/main/settings.hpp, which adds two HTTP retry settings (duckdb/duckdb#28561). The bindings compile against the tip without a source change. The two engine changes main's pin move had to follow in the bindings, the table function stability default (duckdb/duckdb#28324) and the dropped lossless cast on an equality (duckdb/duckdb#28576), are not on the 2.0 branch, so nothing of them is needed here.One expectation changes. Four cases of
test_offered_predicatesin tests/fast/arrow/scan/test_predicate_translation.py pin the text the engine offers the Python Arrow scan for a folded range whose bound is exclusive,n > 1 AND n < 8and the date equalities the engine rewrites into a half open range. The engine printed every folded range as an inclusiveBETWEEN,(n BETWEEN 1 AND 8), which the range never was, and the cases pinned that text. Since duckdb/duckdb#28448 (commit f2090084542, in the pin move) the engine prints an exclusive bound as two comparisons,((n > 1) AND (n < 8))and((ts >= '2024-01-01 00:00:00'::TIMESTAMP) AND (ts < '2024-01-02 00:00:00'::TIMESTAMP)). What the Python source receives did not change; only the engine's text of the predicate did, and the four cases now expect it.One engine function the bindings call per value gets slower. Commit 5b8ee21b05e is the 2.0 backport of duckdb/duckdb#28429 (upstream 467d1ca5ee62, 2026-11-06), which makes the HUGEINT to DOUBLE cast round once instead of twice: the NumPy conversion in src/numpy/array_wrapper.cpp calls that cast once per value, so
fetchnumpy()anddf()of a HUGEINT column, and of a DECIMAL column of width 19 to 38 (what every SUM over a DECIMAL returns), now give the nearest double ('85070591730234625310576617597232480257'::HUGEINTcame out as 8.507059173023462e+37 and is now 8.507059173023463e+37) and pay for it. The cast now negates through uhugeint_t and calls Uhugeint::TryCast out of line, 1.7 to 1.9 times the instructions, and the review measured 27 to 49 percent more time: ten million values, one thread, best of five, df() of DECIMAL(38,2) over the whole range 99.9 ms before against 139.0 ms after; a BIGINT column through the same path is unchanged. Our own run of the same shape, two alternating rounds against the 2.0 build at a7e6fc10, is 98.6 and 104.7 ms before against 176.6 and 183.1 ms after, with BIGINT at 71.0 and 78.7 against 70.3 and 77.1 ms. The cost sits in the engine's cast, on main as well since pull request 761 moved its pin past 467d1ca5ee62; a client side conversion would be a second mechanism beside the engine's and could disagree with it, so the bindings do not get one. The cost is reported to the engine against 5b8ee21b05e and the fix it belongs to.What a duckdb-python user sees of the 59 commits, each checked on both engines: HUGEINT and wide DECIMAL values convert to the nearest DOUBLE (duckdb/duckdb#28429);
(2.0 ** 81)::VARCHARprinted 4.835703278458517e+24, twice the value, and prints 2.4178516392292583e+24 (duckdb/duckdb#28410);'12:00:00 garbage'::TIMEwas 12:00:00 and raises ConversionException (duckdb/duckdb#28451);sum(DISTINCT x) EXPORT_STATEreturned the non-distinct state and raises BinderException (duckdb/duckdb#28455); the DOUBLE value -2**127 casts to HUGEINT instead of raising (duckdb/duckdb#28449); TIMESTAMP_NS plus INTERVAL stays TIMESTAMP_NS instead of becoming TIMESTAMP (duckdb/duckdb#28279); the EXPLAIN text of a filter over a registered Arrow table reads(n > 1) AND (n < 8)instead ofn BETWEEN 1 AND 8(duckdb/duckdb#28448). From the commit titles, not run here: JSON arrow chains and dot calls on lambda parameters parse as before the PEG parser (duckdb/duckdb#28240, #28463), earlier CTEs are visible to a PIVOT in a later CTE body (#28390), lead and lag read the offset and default a 1.x database stored (#28580), a Bloom filter under a selection no longer overruns its buffer (#28505), andquery_progress()no longer counts the result collector pipeline (#28413).Against the old pin the four predicate cases pass and the Release workflow, which builds the sdist against the engine branch tip, has failed them since run 43145686025 (4 failed, 18152 passed on engine 331f91a84f8). With the pin at 76e6bf79426 the four cases pass and the fast suite on the box is at 18167 passed, 113 skipped, 164 xfailed (8 workers).