[ChristopherWirt]

Problem

Zone pruning of col IN (...) stops working once the list has more than 16 values:

1. Up to 16 values, each value gets its own max < v OR min > v test, so zones that fall between values are pruned.
2. Beyond 16, the falsifier only checks the list's overall [min, max]. A zone in a gap between two keys can hold no member of the set, yet it is still scanned.
3. The falsifier also only recognised a literal list. After optimisation the list becomes a PreparedSet, so an optimised IN produced no falsifier. Spark worked around this by spelling out explicit min <= col <= max bounds.

Change

• Add SetDisjoint, an internal stats expression. It holds the set's sorted, de-duplicated non-null members and binary-searches each zone's [min, max] for a member: O(log n) per zone, and the expression stays compact.
• list_contains_falsify now reads the list from either a Literal or a PreparedSet. For more than 16 values it emits SetDisjoint for bool, primitive, decimal, utf8 and binary lists. Other dtypes keep the previous overall-bounds fallback.
• The existing non-NaN guards are kept, and missing or null bounds never prune a zone.
• The Spark explicit IN bounds (inBounds/orderKey) are removed. The Rust rewrite now produces the same bounds or tighter ones.

Tests

New zone_map tests cover:

• the 16/17-value boundary
• 80 integer keys and 80 string keys
• duplicates and nulls in the list
• missing stats
• NaN and signed zero, with and without NaN-count stats
• literal and optimised (PreparedSet) expressions

A stress test with 16,384 members over 8,192 zones checks that pruning matches actual row membership.

Validation

• cargo nextest run -p vortex-expr -p vortex-layout -E 'test(stats::rewrite) | test(layouts::zoned::zone_map)': 61 passed
• cargo clippy -p vortex-expr -p vortex-layout --all-targets --all-features -- -D warnings: clean
• ./gradlew :vortex-spark-3.5_2.12:test --tests SparkFilterToVortexExpressionTest: passed

Follow-ups (not in this PR)

• Date/Timestamp (extension) lists still use overall bounds only.
• SetDisjoint compares Scalars per zone. A typed fast path, and sharing the sorted set with PreparedSet's haystack, could be added if benchmarks warrant it.

Fixes #101

🤖 Generated with Claude Code