[ChristopherWirt]

Fixes #116.

Problem

1. Spark V1 Filters cannot express list predicates such as array_contains(tags, 'x') or exists(items, x -> x.price > 100).
2. So today the V2 scan reads every row and Spark evaluates these predicates afterwards.

Change

1. Translate. VortexScan turns supported Catalyst ArrayContains / ArrayExists data filters into VortexListFilters, which compile to native listContains / listTransform. Supported inside the predicate: comparisons, AND/OR/NOT, null checks, and struct fields of the element.
2. Ship. VortexListFilter holds the Catalyst predicate, not native handles. It travels to executors through a small Java NativePredicate interface, because the Java sources compile before the Scala ones.
3. Validate per file. Each predicate reports the column paths it reads. The reader checks those paths against the file's physical schema, including timestamp units, and drops the predicate for that file if anything doesn't match.
4. Stay safe. Spark still evaluates every predicate as a residual filter. Floats, non-binary collations, correlated or nested lambdas, and other operators are left to Spark.

Not in scope

• V1 (VortexFileFormat) pushdown and list-element zone pruning.
• For array<struct>, any unsupported or missing sibling field in the element struct (a map, or a ms timestamp, for example) disables pushdown for that file. This is conservative, not wrong. Recursing ReadTypes.filterable into arrays would fix it.
• No performance numbers yet.

Testing

• Compiles for Spark 3.5 / Scala 2.12, Spark 3.5 / Scala 2.13, and Spark 4.0 / Scala 2.13. spotlessCheck is clean.
• Passing: all non-JNI unit tests, including the new ReadTypesTest and VortexListFilterTest cases, and the existing V1 VortexFilterPushdownTest cases.
• Not yet run: the new end-to-end cases need a JNI library with NativeExpression.variable. On current develop the Rust code compiles, but a local build hasn't completed yet. Draft until those cases pass.

🤖 Generated with Claude Code