spark-connect

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Apache Spark Connect Client for Rust

Apache Spark Connect Client for Rust (Rust core + Python wrapper)

Sparglim✨ makes PySpark App Configurable and Deploy Spark Connect Server Easier!

Interactive Apache Iceberg debugger for exploring production tables and tracing metadata evolution through AI and real time graph visualization.

velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docker Compose.

Helm charts for Spark Connect, Spark History Server, Trino, Karpenter and Kyverno

Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect

Apache Spark Connect Client for C++.

A reverse proxy server which allows secure connectivity to a Spark Connect server

Apache Spark Connect Gateway

Study system for the Databricks Certified Associate Developer for Apache Spark 3.5 exam. 5 interconnected Claude Code skills covering all 7 exam sections, with sources linked to Apache Spark 3.5 docs and Damji's Learning Spark 2nd Edition.

Docker Compose environment for big data research and machine learning development

Collection of articles, using the Literate Programming style, about Data Engineering and Software Tooling in general

Modern TypeScript connector for Apache Spark — type-safe and ergonomic.

A native Apache Spark Connect client for Scala 3: SparkSession/DataFrame/Column/functions API over gRPC.

The Internals of Spark Connect

Latu ML: Spark MLlib from Elixir, over Spark Connect (a companion package to Latu).

A native Elixir DataFrame API for Apache Spark, over Spark Connect.

Kotlin adapter for Apache Spark™ Connect JVM with dual-backend serialization and Unity Catalog integration. Bachelor's thesis implementation.