Use Apache Spark
Overview⚓︎
The Hydrolix Connector for Apache Spark combines the cost and query efficiency of the Hydrolix platform with the rich data analysis and distributed computing power offered by Apache Spark. By integrating your Apache Spark ecosystem with Hydrolix, the Hydrolix Connector for Apache Spark enables the cost and performance efficiency gains of Hydrolix as the backing store while presenting the data in familiar notebooks and coding environments.
The latest Hydrolix Connector for Apache Spark JAR can be downloaded here. This link always points to the most recent release listed in the compatibility matrix. Pin a specific version from the matrix for reproducible deployments.
Versioning⚓︎
The Hydrolix Connector for Apache Spark version is of the format:
An example of this formatting is the following:
Compatibility matrix⚓︎
Each Hydrolix Connector for Apache Spark version supports a specific range of Hydrolix cluster versions. The following is a compatibility matrix between Hydrolix Connector for Apache Spark versions and their compatible Hydrolix cluster versions:
| Hydrolix Connector for Apache Spark Version | Hydrolix Versions | Changelog |
|---|---|---|
| v3.4.0-v6.2.4 | v6.1.5+ | July 29, 2026 - v3.4.0-v6.2.4 |
| v3.0.0-v5.6.0 | v5.6.0 to v6.1.8 | September 2, 2025 - v3.0.0-v5.6.0 |
| v2.0.0-v5.1.1 | v4.22.1+ | April 22, 2025 - v2.0.0-v5.1.1 |
| v1.0.0-v4.22.1 | v4.22.1 to v5.3.1 | February 4, 2025 - v1.0.0-v4.22.1 |
The connector JAR is built for Scala 2.12 and Spark 3.x. Scala 2.13 and Spark 4 runtimes aren't supported.
Deployment environments⚓︎
The Hydrolix Connector for Apache Spark can be deployed to multiple platforms. Follow the install instructions for your preferred platform.