Skip to content

July 29, 2026 - v3.4.0

Download the Spark Connector JAR.

[v3.4.0-v6.2.4]⚓︎

Added⚓︎

  • Introduced a new optional configuration setting:
    • service_account_token, a JWT used to authenticate to the Hydrolix cluster, as an alternative to username/password.
  • Added additional Spark UI metrics: PartitionReader fetch time (min, max, median, p90), SparkScanBuilder.build time, partition reader start/last timestamps, and Turbine query time statistics
  • Added optional configuration setting hdx_partitions_per_task, which specifies the number of Hydrolix partitions read by each Spark partition
  • Added pushdown support (performance optimization) for ORDER BY + LIMIT (top-N) queries
  • Added pushdown support (performance optimization) for the OFFSET clause
  • Added shading rules for Apache httpcore and httpclient libraries to avoid classpath conflicts on Microsoft Fabric
  • Added logging of the embedded Turbine version at startup
  • Added an experimental columnar (transposition-based) partition reader for use with the force_columnar query mode; the transposition batch size may be set with the columnar_batch_size config
  • Added the optional config keys cluster_url, jdbc_protocol, and jdbc_port as an alternative to api_url and jdbc_url
  • Added support for page-number-paginated Hydrolix API endpoints, recursively fetching all pages when more than one page of data is available

Changed⚓︎

  • Bundled Turbine server updated to v6.2.4
  • Changed the embedded Turbine's listen port from 8088 to 8123
  • Reworked summary table interaction:
    • All summary aliases are now accessed with the syntax SELECT hdxAgg('my_summary_alias') FROM hydrolix.my_project.my_summary_table
    • No user interaction is required to pre-register summary aliases before querying them
    • Using summary tables requires the SQL extension spark.sql.extensions=io.hydrolix.connectors.spark.SummaryUdfExtension

Fixed⚓︎

  • Added workaround for reads failing for tables whose partition catalog spans more than one page (>1000 partitions by default)
  • Fixed presigned reads failing with a config not loaded error when the bundled Turbine server is v6.2.0 or later
  • Fixed the API access-token cache not renewing, which could cause long-running sessions to fail with authentication errors until restarted
  • Fixed TURBINE_VERSION not accepting released Turbine versions when building from source, and the JAR's internal version metadata reporting a stale v1.0.0