Skip to content

November 16, 2026 - v3.6.1

Download the Spark Connector JAR.

[v3.6.1-v6.6.3]⚓︎

v3.6.1 is the first published v3.6 release. Changes for v3.6.0 are included in this release.

Added⚓︎

  • Added six optional scan limits, all unset by default: scan_timerange_required, scan_max_timerange_sec, scan_max_hdx_partitions, scan_max_hdx_rows, scan_max_hdx_columns_to_read, and scan_hdx_catalog_timeout_ms. A scan that would exceed a limit fails on the driver, before any executor work starts. The limits in force are shown in the scan description, and each scan shows the number of Hydrolix partitions it planned as the HDX partitions planned metric in the Spark UI.
  • Added the optional configuration setting query_socket_timeout_ms, which sets the read timeout on the connection to the embedded query engine.
  • Added the optional configuration setting engine_startup_timeout_ms, which sets how long a Spark task waits for the embedded query engine to become ready. The default is 60 seconds.
  • The settings presigner_page_size, columnar_batch_size, hdx_partitions_per_task, and the six scan limits can now also be set for a single read with spark.read.option(...). A value set for a single read overrides the catalog value for that read only, including when it loosens or removes a scan limit. Other settings configure the connection to the cluster and are rejected when passed for a single read, as are unknown settings.

Changed⚓︎

  • Embedded HDX version updated to v6.6.3.
  • Hydrolix API responses are now parsed ignoring fields the connector doesn't use, so Hydrolix versions that add response fields stay compatible.

Removed⚓︎

  • Removed the multi_org setting. The connector now determines this from the cluster, ignores the setting, and logs a warning. Remove it from catalog configurations.
  • Removed the non-presigned partition access path. The connector always reads partitions through presigned URLs and never uses storage bucket credentials. The settings cloud_cred_1, cloud_cred_2, presigned_data_access, storage_bucket_name, storage_bucket_path, storage_region, storage_cloud, storage_endpoint_uri, and partition_prefix are now ignored, with a warning logged for each. Remove them from catalog configurations.

Fixed⚓︎

  • Fixed reads being cut off when the embedded query engine sent no data for 15 minutes, for example on long-running aggregations. The default read timeout is now 6 hours, and can be changed with query_socket_timeout_ms.
  • Fixed executor startup failures reporting only a generic error about port 8123. The connector now reports a specific error when the platform isn't Linux x86-64, when the temporary directory is mounted noexec or has less than 1 GiB free, and when the embedded query engine exits during startup. Tasks now wait for the engine to report that it's ready instead of sleeping a fixed 5 seconds, which also avoids "config not loaded" errors on the first query. The engine's temporary directory is deleted when the engine stops.
  • Fixed summary queries (hdxAgg) failing with "Unsupported ClickHouse type" on DateTime('tz'), DateTime64(p), DateTime64(p, 'tz'), and Date32 columns. These now return Spark timestamps, with Date and Date32 values at midnight UTC. hdxAgg over Array(...) columns now returns a Spark array.

Security⚓︎

  • The generated engine configuration file, which contains the cluster credentials, is now created readable by its owner only.