November 16, 2026 - v3.6.1
Download the Spark Connector JAR.
[v3.6.1-v6.6.3]⚓︎
v3.6.1 is the first published v3.6 release. Changes for v3.6.0 are included in this release.
Added⚓︎
- Added six optional scan limits, all unset by default:
scan_timerange_required,scan_max_timerange_sec,scan_max_hdx_partitions,scan_max_hdx_rows,scan_max_hdx_columns_to_read, andscan_hdx_catalog_timeout_ms. A scan that would exceed a limit fails on the driver, before any executor work starts. The limits in force are shown in the scan description, and each scan shows the number of Hydrolix partitions it planned as the HDX partitions planned metric in the Spark UI. - Added the optional configuration setting
query_socket_timeout_ms, which sets the read timeout on the connection to the embedded query engine. - Added the optional configuration setting
engine_startup_timeout_ms, which sets how long a Spark task waits for the embedded query engine to become ready. The default is 60 seconds. - The settings
presigner_page_size,columnar_batch_size,hdx_partitions_per_task, and the six scan limits can now also be set for a single read withspark.read.option(...). A value set for a single read overrides the catalog value for that read only, including when it loosens or removes a scan limit. Other settings configure the connection to the cluster and are rejected when passed for a single read, as are unknown settings.
Changed⚓︎
- Embedded HDX version updated to v6.6.3.
- Hydrolix API responses are now parsed ignoring fields the connector doesn't use, so Hydrolix versions that add response fields stay compatible.
Removed⚓︎
- Removed the
multi_orgsetting. The connector now determines this from the cluster, ignores the setting, and logs a warning. Remove it from catalog configurations. - Removed the non-presigned partition access path. The connector always reads partitions through presigned URLs and never uses storage bucket credentials. The settings
cloud_cred_1,cloud_cred_2,presigned_data_access,storage_bucket_name,storage_bucket_path,storage_region,storage_cloud,storage_endpoint_uri, andpartition_prefixare now ignored, with a warning logged for each. Remove them from catalog configurations.
Fixed⚓︎
- Fixed reads being cut off when the embedded query engine sent no data for 15 minutes, for example on long-running aggregations. The default read timeout is now 6 hours, and can be changed with
query_socket_timeout_ms. - Fixed executor startup failures reporting only a generic error about port 8123. The connector now reports a specific error when the platform isn't Linux x86-64, when the temporary directory is mounted
noexecor has less than 1 GiB free, and when the embedded query engine exits during startup. Tasks now wait for the engine to report that it's ready instead of sleeping a fixed 5 seconds, which also avoids "config not loaded" errors on the first query. The engine's temporary directory is deleted when the engine stops. - Fixed summary queries (
hdxAgg) failing with "Unsupported ClickHouse type" onDateTime('tz'),DateTime64(p),DateTime64(p, 'tz'), andDate32columns. These now return Spark timestamps, withDateandDate32values at midnight UTC.hdxAggoverArray(...)columns now returns a Spark array.
Security⚓︎
- The generated engine configuration file, which contains the cluster credentials, is now created readable by its owner only.