Skip to content

v6.5

Release history⚓︎

Use the latest version of this release, v6.5.10.

Version Released Notes
v6.5.10 November 6, 2026 First publicly released v6.5 version. Unmarked entries apply to it.

Entries added after the initial release are marked with a patch release number.

Notable new features⚓︎

Config cache⚓︎

Each cluster supports a Config Cache for the published Config API data. An object storage aware HTTP proxy serves all readers in the cluster, which no longer fetch directly themselves.

Turning on the config_cache tunable lowers how often the cluster polls object storage for configuration blobs. Instead of every query pod and ingest indexer sending requests, only config-cache polls object storage. It caches responses for manifest_ttl, 2 seco nds by default, before it checks for changes. The feature helps most on large clusters and is off by default.

Ingestion indexers and query services require configuration blobs published by the Config API into the primary storage under the /config/ path. Enabling the config_cache feature restarts these services to retrieve their manifests and configuration blobs through the configuration caching services.

Upgrade instructions⚓︎

Upgrade steps for this release live on their own page: Upgrade to v6.5.

Changelog⚓︎

Ingest⚓︎

Users can now set column-specific Bloom filter settings in index_options, allowing control over the tradeoffs between the false positive rate and size of the resulting Bloom filter indexes. See Partition Bloom filters.

Changes⚓︎

  • Partition Bloom filters can now be tuned per column through index_options. A column accepts partition_bloom with enabled, max_bytes, and fpr_pct, and each unset value inherits the cluster setting. These apply only to string, uuid, and ip columns; setting them on any other type is rejected.

  • Added support for DNS fallback logic to ingestion services for use when the intake router is unavailable.

  • Improved the autoingest queue source service when intake indexing is backlogged or unavailable. Earlier, the queue source continuously accepted new work and acknowledged external services like SQS until reaching local ephemeral storage limit. Now, the service regulates its acceptance of new work on the health and throughput of intake indexing.

Fixes⚓︎

  • Fixed Google Cloud Storage credentials being applied for the whole process rather than per bucket, which broke deployments using more than one GCS bucket.

  • Corrected a case where a failed transaction poisons a connection in a PostgreSQL pool for subsequent use. Earlier, this connection reuse didn't resubmit a new transaction, provoking CatalogError: current transaction is aborted on a fresh query.

  • Fixed a bug where autoingest's queue-peer service lost a file when its first S3 request failed with read: connection reset by peer. Queue peer now retries, then requeues the message for redelivery if storage still doesn't respond. See the new metric queue_source_deferred_count.

  • Eliminated a stuck storage client scenario and improved error handling for failed GET S3 object storage range requests. Earlier, the client retried GET failures forever, without bound or backoff. Now, the client tries reaching object storage up to five times, with backoff logic and an empty buffer.

  • Improved resilience of newline-delimited JSON when encountering unparseable or invalid JSON. The scanner resets on syntax error dropping the current row and skipping to the next newline. Before the fix, all rows after a syntax error were lost.

Query⚓︎

Two different tunables now provide better control over the ephemeral storage cache for each query peer. An existing tunable, data_cache_max_partition_age_seconds, sets a recency window for caching data files, which tend to be larger. A new tunable, metadata_cache_max_partition_age_seconds, sets a separate, longer recency window for partition metadata like manifests and indexes, which tend to be smaller.

Changes⚓︎

  • Added the metadata_cache_max_partition_age_seconds tunable, which caps the age of partitions whose column dictionaries, manifests, and indexes are cached on local disk. It defaults to 604800, seven days. It's separate from data_cache_max_partition_age_seconds, which defaults to one day, because these files are small and are re-read every time a partition opens.

  • Time filters that wrap a DateTime primary column in toDateTime, toUnixTimestamp, toUInt32, toUInt64, toTimeZone, or CAST now prune partitions and satisfy hdx_query_timerange_required, the same as a filter on the bare column. Truncating functions such as toDate and toStartOfHour, and DateTime64 primary columns, still aren't recognized as time filters.

  • Queries authenticated with a service account token now record the caller as <service-account-id>/<token-id> in the user column of hdx.active_queries and in query log messages. Queries using tokens that carry a username are unchanged.

  • The query system supports Kubernetes liveness probes, which no longer appear in the query log. Two new tunables set the probe queries, query_peer_liveness_sql and query_head_liveness_sql, and an empty string disables that probe. The tunable query_peer_liveness_check_path is deprecated, and only its none value still has an effect. Other queries on hdx.liveliness are now logged.

  • New thread pool exhaustion detection logic refuses queries unless there are enough available threads. Before accepting a query, the server now ensures that query_thread_pool_min_free_threads are available in the pool sized at query_thread_pool_max_threads. Settings are independent and default to 0, matching existing behavior.

Cluster operations⚓︎

Changes⚓︎

  • The Hydrolix cluster manifest accepts extra_containers and extra_init_containers, which add sidecars and init containers to Hydrolix workloads. Both are validated against the Kubernetes container schema when the configuration loads, so a wrong type or an unknown field fails immediately instead of being rejected later by Kubernetes. Extra init containers run before a workload's own init containers, and extra containers are ordered by name alongside existing sidecars. Jobs don't take extra_containers, and daemonsets don't take extra_init_containers.

  • Added a Prometheus scrape for Grafana's own operational metrics, and a startup probe that allows up to ten minutes for a slow start before the liveness probe can restart the pod. The scrape path follows Grafana's root_url when serve_from_sub_path is enabled.

  • Upgraded Traefik from v3.7.7 to v3.7.13 to address CVE-2026-88009.

  • New Helm charts install the Hydrolix operator: hdx-crd for the HydrolixCluster custom resource definition and priority classes, and hdx-operator for the operator. Installing with kubectl apply still works. For details, contact support@hydrolix.io.

  • Extended authentication in the Traefik reverse proxy to cache rejected credentials as well as successful logins. This lowers the load on the Config API from repeated failed logins. New metrics count rejections and failures, and log lines include the username or relevant JWT details.

Fixes⚓︎

  • Traefik no longer warns that trustForwardHeader and maxResponseBodySize aren't set on the unified-auth middleware, which now sets them to false and 4 KB. The auth service logs expected 401 and 403 rejections at INFO instead of ERROR.

  • Fixed query discovery caching a node's capacity as zero cores when the node was briefly unreadable. Discovery runs on pod events, so the stale value persisted until the next pod change. Node capacity is now re-read every five minutes.

  • Added a liveness check to all PostgreSQL pooled connections, to validate and reconnect broken connections before handing them out. Earlier, an unused connection in the pool could be dead and the application query would fail.

  • Fixed irreconcilable, wedged Traefik definition when AWS Load Balancer Controller injects a loadBalancerClass into the live Kubernetes spec. Now, the operator selects and prefers the live, immutable loadBalancerClass, if present. Earlier, Kubernetes refused to accept the operator's update which lacked the immutable setting.

  • Increased the static scale assignment for check-bucket-access memory to 512MiB to avoid crashes. An earlier switch to the Python Kubernetes client and ConfigMaps uses more memory.

  • Fixed the hdx-node Role and RoleBinding landing in the wrong namespace when hdx_node is enabled. The generated operator resources now name the operator's namespace instead of relying on the current kubectl context.

  • Updated dependency kopf 1.38.0 → 1.45.2. The upgrade fixes an interaction between the operator, watchers, and the Kubernetes API which could strand the cluster in a stale state.

  • Skipped unnecessary PostgreSQL password rotation by testing a connection with credential, which could interfere with pgbouncer pass through logins. Also, placed the PostgreSQL monitor user, usually pgmonitor, under management.

  • Fixed RabbitMQ crash at startup after NXDOMAIN for a DNS lookup on its own name. The rabbit service now sets publishNotReadyAddresses to avoid the crash.

  • Corrected an operator race condition that occured when upgrade and reconciliation loops modified resources simultaneously. The operator now guarantees that newly created resources receive a current hdx_update key. Earlier convergence tests failed until manual restart if any resource received an outdated hdx_update key.

Configuration and APIs⚓︎

This release includes correctness and bug fixes for existing models and endpoints.

Fixes⚓︎

  • Fixed a PATCH to a Kafka or Kinesis table source failing when the request left out checkpointer, settings, stream_name, or transform. Partial updates now apply.

  • Fixed the row policy and column policy endpoints returning a server error for a table ID that isn't a UUID. They now return 404.

  • Added the customer assignments to service account API responses. This matches the pattern used for user account responses.

  • Corrected lingering, stale assignment when an account's default customer is deleted. The reference to the deleted customer is now removed.

  • Fixed undeletable invites by removing association with deleted customer objects. On upgrade, a migration tool unlinks orphaned invites from deleted customers. The invites must still be deleted by an admin before the corresponding email addresses can be invited again.

  • Corrected v2 HTTP response status codes to distinguish visibility and permissions. The API now returns HTTP 404 not found to requests for objects to which a user has no visibility rights. For permissions-restricted objects, the API now returns HTTP 403, similar to the v1 endpoints.

  • Corrected multiple permissions and HTTP response code bugs in the object nesting logic for v2 endpoints. Earlier, an account with permissions to a table was incorrectly forbidden from modifying objects like transforms or columns. Similarly, a request listing projects with sufficient project-scoped view_project permission shouldn't have received an HTTP 403. Now, the API correctly returns the list of accessible projects.

  • Tightened the permission check for making a storage the default. Users need the global-scope change_hdxstorage permission or super_admin to set any storage as the default. Permission on the storage itself is no longer enough.

  • Tightened table creation and project modification checks to require a summary table to belong to the same customer as the raw, parent table.

  • The Config API no longer writes full request details to its error log lines, which now include only the request's path, method, and user ID.

Interfaces and tools⚓︎

Changes⚓︎

  • Raised the in-cluster MCP server's default query limits: the timeout from 30 to 120 seconds, memory per query from 2 GiB to 4 GiB, and the raw-table time range from 6 to 24 hours. The old limits stopped ordinary aggregations partway through.

Fixes⚓︎

  • Fixed HTTP redirect generation in the uvicorn web server that handles incoming requests. It now trusts X-Forwarded-* from the forwarded_allow_ips of the mcp_hydrolix tunable. Without this fix, clients requesting /mcp/ received a redirect to a plaintext HTTP URL.

Bundled component versions⚓︎

Third-party components whose bundled or default version changed in this release. Where a version change also changes behavior, the entry in the relevant area covers the detail.

Component From Version in Hydrolix v6.5
Kibana Gateway v2.2.0 v2.3.0
Traefik v3.7.7 v3.7.13