Skip to content

v6.1.5

Notable new features⚓︎

Per-route Traefik IP allowlist⚓︎

Hydrolix now supports per-route IP allowlists in the Traefik proxy. Set the traefik_ip_allowlist tunable to a list of CIDR ranges to restrict cluster access, and use traefik_ip_allowlist_exempt_routes to exempt specific entrypoints or path prefixes. ACME challenge routes are always exempt to allow Let's Encrypt certificate issuance. The load balancer must preserve client IPs for the allowlist to take effect. See the Traefik IP allowlisting guide for configuration details.

Federated authentication for the Config API⚓︎

The Config API now accepts Bearer tokens signed by external JWT issuers, enabling integration with customer-managed identity providers. Configure trusted issuers in the new turbine_api_trusted_token_issuers operator tunable. Existing Keycloak and service account authentication paths are unchanged.

JSON datatype enabled by default⚓︎

Hydrolix now enables JSON datatype support in transforms by default. The json_datatype_enabled tunable defaults to true in v6.1, where earlier versions defaulted to false. JSON columns store schema-flexible objects in a single column, including arrays of scalar values and arrays of JSON objects, and Hydrolix indexes their fields at ingest.

Introduction of parallel hierarchy for Orgs⚓︎

In the future, Hydrolix will support multiple Orgs to offer better support for multi-tenant clusters.

The Config API introduces the Customer in v6.1 as the first step in the transition. See Organizations for a description of the benefits, to be realized in a later release. Read Org transition planning to understand the changes over this and upcoming releases.

  • Single-tenant cluster administrators don't need to do anything.
  • For multi-tenant clusters, set tunable turbine_api_feature_migrate_projects_to_orgs=true.

A continuously running background job creates Customer objects according to the setting. In a future release, the parallel hierarchy becomes Orgs, replacing the sole existing Org.

Upgrade instructions⚓︎

Don't skip minor versions when upgrading or downgrading

Skipping versions when upgrading or downgrading Hydrolix can result in database schema inconsistencies and cluster instability. Always upgrade or downgrade sequentially through each minor version.

Example:
Upgrade from 5.10.10 → 5.11.9 → 6.0.8, not 5.10.10 → 6.0.8.

Step 1: Set tunable⚓︎

Set the tunable turbine_api_feature_migrate_projects_to_orgs to true if you want each of the projects in the cluster to be elevated to its own organization, or skip this step and accept the default.

1
2
spec:
  turbine_api_feature_migrate_projects_to_orgs: false

Default behavior: For a single-tenant cluster, after upgrading to v6.1, three Customer objects are created. One Customer object each is created for the projects hydro and sample_project. All other projects are associated with a third Customer object.

This is an example of explicit declaration of the default setting.

1
2
spec:
  turbine_api_feature_migrate_projects_to_orgs: true

Set this for multi-tenant clusters. This tunable must be explicitly set in the Hydrolix cluster spec before upgrading to v6.1.

A distinct Customer object is created for every existing project.

The tunable setting controls a background job that creates the parallel hierarchy and runs for the lifetime of Hydrolix v6.1.

Step 2: Apply the new Hydrolix operator⚓︎

If you have a self-managed installation, apply the new operator directly with the kubectl command examples below. If you're using Hydrolix-supplied tools to manage your installation, follow the procedure prescribed by those tools.

GKE⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v6.1.5/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

EKS⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v6.1.5/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

LKE and AKS⚓︎

1
kubectl apply -f "https://www.hydrolix.io/operator/v6.1.5/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}"

Step 3: Monitor the upgrade process⚓︎

Kubernetes jobs named init-cluster and init-turbine-api will automatically run to upgrade your entire installation to match the new operator's version number. This will take a few minutes, during which time you can observe your pods' restarts with your Kubernetes monitor tool.

Ensure both the init-cluster and init-turbine-api jobs have completed successfully and that the turbine-api pod has restarted without errors. After that, view the UI and use the API of your new installation as a final check.

If the turbine-api pod doesn't restart successfully, or other functionality is missing, check the logs of the init-cluster and init-turbine-api jobs for details about failures. This can be done using the k9s utility or with the kubectl command:

1
2
% kubectl logs -l app=init-cluster
% kubectl logs -l app=init-turbine-api

If you still need help, contact Hydrolix support.

Downgrade instructions⚓︎

Use these instructions when downgrading from v6.1 to v6.0.

  1. Connect to the Kubernetes cluster using k9s and select the namespace holding the Hydrolix cluster.
  2. Select the turbine-api pod and turbine-api container.
  3. Invoke a shell by entering s.

  4. Execute each of the following commands, which will produce STDERR to terminal. Each must exit cleanly.

    1
    ./manage.py release_6_0

  5. Exit the k9s utility by pressing Ctrl-C.

  6. Follow the upgrade instructions for the latest Hydrolix v6.0 release.

Changelog⚓︎

Updates⚓︎

Intake updates⚓︎

  • Upgraded the operator-managed hydrolix_otel Collector image from hdx-collector:1.0.1 to hdx-collector:v1.2.1. Clusters that set hydrolix_otel.enabled: true pick up the new image and batch settings on upgrade.

  • Updated the Go toolchain to 1.26.3 across the intake codebase to address end-of-life concerns.

  • Upgraded jackc/pgx/v5 from 5.5.5 to 5.9.2 in intake. Resolves critical CVE-2026-41889 (SQL injection in the simple protocol with dollar-quoted strings).

  • Upgraded amazon-kinesis-client from 2.7.1 to 2.7.2 to address an HTTP request smuggling vulnerability in dependent library Netty. Resolves critical CVE-2026-42581.

Cluster operations updates⚓︎

  • Updated software packages across control-plane services per Aikido security scan findings:

    • gunicorn 23.0.0 → 26.0.0, addresses AIKIDO-2026-10742
    • openssl 0.10.78 → 0.10.80
    • urllib3 2.6.3 → 2.7.0
    • rustls-webpki 0.102.8 → 0.103.13
    • idna 3.10 → 3.15
    • yarl 1.18.3 → 1.24.2
    • http 1.4.0 → 1.4.1

  • Updated Go dependencies in the hdx-scaler-go and acme services to patch a gRPC authorization vulnerability. Satisfies CVE-2026-33186.

    • google.golang.org/grpc v1.78.0 → v1.79.3
    • goccy/go-yaml v1.9.8 → v1.15.2
    • go.opentelemetry.io/otel v1.38.0 → v1.39.0

  • Updated Go dependencies in the acme service to patch a path traversal vulnerability. Satisfies CVE-2026-40611.

    • github.com/go-acme/lego v4.31.0 → v4.34.0
    • golang.org/x/exp v0.0.0-20241210194714-1829a127f884 → v0.0.0-20260410095643-746e56fc9e2f

  • Upgraded flagged dependencies across the operator components (acme, hdx-scaler-go, tulugaq, version-service) to resolve security advisories.

    • golang.org/x/crypto v0.50.0 → v0.52.0
    • golang.org/x/net → v0.55.0
    • h11 v0.14.0 → v0.16.0

  • Updated Rust and Python dependencies in hdx-node-rs and hdx-pod-metrics control plane services.

    Updates for hdx-node-rs

    Updated once_cell 1.20.2 → 1.21.4 Updated openssl 0.10.72 → 0.10.78 Updated openssl-sys 0.9.107 → 0.9.114 Updated rand 0.8.5 → 0.8.6

    updates for hdx-pod-metrics

    Updated once_cell 1.21.3 → 1.21.4

    updates for hdx-pod-metrics/validation

    Updated matplotlib 3.10.6 → 3.10.9 Updated pillow 11.3.0 → 12.2.0 Updated pygments 2.19.2 → 2.20.0 Updated pytest 8.4.2 → 9.0.3 Removed importlib-resources 6.5.2 Removed zipp 3.23.0

  • Updated hdx-node-rs software dependencies as part of routine maintenance.

    control/hdx-node-rs/Cargo.lock

    Updated darling 0.20.10 → 0.21.3 Updated darling_core 0.20.10 → 0.21.3 Updated darling_macro 0.20.10 → 0.21.3 Updated getrandom 0.2.15 → 0.3.4 Updated hyper 1.5.2 → 1.9.0 Updated hyper-util 0.1.10 → 0.1.20 Updated js-sys 0.3.76 → 0.3.97 Updated json-patch 3.0.1 → 4.2.0 Updated jsonptr 0.6.3 → 0.7.1 Updated k8s-openapi 0.24.0 → 0.26.1 Updated kube 0.98.0 → 2.0.1 Updated kube-client 0.98.0 → 2.0.1 Updated kube-coordinate 0.2.1 → 2.0.0-rc.0 Updated kube-core 0.98.0 → 2.0.1 Updated kube-derive 0.98.0 → 2.0.1 Updated kube-runtime 0.98.0 → 2.0.1 Updated libc 0.2.169 → 0.2.186 Updated rand 0.8.6 → 0.9.4 Updated rand_chacha 0.3.1 → 0.9.0 Updated rand_core 0.6.4 → 0.9.5 Updated reqwest 0.12.9 → 0.12.28 Updated schemars 0.8.21 → 1.2.1 Updated schemars_derive 0.8.21 → 1.2.1 Updated socket2 0.5.8 → 0.6.3 Updated system-configuration 0.6.1 → 0.7.0 Updated tower-http 0.6.2 → 0.6.8 Updated wasm-bindgen 0.2.99 → 0.2.120 Updated wasm-bindgen-futures 0.4.49 → 0.4.70 Updated wasm-bindgen-macro 0.2.99 → 0.2.120 Updated wasm-bindgen-macro-support 0.2.99 → 0.2.120 Updated wasm-bindgen-shared 0.2.99 → 0.2.120 Updated web-sys 0.3.76 → 0.3.97 Updated windows-registry 0.2.0 → 0.6.1 Updated windows-result 0.2.0 → 0.4.1 Updated windows-strings 0.1.0 → 0.5.1 Updated windows-sys 0.59.0 → 0.61.2 Added backon 1.6.0 Added derive_more 2.1.1 Added derive_more-impl 2.1.1 Added gloo-timers 0.3.0 Added iri-string 0.7.12 Added r-efi 5.3.0 Added ref-cast 1.0.25 Added ref-cast-impl 1.0.25 Added rustc_version 0.4.1 Added semver 1.0.28 Added wasip2 1.0.3+wasi-0.2.9 Added windows-link 0.2.1 Added wit-bindgen 0.57.1 Removed backoff 0.4.0 Removed headers 0.4.0 Removed headers-core 0.3.0 Removed hyper-http-proxy 1.0.0 Removed instant 0.1.13 Removed rustls-pemfile 2.2.0 Removed sha1 0.10.6 Removed wasm-bindgen-backend 0.2.99

  • Published a CloudNativePG-based CatalogDB image with PostgreSQL upgraded from 15.2 to 18.3 to the Hydrolix registry, and bumped the pg_dump and pg_repack client tooling. No running CatalogDB changes version; the built-in PostgreSQL image stays at 15.2.

  • Bumped the operator-bundled Hydrolix Grafana datasource plugin from 0.6.0 to 0.10.1, picking up:

    • Per-query Hydrolix options in the Grafana UI (GRAP-59)
    • Regex patterns for ad-hoc filtering (GRAP-77)
    • Exposure of datasource errors to Grafana variables (GRAP-155)

    See the plugin's GitHub releases for full per-version changelogs.

  • Upgraded the operator-bundled Hydrolix MCP server from 0.2.3 to 0.3.1, which supports ability to disable truststore OS certificate store integration.

  • Upgraded the janus-proxy default version from 0.2.2 to 0.2.3. This fixes a schema validation error when interacting with the Config API by dropping the requirement for the is_superuser attribute.

  • Updated turbine-api dependencies for security fixes: djangorestframework 3.16.1 → 3.17.1, pyjwt 2.11.0 → 2.12.0, and requests 2.32.0 → 2.33.0.

  • Applied a further round of Aikido security remediations across operator components, including an aiohttp bump to 3.13.4, pyasn1 to 0.6.3, urllib3 updates, non-root Dockerfiles, SHA-512 checksum validation for pulled artifacts, and a SQL-injection hardening fix in the tulugaq transformer command-line interface.

  • Removed the internal Redpanda dependency and the legacy stream-head and stream-peer resources from the init-turbine job. Redpanda is no longer deployed as a cluster component.

Core updates⚓︎

  • Upgraded the ClickHouse query engine from 24.8.6.70-lts to 25.8.16.34-lts.
    • JSON query results no longer quote 64-bit integers as strings; Int64 and UInt64 values now serialize as JSON numbers. Clients that parse query output may need to adjust their handling.
    • Ordering by a value extracted from a JSON column now requires an explicit cast to a concrete type, because the dynamically typed Variant and Dynamic types are no longer allowed in ORDER BY keys.

Improvements⚓︎

Config API⚓︎

  • Added Config API support for the http-head service. Operators can now create and manage http-head resources through the Config API.

  • Added Config API support for the hdx_query_batch_size query setting.

  • Enabled JSON datatype support in transforms by default. The json_datatype_enabled tunable now defaults to true in v6.1. In earlier versions, it defaulted to false.

  • Added a new turbine_api_trusted_token_issuers operator tunable that lets the Config API accept Bearer tokens signed by external JWT issuers. Each issuer entry takes an iss value, a JWKS URI, optional signing algorithms, and an optional audience. The engine-side AccessStorage honors token TTL by reading the JWT exp claim into Credentials.token_expiration_time. Existing Keycloak and service account authentication paths are unchanged.

  • Refactored and simplified HTTP application proxy logic supporting the Hydrolix console's interaction with Semantix through the Config API. These changes improve separation between HTTP plumbing logic and business logic. These changes also provide comprehensive support and corrections for HTTP 400 and 500 status codes.

  • Added POST /config/v1/users to create users linked to an external identity (issuer + external_id), bypassing the invite flow; tokens from that issuer authorize as the linked user. A new search_external_users parameter on GET /config/v1/users lists them, opt-in now, default after a short deprecation cycle.

  • Added GET /config/v1/orgs/{org_id}/dictionaries/ to list every dictionary in an org in a single RBAC-scoped request, instead of one call per project. The endpoint is list-only; create, update, and delete remain on the project-nested dictionaries endpoint.

  • Tightened request validation to require an exact match on hdx_deployment_id before a request is rejected. Earlier, looser matching could reject requests intended for the current deployment.

  • The Columns API now returns column descriptions and the transforms that reference each column. Descriptions populate from the associated transforms when available.

  • Disabled implicit SQL validation by default. The HDXAPI_QUERY_VALIDATION_DISABLED setting now defaults to true, which avoids false-positive rejections from the hdx_verify_sql path on transforms, merge SQL, and function bodies. The explicit /parse-sql endpoint is unaffected.

Cluster operations⚓︎

  • Added a thanos_initial_delay operator tunable for controlling startup delay on Prometheus Thanos containers. Earlier, the initial_delay setting was taken incorrectly from query application settings.

  • Added a liveness probe to the Hydrolix operator. The probe improves resilience by allowing Kubernetes to restart the operator pod if it becomes unresponsive.

  • Improved the o6r_hdx_ready metric and cluster readiness report with more accurate and granular readiness signals.

  • Decoupled query_head_pooling_enabled from default-pool HTTP routing. The default-pool catch-all route is now controlled by all_clickhouse_http_via_chproxy_enabled and http_proxy.enabled independently. query_head_pooling_enabled retains its pooling-only role: gating per-pool routing and pool-aware HTTP proxy configuration.

  • Added optional and experimental json output format to the tulugaq metrics ingestion transform, controlled by the new tulugaq.format tunable. Scraped metrics nest under a single m JSON column.

  • Added traefik_ip_allowlist and traefik_ip_allowlist_exempt_routes operator tunables for per-route IP allowlists in the Traefik proxy.

  • Bumped default image versions for the opt-in Hydrolix Innovation Lab services Janus and Anomaly RCA (root cause analysis). Clusters that haven't opted in are unaffected; clusters that have opted in pick up the new images on upgrade unless the version field is pinned.

    • Janus v0.3.0 → v0.3.1: LiteLLM upstream version bump, RBAC for the Janus Proxy admin API, keys list pagination, security and dependency fixes.
    • Anomaly RCA to v0.2.0 (the v6.0 default v1.0.0 was a placeholder tag that was never published): unified Traefik/Keycloak authentication, SQL-injection hardening, migration of the LLM client to use Janus Proxy, RCA privilege-escalation fix, urllib3 upgrade, and accumulated bug fixes.

  • Added new services semantix-agent, semantix-llm, and semantix-peer as valid and known to the Hydrolix custom resource definition.

  • Introduced optional support to export OpenTelemetry data from anomaly_rca, janus, and embedded LiteLLM. To enable, set cluster tunable hydrolix_otel or otel_endpoint.

  • Introduced liveness and readiness probes for different versions of the MCP Hydrolix service. This is an implementation of the familiar pattern, which improves resilience and controllability for applications in a Kubernetes cluster. Versions of mcp_hydrolix older than v0.3.1 use only a TCP probe liveness check. Newer versions use both, as HTTP checks.

  • Replaced user and password credentials when configuring services for Kibana Gateway by matching its new support for service account tokens. Since version 2.0, the Kibana Gateway serviceAccountToken variables can hold tokens.

  • The HDX Scaler VPA now caps scale-up targets to the per-node free capacity for the scaled resource. Workloads that can't fit on any available node now return Capped or UnfitNotSchedulable decisions instead of being stuck in Pending.

  • Added spec.scheduler.apply_to_all and spec.scheduler.apply_to_all_exclude operator tunables for opting all operator-managed workloads into hdx-binpack-scheduler globally. When apply_to_all is true, the scheduler is set as the default schedulerName on every operator-created workload, except those in apply_to_all_exclude (default ["descheduler"]). Per-service scale.<svc>.scheduler_name overrides and explicit targeting still take precedence.

  • Hardened the descheduler's eviction policy. The descheduler and operator pods are now protected via the DefaultEvictor labelSelector instead of maxUnavailable: 0 PDBs, so their nodes no longer block kubectl drain or cluster-autoscaler scale-down. A new descheduler.min_replicas tunable (default 2) stops the descheduler from evicting pods in workloads with fewer replicas; set it to 0 to disable the guard.

  • Bumped the operator-managed Grafana from grafana/grafana-enterprise:12.4.2 to 13.0.3, with several quality-of-life improvements. Authentication is now required for Grafana image rendering, custom Grafana URLs are recognized by Keycloak and consolidated in Traefik routing, and the provisioned Prometheus datasource is removed when it's disabled.

  • Added a cluster-wide engine field to hdx-scaler-go as an alternative to setting the scaler engine per service.

  • Added a per-service engine: python | shadow | go field and a dryRun option to hdx-scaler-go, allowing shadow validation of the Go scaler against the Python scaler before migrating a service.

  • Added a grpc.health.v1.Health service and Kubernetes startup and liveness probes to intake-router, matching the health-probe pattern used by other cluster services.

Core⚓︎

  • Reduced memory allocation when reading JSON partitions: blocks where a path has no data now skip column allocation entirely.

  • Added per-pool Prometheus metrics for the PeerCoordinator lifecycle: partition queue depth, peer counts, connection failures, and batch processing duration.

  • Added pod-level Prometheus gauges hdx_cache_capacity_bytes (the configured cache quota for the pod) and hdx_cache_used_bytes (bytes currently occupied by the local file cache). Both gauges always emit, reporting 0 when the cache is disabled, so cache sizing stays observable.

  • Added on-disk caching of partition byte-range data, enabled by default. The new disable_data_cache operator tunable (default false) turns it off, and data_cache_max_partition_age_seconds limits caching to partitions under a given age. Caching reduces repeated object-storage reads on query-heavy workloads at the cost of local disk.

Bug fixes⚓︎

Cluster operations fixes⚓︎

  • Fixed CORS preflight (OPTIONS) requests returning 401 on routes that use unified-auth middleware (for example, /version, /validator, /ingest). Moving the clobber-headers middleware before auth ensures the browser receives CORS response headers before the auth check intercepts the request. Non-preflight requests are unaffected.

  • Fixed the operator so it now restarts the periodic-service automatically when the general configmap or secret changes (for example, on DEPLOYMENT_ID rotation or Argus credential changes). Earlier, those values were silently picked up only on manual restart.

  • Fixed PodDisruptionBudgets to be emitted for all services regardless of whether the descheduler is enabled. Previously, PDBs were gated on descheduler.enabled, which allowed the cluster autoscaler to drain nodes and reduce service replicas to zero.

  • Fixed pgbouncer userlist creation to occur only when pgbouncer is enabled. Previously, the userlist was created unconditionally, causing spurious errors on clusters where pgbouncer is disabled.

  • Fixed the traefik-cfg daemon's routing allowlist, which was using substring matching instead of set membership.

  • Fixed missing Kubernetes permissions for the binpacking scheduler so it can read the extension-apiserver-authentication ConfigMap.

  • Fixed four bugs in hkt render-spec so its output respects per-Kubernetes scale, doesn't crash on partial-patch pools, and round-trips cleanly through hkt get-resources:

    • kubernetes_profile is respected when looking up per-Kubernetes scale values for query-peer, query-head, and postgres.
    • render-spec --include-pools no longer crashes with KeyError: 'cpu' on partial-patch pools that omit a service or cpu override.
    • Rendered specs round-trip cleanly through get-resources.
    • Phantom merge-i, merge-ii, merge-iii pool entries no longer appear in rendered specs.

  • Fixed hdx-traefik-auth so authorization lookups strip query parameters from X-Forwarded-Uri. Endpoints like /version?component=operator no longer return 403 when enable_traefik_authorization is on.

  • Fixed Traefik OpenTelemetry tracing flag ordering so the rendered container command is deterministic across operator restarts. Earlier, hash-randomized iteration produced a different command on each restart, changed the Deployment's pod template hash, rolled a new ReplicaSet, and exposed Traefik's brief readiness gap as a burst of 503s on every operator deploy.

  • Fixed the acme service to retry failed certificate checks after 15 minutes instead of waiting until the next daily tick. A transient issuance failure on a freshly provisioned cluster previously left Traefik on its self-signed default cert for up to 24 hours.

  • Fixed the operator mutating cached resource state when applying an hdx-operator-config patch, which caused reconcile loops to drift and index convergence to fail. The operator now deep-copies resources before patching, keeping the desired state stable across reconcile loops.

  • Added the deployment ID and cluster manifest version to the OpenTelemetry collector service.

  • Improved efficiency and error handling in transform management for Tulugaq metrics collection. Since output columns are dynamically discovered and added, performance and resilience are important.

  • Collected metrics from keycloak services. Earlier, Prometheus wasn't scraping metrics from Keycloak.

  • Fixed token authentication on the query-head-api container by mounting the missing service tokens volume. Earlier, the container couldn't read its service account token.

Config API fixes⚓︎

  • Enabled PostgreSQL connection health checks in turbine-api and turbine-worker. When persistent database connections are turned on (turbine_api_database_connection_max_age greater than 0), the application detects and recycles unhealthy connections before use, reducing API errors from stale connections. The default configuration closes each connection after its request, so clusters that keep the default see no change.

  • Added the GET /config/v1/logout/ endpoint to the published OpenAPI schema under the Auth tag.

  • Corrected several bugs in management of custom views.

    • Addressed several autoview lifecycle problems. Earlier, the autoview didn't become default view after all other views were deleted. Additionally, the autoview was unmodifiable. Accounts with sufficient permission can now set is_default on the autoview.
    • Fixed a buggy order of operations when creating the autoview and setting it as the default.
    • Relaxed the requirement for a description on columns created in views.

  • GCP-deployed clusters can now PATCH Kinesis sources (including for credential rotation). Fixed the Config API's Kinesis source validator to accept https://datastore.googleapis.com/... URLs as valid checkpointer names alongside AWS ARNs.

  • Fixed the Config API to allow primary columns to be marked as virtual: true. Earlier, the API silently set virtual: false on all primary columns and rejected explicit virtual: true on them.

Intake fixes⚓︎

  • Fixed two storage interaction failures in the reaper system that repeatedly enqueued impossible deletion work. Earlier, when the reaper handled deleted storage paths or missing storage locations, it didn't remove the orphaned catalog entries. Subsequent runs would then enqueue the orphans again leading to deep queues, starving other RabbitMQ operations.

  • Fixed intake configuration parsing that aborted when token_list in stream settings was null rather than absent or an empty list.

Core fixes⚓︎

  • Fixed a LOGICAL_ERROR: Chunk info was not set error in MergingAggregatedTransform that occurred on heavy GROUP BY queries with many groups.

  • Fixed an intermittent hang in FileLruCache that could stall query execution on clusters with data caching enabled.

  • Fixed a data-caching crash (SIGSEGV) when zero-size iocbs were present in the create_io_stripes / remove_if path. Aligned stripes are now rebuilt to handle the zero-size case.

  • Fixed range filters using > or < on primary DateTime64 columns to return correct results. Earlier, comparisons on DateTime64 primary keys could produce incorrect filter behaviour.

  • Fixed a crash (SIGSEGV) caused by a column-index shift in remove_if on partial cache hits where one or more requested columns were absent from the cached partition.

  • Fixed a query-head crash (SIGSEGV) when querying columns that contain arrays of JSON values, caused by the new analyzer's handling of those types in the query transformer. This path is exercised more often in v6.1 because JSON datatype support is enabled by default.

  • Fixed access to JSON fields in summary tables. Earlier, querying or writing JSON subcolumns through summary storages could fail or skip data; summary peers schemas now register JSON subcolumns and the read path is bounds-checked.

  • Fixed queries like SELECT json.records[].field returning empty arrays when the field was missing from the first row's array elements. The query now returns the correct values regardless of row order.

  • Fixed JSON ingest to accept unsigned integers in the (INT64_MAX, UINT64_MAX] range, storing them as double instead of raising HdxJsonError.

  • Improved debugging diagnostics by logging the full path when encountering errors during requests to block storage services. Earlier, only the path was logged, limiting visibility. For security reasons, HTTP query parameters are stripped before logging.

  • Silenced a duplicate console debugging log stream from the ClickHouse engine. Turbine already logs to its logging stream.

  • Fixed summary-table queries failing with Illegal type String of argument for aggregate function with Merge suffix when hdx_query_batch_size is set. The batched-dispatch path was using the wrong table function (turbine instead of turbine_summary).

  • Fixed a memory leak in the engine's access storage that grew unbounded as users and row policies were reloaded. Long-running clusters with active user or RBAC churn no longer accumulate memory from each reload cycle.

  • Fixed query cache settings not applying as expected. Picked up as part of the ClickHouse 25.8 upgrade.

  • Removed spurious MySQL-interface log entries from query peers. Picked up as part of the ClickHouse 25.8 upgrade.

  • Fixed IN and OR filters with multiple values on UInt128 columns (such as IP and UUID types) silently dropping rows in multi-partition tables. Affected queries now return all matching rows.

  • Fixed a bug where a storage filesystem was never rebuilt when its credential arrived late in the config.

  • Fixed request-signing failures on GET requests that don't contain a body.

  • Removed ZooKeeper I/O from the query execution path. A background poller now refreshes the ZooKeeper peer list so the query path reads a cached snapshot instead of blocking on ZooKeeper. These changes addressed a query-peer connection reset, a ZooKeeper async-handler context leak, segfaults, and a race condition bug.

  • Fixed point queries with an exact-match timestamp on a primary key (equal minimum and maximum) silently dropping the time filter, which could bypass hdx_query_timerange_required enforcement.