v6.2.5

Partition-level bloom filters to skip unnecessary data scans earlier during query, partition data byte-range caching, raised default resource allocation for the query-head-api and http-head pods in production-tier scale profiles, added support for running pgbouncer in front of an Amazon RDS catalog database, added a liveness probe for query peers, and added shortened, v2 Config API endpoints.

Notable new features

Partition-level Bloom filters

The indexer now writes a per-column Bloom filter into each partition manifest. Eligible column types are string, uint128, ip, and uuid.

Queries containing equality or IN predicates use this filter to skip partitions that don't contain the filtered value, improving query performance.

The feature is on by default and can be controlled using the partition_bloom tunable. Set enabled=False to turn off the feature.

Partitions written with Bloom filters use a new Hydrolix partition format version, v4.

• Hydrolix v6.2 writes v4 partitions when Bloom filters are enabled and v3 when disabled.
• Hydrolix v6.1 can read the v4 format, but not the Bloom filter indexes.
• Hydrolix v6.0 can read the v4 format, but skips all indexes, resorting to a full scan for all queries.
• All existing data in v3 partitions remains readable.

Versions of older than Hydrolix v6.0 can't read the new v4 format.

Clusters that may need to roll back further should keeppartition_bloom.enabled set to false.

Partition data caching

Query peers now use local disk to cache column data in addition to caching the partition manifest and index.

This reduces repeated object storage fetches for the same byte ranges of recent column data. Set data_cache_max_partition_age_seconds to exclude older partitions from local caching. Default maximum age for cache eligibility is one day.

Use the disable_data_cache tunable to turn off the feature, which is on by default in Hydrolix v6.2.

Shorter Config API URLs

The Config API introduces shorter URLs in a /config/v2/ namespace. Where the v2 endpoints require an ID, it's accepted as a parameter. Implicit relationships, for example between a table and its project, are no longer required. See also Config API v2 Endpoints. This change is part of the transition stages to improve support for multiple Organizations in a cluster.

---

Breaking changes

query-head-api and http-head use more resources in production-tier scale profiles

The Hydrolix operator default resource assignments for query-head-api and http-head increase for prod, bench, and mega ("production-tier") profiles.

Service	Resource	Before	v6.2
query-head	Memory	1Gi	2Gi
query-head	Storage	1Gi	4Gi
http-head	Memory	512Mi	2Gi
http-head	Storage	512Mi	4Gi

On upgrade, each query-head-api and http-head pod on a production tier cluster requests additional memory and storage by default. This can increase node capacity needs. Clusters that already override the scaling configuration for query-head-api and http-head keep their values.

Mitigation: If you're using the prod, bench, or mega profiles for your workloads and you haven't overridden the query-head-api and http-head pod scaling configurations, ensure your nodes have the available overhead to absorb the additional demand per pod.

Deprecation notices

Redpanda tunables

The internal Redpanda message bus has been decommissioned and their tunables no longer have any effect:

• stream_partition_count
• stream_partition_block
• stream_replication_factor
• logs_topic_partition_count

The operator logs warnings for deprecated tunables present in the Hydrolix cluster spec. A future release will reject these tunables.

Upgrade instructions

Don't skip minor versions when upgrading or downgrading

Skipping versions when upgrading or downgrading Hydrolix can result in database schema inconsistencies and cluster instability. Always upgrade or downgrade sequentially through each minor version, moving to every intermediate minor release in turn rather than jumping directly from the starting version to the target.

For example, it's safe to upgrade to v6.2 from any version in the v6.1 release.

If downgrading from v6.2, downgrade to the latest v6.1 point release.

Apply the new Hydrolix operator

If you have a self-managed installation, apply the new operator directly with the kubectl command examples. If you're using Hydrolix-supplied tools to manage your installation, follow the procedure prescribed by those tools.

GKEEKSLKE and AKS

VERSION=v6.2.5
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

VERSION=v6.2.5
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

VERSION=v6.2.5
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}"

Monitor the upgrade process

Kubernetes jobs named init-cluster and init-turbine-api will automatically run to upgrade your entire installation to match the new operator's version number. This will take a few minutes, during which time you can observe your pods' restarts with your Kubernetes monitor tool.

Ensure both the init-cluster and init-turbine-api jobs have completed successfully and that the turbine-api pod has restarted without errors. After that, view the UI and use the API of your new installation as a final check.

If the turbine-api pod doesn't restart successfully, or other functionality is missing, check the logs of the init-cluster and init-turbine-api jobs for details about failures. This can be done using the k9s utility or with the kubectl command:

% kubectl logs -l app=init-cluster
% kubectl logs -l app=init-turbine-api

If you still need help, contact Hydrolix support.

Downgrade instructions

Use these instructions when downgrading from v6.2 to v6.1.

1. Connect to the Kubernetes cluster using k9s and select the namespace holding the Hydrolix cluster.
2. Select the turbine-api pod and turbine-api container.
3. Invoke a shell by entering s.

4. Execute this command, which will produce STDERR to terminal. It must exit cleanly.

   ./manage.py release_6_1
   

5. Exit the k9s utility by selecting Ctrl+C.

6. Follow the upgrade instructions for the latest Hydrolix v6.1 release.

Changelog

Updates

UI updates

• Updated the Config UI framework and toolchain to current major versions.

  • @next 14.2.32 → 16.2.1
  • react 18.0.17 → 19.2.16
  • react-dom 18.2.0 → 19.2.7
  • typescript 4.7.4 → 5.9.3
  • eslint 8.57.1 → 9.39.0

Cluster operations updates

• Upgraded the operator-bundled Grafana from v12.4.2 -> v13.0.3 to add support for the Git Sync feature

• Upgraded the operator-bundled MCP server software from v0.3.1 -> v0.3.3 to add support for query pools, improve logging, and update software dependencies.

Improvements

Cluster operations improvements

• Improved availability and observability of the hdx-binpack-scheduler: Enabled Prometheus metrics collection and added the scheduler in the scale_min service list. Scheduler state is now visible in metrics dashboards, and Kubernetes restarts the scheduler pod if it becomes unresponsive.

• Added datasource and plugin health monitoring for the in-cluster Grafana deployment which can detect when a datasource plugin or the Prometheus datasource stops responding.

• Added support for running pgbouncer in front of an Amazon RDS catalog database. Set the new catalog_is_rds tunable to true to connect pgbouncer to RDS over TLS.

• Raised the operator's default query-head-api resources on production-tier profiles to 2Gi memory and 4Gi storage, up from 1Gi each, to prevent out-of-memory restarts. Per-cluster overrides are preserved. See the Resource requirement changes note under Upgrade instructions for the upgrade-time footprint impact.

• Added support to specify a query pool in the mcp_hydrolix tunable. The MCP server sets the query_pool in the query options it sends to the query system.

• Added a startup poll for SIEM pool configuration so the POOL_NAME variable can arrive in config data after the SIEM process starts. The process now polls for up to 60 seconds, rather than failing immediately when the configuration isn't yet present at startup.

• Hardened native OpenTelemetry ingest so the endpoint terminates TLS at the ingress and uses the unified-auth system by default. The operator also now blocks intake_head_df_enabled: true unless the required intake services are scaled on.

Config API improvements

• Added shorter, v2 Config API endpoints for transforms, tables, projects, semantix, jobs-alter and jobs-batch. The new endpoints support the same CRUD operations and search filters as their v1 counterparts, but don't require the long endpoints with IDs nested in the paths. The v1 endpoints remain available.

• Created API worker jobs to establish relationships between Credential, Storage, Invite and User objects and their Customer. The jobs run in the background for the duration of the release as a part of the transition to support multiple Orgs.

• Created API worker jobs to derive existing relationships between projects and users to determine default and additional relationships assignments in the new Customer hierarchy. Existing Config API endpoints allow modification of User to Customer relationships. The automatic assignment background job respects manual assignments. This work is part of the transition to support multiple Orgs.

• Created an API worker job to assign the owned_by_customer attribute for all Storage and Credential objects. When either is associated with a single Customer, the job assigns the customer as the owner. Primary storage and corresponding credentials belong to the cluster and are assigned to the Customer which also owns the hydro system project. The job runs in the background for the duration of the release as a part of the transition to support multiple Orgs.

• Added optional audit filters to endpoints for objects whose relationships were derived from existing permissions and project assignment. Administrators can now examine credentials and storage definitions that are unassigned, assigned_multiple, or lacking ownership, owned_by_customer_is_none. Similar filters apply to report on orphaned users or users associated with multiple Customers. This allows inspection of the automated relationship establishment and is a part of the transition to support multiple Orgs.

Intake improvements

• Added table, project, partition, storage, and timestamp context to intake sink error logs. Empty-partition sink outcomes (a no-op from summary indexers that filter all rows) now log at warn level with the message Sink produced empty partition. instead of error level, so they no longer trip alerts.

• Added an OTLP/HTTP receiver on port 4318 (POST /v1/{traces,logs,metrics}) to the intake stream head, accepting protobuf or JSON with optional gzip and a configurable body limit (--otlp-max-body-bytes, default 64 MiB). The existing OTLP/gRPC receiver also gained gzip support.

• Added a bucket label to object store metrics for the GCS, Azure, and S3 intake clients, so operators can break down object store request metrics by bucket.

Core improvements

• Enabled partition data caching by default.

• Added support for tracking and discovery of query peers using the Kubernetes informers instead of Zookeeper. The default is off.

• Added four Bloom filter tunables (partition_bloom_fpr_pct, block_bloom_fpr_pct, partition_bloom_max_bytes, partition_bloom_max_k) and wired them through to indexer sidecars alongside an internal HDX_BLOCK_SIZE config.

• Added partition Bloom filter statistics (bloom_columns, bloom_bytes, and bloom_build_msecs) to the catalog record.

• At partition write time, construct a partition-level Bloom filter for each indexed string and UInt128 column and serialize it in the manifest extension region, implementing the adaptive indexing feature.

Bug fixes

Cluster operations fixes

• Improved Grafana, Traefik, Keycloak, and certificate acquisition configurations to support successful OAuth on the primary cluster URL and the Grafana URL. Ensured that both Grafana auth-required redirects and responding Keycloak redirects interoperate correctly.

• Fixed Keycloak configuration not re-syncing when hydrolix_url changes (for example, on cluster rename). Earlier, the drift produced Invalid redirect uri errors on logout and 500 errors on /config/v1/reset_password.

• Corrected http-head scale profile defaults. Previously, http-head inherited the 512 MiB dev default in the prod scale profile, which caused pod evictions.

• Corrected the construction of the turbine container to include an executable suitable for the exec mode query peer liveness checker. Earlier, only the turbine server, rather than the standalone binary was included in the container. The fix allows the query_peer_liveness_mode tunable to operate properly in exec mode.

• Improved tightness of error detection to exception handling for the operator's Hydrolix cluster spec validating webhook. The change reduces the execution distance between any failed validation and the resulting error reporting. Earlier, this gap combined with another bug to fail all validations and block all changes to the operator.

• Added middleware logic to convert the Grafana orgId HTTP query parameter to an X-Grafana-Org-Id HTTP header. Without this fix, every request to Grafana with both a JWT Authorization header and the orgId query parameter would provoke Grafana's organization role synchronization and return an HTTP 302 to the client. When combined with identity and access management applications that always supply a JWT Authorization header, the client entered a redirection loop. Converting the query parameter to the corresponding supported Grafana HTTP header eliminated the redirection loop.

• Added support for configuring the Kubernetes autoscaler safe-to-evict annotation on Prometheus workloads. The default is false.

• Fixed process stall in Python hdx-scaler after HTTP 410, version too old, response from Kubernetes ConfigMap watcher. After the fix the scaler creates a new watcher when receiving this response and can continue to dynamically update its configuration settings.

• Corrected the rate computation on sampled counters in hdx-scaler-go. Clusters with a pod count exceeding the sample size exposed the faulty logic and didn't scale the workloads properly. Now, the scaler computes the counter rate changes correctly per pod.

• Corrected a logic error in creating horizontal pod autoscalers (HPA) jobs in hdx-scaler-go. For deployments with multiple workloads the operator would generate identical HPAConfig custom resources and create only a single HPA. The operator now correctly matches workloads with HPAConfigs, each suffixed with an integer to distinguish them.

• Fixed a bug that prevented the usage of any autoscaling for Traefik services. Earlier, scale.traefik.hdxscaler was accidentally ignored, requiring static configuration of the replica count.

• In scale_min mode, the single-replica service defaults can now be overridden with scale.<service> tunables so services like Traefik can run in highly available (HA) mode.

Config API fixes

• Fixed transform SQL validation being silently disabled because the QUERY_VALIDATION_DISABLED setting was read as a string. The string 'False' from the environment evaluated as truthy, leaving validation off; the setting now coerces to a real boolean so validation runs as configured.

• Fixed the Config API allowing column names with special characters that ClickHouse would later reject. Column name validation now enforces ClickHouse's alphanumeric/underscore requirements at create and update time.

• Fixed users staying logged in after calling /config/v1/logout/ when Keycloak was misconfigured or unreachable. Logout now reliably clears the local session and cookies regardless of Keycloak state, while still triggering federated SSO logout when Keycloak is reachable.

• Fixed concurrent logins with the same valid credentials tripping Keycloak's brute-force protection and locking out the user. Login requests are now serialized.

Core fixes

• Fixed IN and OR predicates with multiple values on IP and UUID columns silently dropping rows in multi-partition tables. Earlier, such a query returned rows from only the first listed value's partition, with no error; the predicate now matches every value across all partitions.

• Fixed telemetry labels accumulating duplicate entries (for example, region="us",region="us") when a label key was set more than once. Labels now upsert by key, so Prometheus output carries one entry per key.

• Fixed a locking issue during dictionary downloads that was slowing cluster startup.

• Fixed streaming query responses (hdx_query_streaming_result enabled) emitting malformed JSON that broke ClickHouse-protocol clients such as the Grafana plugin. Query stats remain in the X-HDX-Query-Stats header.

• Fixed signed SharedKey and SharedKeyLite requests to Azure object storage which occasionally received HTTP 403 AuthenticationFailed. The signature is always computed over the HTTP headers. Requests without any corresponding body content don't require Content-Type. Intermediary HTTP proxies may optionally remove Content-Type from GET and DELETE requests, making the signature impossible to validate. The correction omits Content-Type from requests without body content.

• Eliminated a configuration race condition for query peers resulting in persistently unavailable storage locations until application restart. Earlier, an object storage definition could be constructed into memory before the corresponding credential was available. Now, the storage definition isn't created until the credential is available.

• Corrected false negative in hdx_verify_sql. Before the fix, validation didn't account for suppressed columns, which are available during transform SQL execution and before they're discarded and the table is written.

• Corrected a segfault bug in the query peer that occurred while reading HDX partition data over HTTP on a partial cache hit. Under certain circumstances, a completely empty column could be elided, resulting in a desynchronization in the data layout.

• Disabled disk (fs) data caching for query head servers. The features consumes memory and is only useful on query peers which read from object storage, so don't configure the feature on query head instances.

• Fixed a status problem when assigning rolling partitions in the query system. Earlier, the peer coordinator finite state machine accidentally treated actively assigned and working peers the same as unassigned peer discovery mode. New partition assignment occurred only on the peer discovery interval. Now, the coordinator assigns new work immediately when an active peer completes its assignment.

Intake fixes

• Fixed a slow disk leak on intake stream-head pods. Empty shard files left behind when low-volume buckets closed are now deleted instead of accumulating until pod restart.

• Fixed multi-org cluster config loading so all enabled orgs' projects, tables, sources, and storages are correctly merged into cluster state. Earlier, only one org's resources would persist on each config reload.

• The s3_error_count and s3_duration intake metrics no longer carry the high-cardinality error label which held the full S3 error string. Update any Grafana dashboards or Prometheus alerts that group or filter these metrics by error before upgrading as the label no longer returns data.

• Adapted to Go 1.25 runtime change for setting GOMAXPROCS. The fix retains earlier behavior for all intake systems by setting containermaxprocs=0.

Merge fixes

• Fixed a merge service tracing startup failure caused by conflicting OpenTelemetry schema URLs (1.21.0 and 1.26.0). Earlier, merge logged Failed to start tracing and emitted no spans.

Query fixes

• The Hydrolix MCP server can now authenticate clients that supply a service-account token in the URL. For example, a Claude.ai custom connector pointed at /mcp?token=<token>. Earlier, the authentication layer rejected the query-parameter token with 401 Unauthorized before the request reached the MCP server.