Skip to content

v6.3.2

Shorter v2 Config API endpoints for views, dictionaries, functions, and table sources. Adaptive Indexing chooses a posting list or a block-level bloom filter per column.

Notable new features⚓︎

Shorter v2 Config API endpoints⚓︎

  • The Config API now offers shorter v2 endpoints that don't require Org and Project IDs in the URL request path. This release adds v2 endpoints for views, dictionaries, and functions, and shortens the table source endpoints with v2 variants for Kafka, Kinesis, queue, SIEM, and syslog source types. The v2 endpoints support the same operations as their v1 equivalents.

    For how to move a client from v1 to v2, see Config API v2 Endpoints.

Adaptive Indexing⚓︎

  • When Hydrolix writes a partition, Adaptive Indexing chooses, per column, either a posting list or a block-level bloom filter, based on the column's cardinality in that partition. Posting lists are what Hydrolix writes today. The choice aims to skip more blocks at query time, so the same column can be indexed either way in different partitions. Adaptive Indexing is off by default in this release and is gated by the adaptive_encoding_enabled tunable.

Native OpenTelemetry ingest⚓︎

  • Applications can send OpenTelemetry data straight to a Hydrolix cluster, with no collector-side setup in between. Hydrolix accepts OTLP over gRPC on port 4317, and over HTTP at /v1/traces, /v1/metrics, and /v1/logs. The HTTP routes answer on the cluster's standard HTTPS port, and port 4318 serves the same endpoint for senders that prefer it. Those routes return CORS headers and answer the preflight OPTIONS request, so a browser application can post telemetry directly. Both OTLP encodings are accepted, protobuf and JSON. The HTTP receiver itself arrived in v6.2.5; this release exposes it through the ingress. Native ingest is off by default and is gated by the intake_head_df_enabled tunable. Transform column naming and array-valued fields both affect whether records land, so follow the transform conventions when building one.

  • The intake head now forwards records to stream-head instead of transforming them in the receiver. It finds its forwarding target through Kubernetes EndpointSlice discovery. The ingest service account carries the discovery.k8s.io list, get, and watch permissions this needs.

Breaking changes⚓︎

This release has no breaking changes.

Deprecation notices⚓︎

This release has no deprecations.

Upgrade instructions⚓︎

Don't skip minor versions when upgrading or downgrading

Skipping versions when upgrading or downgrading Hydrolix can result in database schema inconsistencies and cluster instability. Always upgrade or downgrade sequentially through each minor version, moving to every intermediate minor release in turn rather than jumping directly from the starting version to the target.

Point releases are safe to skip. Upgrade through each minor version, using the latest point release of each one. Don't install every point release along the way.

For example, to go from v5.10 to v6.1, upgrade sequentially to v5.11.10 -> v6.0.19 -> v6.1.10. Downgrading works the same way, in reverse.

Apply the new Hydrolix operator⚓︎

If you have a self-managed installation, apply the new operator directly with the kubectl command examples. If you're using Hydrolix-supplied tools to manage your installation, follow the procedure prescribed by those tools.

1
2
3
4
VERSION=v6.3.2
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

1
2
3
4
VERSION=v6.3.2
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

1
2
3
4
VERSION=v6.3.2
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}"

Monitor the upgrade process⚓︎

Kubernetes jobs named init-cluster and init-turbine-api will automatically run to upgrade your entire installation to match the new operator's version number. This will take a few minutes, during which time you can observe your pods' restarts with your Kubernetes monitor tool.

Ensure both the init-cluster and init-turbine-api jobs have completed successfully and that the turbine-api pod has restarted without errors. After that, view the UI and use the API of your new installation as a final check.

If the turbine-api pod doesn't restart successfully, or other functionality is missing, check the logs of the init-cluster and init-turbine-api jobs for details about failures. This can be done using the k9s utility or with the kubectl command:

1
2
% kubectl logs -l app=init-cluster
% kubectl logs -l app=init-turbine-api

For more assistance, contact Hydrolix support.

Downgrade considerations⚓︎

Adaptive Indexing changes how Hydrolix stores high-cardinality indexed string and UInt128 columns, writing a block-level bloom filter in place of a posting list. It's off by default in this release, gated by the adaptive_encoding_enabled tunable. If you turned it on, downgrade only as far as v6.2, because earlier versions can't read the partitions it writes.

Changelog⚓︎

Updates⚓︎

Cluster operations updates⚓︎

  • Upgraded the operator-bundled Kibana Gateway software from v2.0.0 to v2.1.0 to add optional support for multi-table query.

  • Upgraded the operator-bundled Traefik proxy to v3.7.7. It's built with Go 1.26, which sets GOMAXPROCS from the container's CPU limit instead of the host's CPU count.

  • Upgraded the operator-bundled Grafana from v12.4.2 to v13.0.3. Authentication is now required for Grafana image rendering. The grafana_config tunable gained two new blocks. The rendering block enables or disables the image renderer sidecar and sets its version. The datasources block controls whether the local Hydrolix and Prometheus datasources are provisioned. Both keep their previous behavior by default.

Improvements⚓︎

Config API⚓︎

  • Added the shorter v2 Config API endpoints for dictionaries and functions at /config/v2/dictionaries/ and /config/v2/functions/ respectively. These endpoints support the same CRUD operations as v1 but don't require Org and Project IDs in the URL request path. The v1 endpoints remain available.

  • Added an optional free-form text description field to service account tokens. Users may only set description at token creation. The field provides an alternate way to distinguish service account tokens, other than UUID and expiration.

  • Added an optional cardinality_ratio_pct setting to a column's index_options. The value is a percentage that Adaptive Indexing uses when choosing how to index that column, overriding the cluster default. It applies to string columns only.

  • Defined credential types to support API keys for interacting with Anthropic and OpenAI services.

  • Added the shorter v2 Config API endpoint for views at /config/v2/views/. It supports the same CRUD operations as v1 but doesn't require Org and Project IDs in the URL request path. The v1 endpoint remains available.

  • Shortened the table source endpoint URLs and added v2 variants for Kafka, Kinesis, queue, SIEM, and syslog source types that don't require Org and Project IDs in the URL request path. The v1 table source endpoints remain available.

  • Made the job's table field read-only on the Config API and derived it from the job's settings, rather than accepting it directly from the client. Alter jobs take the table from settings.table_id; batch jobs take the table their source writes to. Existing jobs were backfilled, and the job audit snapshot now records table_id.

Core⚓︎

  • Reduced the default per-column size cap for partition bloom filters from 128 KiB to 16 KiB. A column whose filter would exceed the cap gets a degraded false-positive rate, so the smaller default trades a little accuracy on very wide columns for less index size. The partition_bloom.max_bytes_per_column tunable sets the cap.

  • Added the hdx_query_max_perc_before_external_sort tunable, which caps the memory a sort may use as a percentage of total available memory before spilling to disk. Both this tunable and the existing hdx_query_max_perc_before_external_group_by now apply to query pods as well as summary merge queries. Zero, the default for both, deactivates the restriction.

Intake⚓︎

  • Changed the definition of merge_duration histogram buckets to range exponentially from 1 to 1024 seconds. The prior definition obscured diagnostic usefulness by capturing most observations in the uppermost bucket range.

Cluster operations⚓︎

  • Added configuration variables to the mcp_hydrolix tunable. It's now possible to require and limit timerange, limit max memory usage, query attempts, result rows, and cells for MCP queries. MCP worker timeout settings are configurable. All defaults remain unchanged.

  • Introduced support for MCP server resource pooling. Administrators can deploy and scale MCP servers with distinct configurations for separate tenants in the Hydrolix cluster spec. Each pool is assigned a URL based on the pool name, using the pattern /{pool}/mcp.

  • Allowed spec.scale.hdxscalers in the HydrolixCluster resource to be defined as either a list or a dictionary, adding flexibility to how administrators configure scalers.

  • Made the ACME certificate key type configurable through the traefik_tls_key_type tunable, or the TRAEFIK_TLS_KEY_TYPE environment variable. Existing clusters keep the previous RSA2048 default with no change, and a changed key type doesn't take effect until the certificate is next issued or renewed. Supported values are:

    • EC256
    • EC384
    • RSA2048 (default)
    • RSA3072
    • RSA4096
    • RSA8192

  • Added the hkt image-list command, which prints the full list of container images a Hydrolix release needs, including the registry path for each. Use it to mirror a version's images into a private container registry for self-managed or air-gapped installs.

  • Added a secret_env attribute to the grafana_config tunable. It names a Kubernetes secret whose keys are set as Grafana environment variables. This keeps credentials out of the cluster spec. It defaults to unset.

  • Allowed Grafana to run in a highly available configuration on clusters using scale_min. Resource scale can now be overridden when scale_min is set. The Vector local sink is disabled while scale_min is enabled.

  • Added validation for the service and pool replicas values in spec.scale.hdxscalers. A non-integer value now falls back to the server resource's replicas value and logs a warning, rather than being applied as written. Traefik is included in the scalers handled this way.

Bug fixes⚓︎

Core fixes⚓︎

  • Fixed ORDER BY <primary> LIMIT N queries returning fewer than N rows when the catalog held partially-overlapping partitions. The LIMIT N optimization over-counted rows across overlapping partitions and pruned the timestamp range too aggressively. Now only fully-contained partitions count toward the limit, which returns the correct number of rows while keeping the LIMIT N catalog-pruning optimization.

  • Fixed the query authorization cache dropping a live user on a transient failure in the Config API service (a timeout, 5xx, or connection error), rather than only on a genuine invalid or expired token. A transient failure could leave persistent query connections, for example the Grafana ClickHouse plugin, failing with error 192 "User has been dropped" until the connection was recreated, and sometimes required restarting query pods. Authorization now distinguishes a transient failure from a definitive HTTP 401 and keeps the user in place on a transient failure.

  • Fixed #.catalog queries that filter on the partition bloom filter columns bloom_bytes, bloom_columns, and bloom_build_msecs. Those columns hold partition metadata, and the catalog didn't treat them as metadata when pushing a WHERE clause down, so filtering on them could return incorrect results.

  • Fixed aggregation queries on the rolling-partition path returning incorrect results. The memory-efficient distributed aggregation merge requires each input stream to deliver its buckets in ascending order, and the coordinator interleaved them, which silently corrupted the merge. Affected queries now fall back to the order-insensitive merge.

Cluster operations fixes⚓︎

  • Avoided crashing when constructing Kubernetes resources with multiple Presidio PII guard rules for the Janus AI gateway. Earlier, the code would crash when multiple rules were enabled.

  • Corrected the priorityClass for the vector log collection service to match other system-node-critical services. Prior to this fix, the Kubernetes scheduler couldn't preempt other services on a node because they were of higher priority. Setting system-node-critical for vector protects the logging service and allows the scheduler to mark other services for preemption, possibly scaling up additional nodes.

  • Corrected a missing priorityClass attribute on the definition of mcp-hydrolix pods. The MCP server doesn't require a high priority scheduling priority class, so is set to hdx-low, making it more likely to be rescheduled onto different nodes during resource contention.

  • Restarted the autoscaler's Kubernetes watch client when encountering an HTTP 410 resource version too old error. Earlier, the hdx-scaler stalled until a manual pod restart. Now, it keeps responding to configuration changes and scaling decisions.

  • Fixed HTTP query routing that returned 502 or 503 errors when the http-proxy service was enabled but scaled to zero replicas. Queries that carry a database URL parameter now fall back to the query head when http-proxy has no running replicas, matching the native query path's existing fallback.

  • Fixed scheduled scaling overrides staying applied after their window expired when a reconciliation failed, which could pin services to the override's values until the operator restarted. New o6r_hdx_active_overrides and o6r_hdx_override_drift_seconds metrics help monitor for this.

  • Fixed the routing.priority field being dropped when the operator renders query-head pool routing, so priority-based routing between pools now takes effect. Pools that omit routing.priority are unaffected.

  • Fixed hdxscalers entries overwriting each other without warning. HPAConfig and VPAConfig custom resources were named after the deployment alone. A workload with several scalers therefore produced several identically named resources, and only the last one applied survived. Workloads with more than one entry now get a metric suffix per entry. Single-entry workloads keep their existing names, so their custom resources aren't reset on upgrade.

Config API fixes⚓︎

  • Prevented an account from granting permissions it doesn't hold. Creating, modifying, or deleting a role, adding or removing an account from a role, changing an account's roles, inviting an account with roles, and deleting an account all now return 403 when the target carries permissions the acting account lacks. This corrects a permission escalation bug.

  • Corrected global object publishing to include active objects only, and renamed the published key for user-defined functions from udfs to executable_functions to match the v2 naming. Previously, inactive global objects could be published.

  • Kept storage locations and credentials discoverable after their owning Customer is deleted. Because customers are soft-deleted, the owned_by_customer reference previously kept pointing at a customer no longer visible through the API. Deleting a customer now clears owned_by_customer on its storage locations and credentials, so those resources remain reachable through the owned_by_customer filters.

  • Fixed the Config API /users endpoint so service accounts can be filtered with the unassigned=True and assigned_multiple=True parameters. Earlier, those filters excluded service accounts.

  • Fixed a server error when deleting a storage location or resolving a table's storage IDs. Setting only one of column_value_mapping or spread_list on a table's bucket settings persisted the other as a null value instead of leaving it unset. Later operations treated the null as a missing key. Both are now handled as empty, including on tables where the null was already stored.