Skip to content

v6.0.17

Added query-peer liveness probe options and external identity provider support, made hdx-scaler globally configurable, bumped in-cluster Grafana to version 13.0.3, and fixed late-arriving storage credentials and request signing on GET requests without a body.

This release contains bug fixes to Hydrolix v6.0. This changelog covers changes from v6.0.13 to v6.0.17. Refer to the release notes to see other notable feature announcements and information for this version.

Upgrade instructions⚓︎

Install the latest Hydrolix v6.0 patch release.

Don't skip minor versions when upgrading or downgrading

Skipping versions when upgrading or downgrading Hydrolix can result in database schema inconsistencies and cluster instability. Always upgrade or downgrade sequentially through each minor version, moving to every intermediate minor release in turn rather than jumping directly from the starting version to the target.

Point releases are safe to skip. Upgrade through each minor version, using the latest point release of each one. Don't install every point release along the way.

For example, to go from v5.10 to v6.1, upgrade sequentially to v5.11.10 -> v6.0.19 -> v6.1.10. Downgrading works the same way, in reverse.

Apply the new Hydrolix operator⚓︎

If you have a self-managed installation, apply the new operator directly with the kubectl command examples. If you're using Hydrolix-supplied tools to manage your installation, follow the procedure prescribed by those tools.

1
2
3
4
VERSION=v6.0.22
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

1
2
3
4
VERSION=v6.0.22
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

1
2
3
4
VERSION=v6.0.22
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}"

Monitor the upgrade process⚓︎

Kubernetes jobs named init-cluster and init-turbine-api will automatically run to upgrade your entire installation to match the new operator's version number. This will take a few minutes, during which time you can observe your pods' restarts with your Kubernetes monitor tool.

Ensure both the init-cluster and init-turbine-api jobs have completed successfully and that the turbine-api pod has restarted without errors. After that, view the UI and use the API of your new installation as a final check.

If the turbine-api pod doesn't restart successfully, or other functionality is missing, check the logs of the init-cluster and init-turbine-api jobs for details about failures. This can be done using the k9s utility or with the kubectl command:

1
2
% kubectl logs -l app=init-cluster
% kubectl logs -l app=init-turbine-api

If you still need help, contact Hydrolix support.

Post-upgrade instructions⚓︎

This one-time post-upgrade step removes the orphaned objects, if they still exist.

1
2
kubectl delete clusterrolebinding "reloader-${HDX_KUBERNETES_NAMESPACE}"
kubectl delete clusterrole "reloader-${HDX_KUBERNETES_NAMESPACE}"

The operator reconciliation logic can't delete the role and binding which become unnecessary after upgrade to v6.0.21 or later.

Downgrade restrictions⚓︎

If you create service accounts with names longer than 64 characters in v6.0, you must remove or rename those service accounts before downgrading to v5.11.

Service account names are limited to 64 characters in v5.11, so the downgrade migration will fail if any longer names exist.

If you need to downgrade from v6.0 to v5.11, run the schema rollback before applying the v5.11 operator:

  1. Connect to the Kubernetes cluster using k9s and select the namespace holding the Hydrolix cluster.
  2. Select the turbine-api pod and turbine-api container.
  3. Invoke a shell by pressing s.

  4. Run the following command, which produces STDERR to the terminal and must exit cleanly:

    1
    ./manage.py release_5_11

  5. Exit k9s by pressing Ctrl+c.

Then apply the v5.11 operator using the standard kubectl apply procedure shown above, substituting the v5.11 operator URL.

Changelog⚓︎

Improvements⚓︎

Cluster operations improvements⚓︎

  • Added the tunable query_peer_liveness_mode which selects a liveness probe: exec (the ClickHouse-native check), http (the legacy SQL-probe path, and the default), tcp (socket only), or none. The query-peer liveness probe opens a ClickHouse-native TCP connection to the local peer and runs select 1, exercising the query path that query-head uses rather than only the HTTP path. Distinct exit codes let kubectl describe distinguish failure modes.

  • HDX Scaler: Added cluster-wide engine field as an alternative to setting per-service scalers.

  • Bumped the operator-managed Grafana from grafana/grafana-enterprise:12.4.2 to 13.0.3, with several quality-of-life improvements. Authentication is now required for Grafana image rendering, custom Grafana URLs are recognized by Keycloak and consolidated in Traefik routing, and the provisioned Prometheus datasource is removed when it's disabled.

Config API improvements⚓︎

  • The Config API can trust tokens signed by external issuers. The tunable turbine_api_trusted_token_issuers specifies the trusted issuers.

  • Added direct user creation. Create users without going through the invitation flow.

Bug fixes⚓︎

Cluster operations fixes⚓︎

  • Hardened protections for important pods during autoscaling. The operator now emits a PodDisruptionBudget (PDB) for every pool instead of only when the descheduler is enabled, so the cluster autoscaler no longer drains a node in a way that takes all of a service's pods to zero. Protected services are expressed through the descheduler.protected_services tunable and honored only when the descheduler is enabled, so the default PDBs don't block kubectl drain or autoscaler scale-down on clusters that don't run the descheduler.

Config API fixes⚓︎

  • Resolved an issue that prevented Keycloak-issued tokens from being accepted on clusters where the hydrolix_url isn't publicly accessible. This release also includes an operator fix that prevents JSON serialization errors.

Core fixes⚓︎

  • Fixed request-signing failures on GET requests that don't contain a body.

  • Fixed a bug where a storage filesystem was never rebuilt when its credential arrived late in the config.

  • Removed ZooKeeper I/O from the query execution path. A background poller now refreshes the ZooKeeper peer list so the query path reads a cached snapshot instead of blocking on ZooKeeper. These changes addressed a query-peer connection reset, a ZooKeeper async-handler context leak, segfaults, and a race condition bug.