Skip to content

v5.10.9

Fix database reconnection on transport error for API workers, allow more flexible argument passing to Prometheus.

This release contains bug fixes to Hydrolix v5.10. Refer to the release notes to see other notable feature announcements and information for this version.

Upgrade instructions⚓︎

Install the latest Hydrolix v5.10 patch release.

Don't skip minor versions when upgrading or downgrading

Skipping versions when upgrading or downgrading Hydrolix can result in database schema inconsistencies and cluster instability. Always upgrade or downgrade sequentially through each minor version, moving to every intermediate minor release in turn rather than jumping directly from the starting version to the target.

Point releases are safe to skip. Upgrade through each minor version, using the latest point release of each one. Don't install every point release along the way.

For example, to go from v5.10 to v6.1, upgrade sequentially to v5.11.10 -> v6.0.19 -> v6.1.10. Downgrading works the same way, in reverse.

Apply the new Hydrolix operator⚓︎

If you have a self-managed installation, apply the new operator directly with the kubectl command examples. If you're using Hydrolix-supplied tools to manage your installation, follow the procedure prescribed by those tools.

1
2
3
4
VERSION=v5.10.12
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

1
2
3
4
VERSION=v5.10.12
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

1
2
3
4
VERSION=v5.10.12
BASEURL="https://www.hydrolix.io/operator/${VERSION}/operator-resources"

kubectl apply -f "${BASEURL}?namespace=${HDX_KUBERNETES_NAMESPACE}"

Monitor the upgrade process⚓︎

Kubernetes jobs named init-cluster and init-turbine-api will automatically run to upgrade your entire installation to match the new operator's version number. This will take a few minutes, during which time you can observe your pods' restarts with your Kubernetes monitor tool.

Ensure both the init-cluster and init-turbine-api jobs have completed successfully and that the turbine-api pod has restarted without errors. After that, view the UI and use the API of your new installation as a final check.

If the turbine-api pod doesn't restart successfully, or other functionality is missing, check the logs of the init-cluster and init-turbine-api jobs for details about failures. This can be done using the k9s utility or with the kubectl command:

1
2
% kubectl logs -l app=init-cluster
% kubectl logs -l app=init-turbine-api

Post-upgrade instructions⚓︎

This one-time post-upgrade step removes the orphaned objects, if they still exist.

1
2
kubectl delete clusterrolebinding "reloader-${HDX_KUBERNETES_NAMESPACE}"
kubectl delete clusterrole "reloader-${HDX_KUBERNETES_NAMESPACE}"

The operator reconciliation logic can't delete the role and binding which become unnecessary after upgrade to v5.10.12 or later.

Changelog⚓︎

Bug fixes⚓︎

Config API⚓︎

  • Enabled PostgreSQL database connection health checks between API worker tasks and their databases. Earlier, API workers experiencing network transport disruption wouldn't detect failure of the long-lived connection and reconnect. Logs showed a repetition of messages like psycopg.OperationalError: the connection is lost until service restart. Connection handling in the API itself wasn't affected.

Cluster operations⚓︎

  • Added the prometheus_extra_args operator tunable, which passes additional CLI flags to the Prometheus binary. Set it to a list of strings such as --storage.tsdb.max-block-duration=2h. Flags that conflict with Prometheus's built-in defaults (such as --config.file and --storage.tsdb.path) override those defaults rather than duplicating them.