Skip to content

This version was never turned live: instead it was merged into the next versin, v5.0. Consider deleting it.⚓︎

Notable new features⚓︎

ClickHouse upgrade⚓︎

The ClickHouse library has been upgraded to version 24.8.6.70.

Better API documentation⚓︎

The API documentation available from the /config/schema/endpoint has been reorganized and improved, allowing for more complete API documentation in future releases.

Simplified Kibana, Quesma, and Elasticsearch integration⚓︎

Deployment of Kibana, Quesma, and Elasticsearch is now provided automatically within Hydrolix.

Upgrade⚓︎

Upgrade on GKE⚓︎

kubectl apply -f "https://www.hydrolix.io/operator/v4.23.0/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

Upgrade on EKS⚓︎

kubectl apply -f "https://www.hydrolix.io/operator/v4.23.0/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

Upgrade on LKE⚓︎

kubectl apply -f "https://www.hydrolix.io/operator/v4.23.0/operator-resources?namespace=$HDX_KUBERNETES_NAMESPACE"

Changelog⚓︎

General⚓︎

API

  • The API documentation available from the/config/schema/ endpoint has been reorganized and improved, making way for more complete API documentation in future releases.
  • The ?force_operation=true query string parameter can now be used to bypass verification when updating summary table SQL. This allows recovery from poor configuration.

Control Plane

  • Deployment of Kibana, Quesma, and Elasticsearch is now provided automatically within Hydrolix using the data_visualization_tools and quesma_configtunables.
  • For deployments running at near-maximum capacity, two new tunables have been added to control the behavior of rolling updates: rollout_strategy_max_surge and rollout_strategy_max_unavailable.

Core

  • Updated ClickHouse from version 23.8.10.43 to 24.8.6.70.

  • Also updated these libraries:

  • openssl from 1.1.1n to 3.3.2

  • absl from 20211102.0 to 20240722.0
  • fmt from 8.1.1 to 9.1.0
  • libpq from 16.4 to 16.5

Data

  • Removed partition-vacuum. It was replaced with a lightweightpartition-cleaner.

  • Amazon Data Firehose can now decode more than one data object from a single data segment.

  • The SaaS metering functionality ("Usagemeter") does better housekeeping than before, cleaning out reported data older than the number of days specified in the usagemeter_preserve tunable.

  • Updated the idna library from version 0.5.0 to 1.0.3 to address a security vulnerability.

UI

  • The Advanced Options rows in tables can now be clicked, and an editor opens in the sidebar for better visibility.

  • The System Health page now has a multi-selection dropdown that includes a new All option for logs.

  • Added a refresh interval of three minutes for the table health widget.

  • Added a Delete Project page to improve the deletion process. The page shows statistics about ingest latency and size, and warns before deletion. You must have a superuser or super_admin role to delete projects.

  • Added a Project Health page to show information about project health, including statistics about ingest latency and size per table.

  • Next.js has been upgraded from 14.2.20 to 14.2.22 to address security issues.

Bug Fixes⚓︎

API

  • Introduced a script that automatically runs on upgrade to patch up existing views that were affected by a previously fixed bug involving datetime and datetime64.
  • Storages can no longer be deleted without force_operation if they are used in a column_value_mapping.
  • The Rust URL pre-signer now accepts blank entries for endpoint_url.
  • Fixed a bug involving burst and limit when burst was set to 0. This affected the creation of projects, tables, and transforms.
  • When using a GET pool endpoint, the number of replicas now returns a range matching the setting in the hydrolixcluster.yaml file. Use current_replicas to see all current replicas.

Control Plane

  • A liveness check has been added to traefik-cfg so it terminates instead of hanging while waiting for timeouts.

Core

  • When issuing SELECT queries that find no data in the specified time range, using LIMIT no longer results in a full table scan.
  • Fixed an issue where a segmentation fault occurred when running a catalog query if the turbine service was started before the postgresql service.
  • Fixed a segfault triggered when using the empty() SQL function with map columns.
  • When using INSERT INTO, if the hdx_storage_id is invalid, the error is now caught and exits instead of displaying a long error message.

Data

  • When you delete a file from a bucket with soft delete enabled, files set as permanent delete are now deleted.
  • You can now use multiple autoingest sources with the same transform. Using the same transform for multiple sources reduces resource costs during ingestion.
  • Updated autoingest to normalize URI filtering across Azure, GCP, and AWS. The overall behavior remains the same.
  • Errors during batch autoingest are now reported fully, including bucket name, error message, and other fields. Previously, only a short text message was returned.
  • JSON-encoded storages are now escaped properly for the batch-head to read, preventing data loss with autoingest.
  • Updated the golang.org/x/net Go package from v0.28to v0.33 to address a security vulnerability that could allow denial-of-service (DoS) attacks.

UI

  • When editing a role, the sidebar now behaves appropriately when adding or removing a policy.