Skip to content

v3.23.1

Kubernetes (HKT) hkt-v3.23.1

1
wget -O hkt https://hdx-infrastructure.s3.amazonaws.com/hkt/hkt-v3.23.1 && chmod +x hkt

General⚓︎

  • [Feature] Custom Pool Scale Profiles - Enable the Definition of custom sizing defaults for scale pools - Custom Scale Profiles
  • [Feature] Indexer Scaling - Enable Indexer scaling capabilities to be able to scale the Indexer container within a Pod by service - Custom Scale Profiles
  • [Feature] Logs - Import Hydrolix Logs into the database under hydro.logs table. This is in addition to the already stored logs within the storage bucket
  • [Feature] Logs - Add names rather than just UUID's to Logs for projects, tables and transforms.
  • [Feature] Transform - Add additional catchall type that if a column exists and is the wrong type the data will be stored here. Transform Structure
  • [Improvement] Query Improved distributed JOIN and IN query execution - improves the overall execution of the joined queries and queries with sub-queries. Pushes intermediate results of a query to the query-peer, thus saving the query-head from completing un-necessary extra processing.
  • [Improvement] Query - Improve Shard Key positioning WHERE my_shard_col = 'x' AND (my_other_col = 'y' OR another_col = 'z')
  • [Improvement] Stream-Summary - Summary tables now use their own streaming table settings rather than the parent tables.
  • [Improvement] Scale Profiles - Updated scale profiles https://docs.hydrolix.io/docs/scale-profiles
  • [Improvement] Traefik - Enhanced traefik availability using a different architecture with kube-proxy

Services⚓︎

  • [Feature] Alter - Support multiple shard key hashes in Alter jobs Altering Data
  • [Feature] API - Enable Pool Scaling via the API
  • [Feature] Ingest Authorisation - Add Basic HTTP Authentication to Ingest end-points Enable Basic Authorization
  • [Feature] External Catalog - Enable the ability to have an external PostGres (Catalog) instance. Aimed for HA Environments.
  • [Feature] Merge - Add support for Merge SQL. Merging Summary Tables
  • [Feature] Merge Pools - Moved Merge to a Pools model for better table/paritition optimisation.Merge
  • [Feature] Query - Add additional query stats to HTTP response headers, including x-hdx-query-stats Query Stats
  • [Feature] Query Full Text Search - Add Indexing options for full text search including defining custom separators
  • [Feature] Stream/Kafka Summary - Ability to set the offset option when reading data for Summary. Default is to read the latest first.
  • [Feature] Vacuum - A new automated service that cleans up orphan partitions and catalog entries.
  • [Feature] RabbitMQ - Add Metrics for RabbitMQ - Batch RabbitMQ, Merge RabbitMQ
  • [Feature] Prometheus - Reaper - Add Metrics for Reaper and Reaper Object Storage Metrics Age/Reaper Metrics
  • [Feature] Prometheus - Add Pools information to Metrics
  • [Feature] Prometheus - Add names rather than just UUID's to Prometheus for projects, tables and transforms.
  • [Feature] Transform - Limits on inputs can now be set on the transform. Any value falling outside these values is rejected. Limiters.
  • [Feature] Authorisation - Create invite URLS via an API end-point without the need to send email

Platforms⚓︎

Observability⚓︎

  • [Feature] Dashboard - New Logs dashboard Grafana Template
  • [Feature] Dashboard - New Kubernetes Streaming Dashboard

Notable Bug Fixes⚓︎

  • [Bug] Query - Bug with incorrect query results when tuples are used speficially WHERE (tuple) IN (tuples).
  • [Bug] Query - Bug fix where the index wasn't used when querying an indexed map.
  • [Bug] Query - Bugfix for empty results on predicates of the form: my_col = ... AND my_col <anything>
  • [Bug] Query - Fixed bug where Query-Head could loose connection to PostGres Catalog.
  • [Bug] Kafka - Add Missing Prometheus labels to metric reporting.
  • [Bug] Kinesis - Add Missing Prometheus labels to metric reporting.
  • [Bug] Merge - Fix Labels and Add human-readable names to merge metrics.
  • [Bug] Logs - Add Human readable Errors for ingest failures.
  • [Bug] Logs - JSON Log Formatter wasn't correctly handling utf8 in some cases. This has now been resolved
  • [Bug] Kinesis - Ensure events are only passed once for ingest.
  • [Bug] Transforms - Add forgotten cs (or centiseconds, centis, ...) to understood regexp fields for timestamps.
  • [Bug] Object Storage - Don't consider 408 and 429 http responses from object storage as permanent errors, apply exponential back-off retry behaviour.
  • [Bug] API Schema - Removed non supported content types from form POST definitions
  • [Bug] Kubernetes Init - Fix for missing KUBERNETES_NAMESPACE envar.

Deprecations⚓︎

  • [Deprecated] Transform - Update transform format to remove deprecated options like position and treatment.
  • [Deprecated] CloudFormation - Removed Cloudformation support.