08 November 2022

Kubernetes (HKT) hkt-v3.23.1

wget -O hkt https://hdx-infrastructure.s3.amazonaws.com/hkt/hkt-v3.23.1 && chmod +x hkt

General

[Feature] Custom Pool Scale Profiles - Enable the Definition of custom sizing defaults for scale pools - Custom Scale Profiles
[Feature] Indexer Scaling - Enable Indexer scaling capabilities to be able to scale the Indexer container within a Pod by service - Custom Scale Profiles
[Feature] Logs - Import Hydrolix Logs into the database under hydro.logs table. This is in addition to the already stored logs within the storage bucket
[Feature] Logs - Add names rather than just UUID's to Logs for projects, tables and transforms.
[Feature] Transform - Add additional catchall type that if a column exists and is the wrong type the data will be stored here. Transform Structure
[Improvement] Query Improved distributed JOIN and IN query execution - improves the overall execution of the joined queries and queries with sub-queries. Pushes intermediate results of a query to the query-peer, thus saving the query-head from completing un-necessary extra processing.
[Improvement] Query - Improve Shard Key positioning WHERE my_shard_col = 'x' AND (my_other_col = 'y' OR another_col = 'z')
[Improvement] Stream-Summary - Summary tables now use their own streaming table settings rather than the parent tables.
[Improvement] Scale Profiles - Updated scale profiles https://docs.hydrolix.io/docs/scale-profiles
[Improvement] Traefik - Enhanced traefik availability using a different architecture with kube-proxy

Services

[Feature] Alter - Support multiple shard key hashes in Alter jobs Altering Data
[Feature] API - Enable Pool Scaling via the API
[Feature] Ingest Authorisation - Add Basic HTTP Authentication to Ingest end-points Enable Basic Authorization
[Feature] External Catalog - Enable the ability to have an external PostGres (Catalog) instance. Aimed for HA Environments.
[Feature] Merge - Add support for Merge SQL. Merging Summary Tables
[Feature] Merge Pools - Moved Merge to a Pools model for better table/paritition optimisation.Merge
[Feature] Query - Add additional query stats to HTTP response headers, including x-hdx-query-stats Query Stats
[Feature] Query Full Text Search - Add Indexing options for full text search including defining custom seperators - Full Text Search
[Feature] Stream/Kafka Summary - Ability to set the offset option when reading data for Summary. Default is to read the latest first.
[Feature] Vacuum - A new automated service that cleans up orphan partitions and catalog entries.
[Feature] RabbitMQ - Add Metrics for RabbitMQ - Batch RabbitMQ, Merge RabbitMQ
[Feature] Prometheus - Reaper - Add Metrics for Reaper and Reaper Object Storage Metrics Age/Reaper Metrics
[Feature] Prometheus - Add Pools information to Metrics
[Feature] Prometheus - Add names rather than just UUID's to Prometheus for projects, tables and transforms.
[Feature] Transform - Limits on inputs can now be set on the transform. Any value falling outside these values is rejected. Limiters.
[Feature] Authorisation - Create invite URLS via an API end-point without the need to send email

Platforms

[Feature] Linode - Support for Linode is now available -Getting Started Linode LKE
[Feature] Minio - Support for Minio Storage

Observability

[Feature] Dashboard - New Logs dashboard Grafana Template - https://grafana.com/grafana/dashboards/17165-hydrologs/
[Feature] Dashboard - New Kubernetes Streaming Dashboard - https://grafana.com/grafana/dashboards/16154-hydrolix-streaming/

Notable Bug Fixes

[Bug] Query - Bug with incorrect query results when tuples are used speficially WHERE (tuple) IN (tuples).
[Bug] Query - Bug fix where the index wasn't used when querying an indexed map.
[Bug] Query - Bugfix for empty results on predicates of the form: my_col = ... AND my_col <anything>
[Bug] Query - Fixed bug where Query-Head could loose connection to PostGres Catalog.
[Bug] Kafka - Add Missing Prometheus labels to metric reporting.
[Bug] Kinesis - Add Missing Prometheus labels to metric reporting.
[Bug] Merge - Fix Labels and Add human-readable names to merge metrics.
[Bug] Logs - Add Human readable Errors for ingest failures.
[Bug] Logs - JSON Log Formatter was not correctly handling utf8 in some cases. This has now been resolved
[Bug] Kinesis - Ensure events are only passed once for ingest.
[Bug] Transforms - Add forgotten cs (or centiseconds, centis, ...) to understood regexp fields for timestamps.
[Bug] Object Storage - Do not consider 408 and 429 http responses from object storage as permanent errors, apply exponential back-off retry behaviour.
[Bug] API Schema - Removed non supported content types from form POST definitions
[Bug] Kubernetes Init - Fix for missing KUBERNETES_NAMESPACE envar.

Deprecations

[Deprecated] Transform - Update transform format to remove deprecated options like position and treatment.
[Deprecated] CloudFormation - Removed Cloudformation support.