1 February 2023

Kubernetes (HKT) hkt-v3.28.9

wget -O hkt https://hdx-infrastructure.s3.amazonaws.com/hkt/hkt-v3.28.9 && chmod +x hkt

Upgrade on GKE:

kubectl apply -f "https://www.hydrolix.io/operator/v3.28.9/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

Upgrade on EKS:

kubectl apply -f "https://www.hydrolix.io/operator/v3.28.9/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

General

[Feature] Summary - Summary service automatically create a new summary table from SQL statement and simplify query for end users.
[Feature] Stream Auth - Setup token based authentication for ingest streaming.
[Feature] Cron - Move scheduled tasks into K8s cronjob to free more ressources
[Feature] UI - Lots of improvement and UI greatness !
[Feature] Core - Allow authentication via username/password over HTTP

Services

[Optimisation] Cron - Move all the scheduled tasks decay, job-purge, merge-cleanup, partition-vacuum, prune-locks, rejects-vacuum, stale-job-monitor, task-monitor into k8s cron.
[Optimisation] Merge - Use of variable merge_memory_coefficient to 2 for summary table to avoid using too much memory.
[Optimisation] Intake (Stream ingest etc) - New settings in table information expected_tb_per_day to set default scale automatically.
[Feature] Kafka - support CSV format for Kafka ingest
[Feature] Transform - include description in the column name

Notable Bug Fixes

[Bug] UI - Ability to upload transform directly from the UI.
[Bug] UI - Limit query response size to avoid blocking the portal.
[Bug] Intake (Stream ingest etc) - Fixed a current_time bug in Auto values not working as expected.
[Bug] Intake (Stream ingest etc) - Remove json file from temporary storage if indexer crash
[Bug] Operator - fix operator crashes on float and int for cpu usage

Platforms

[Feature] Redpanda - Upgrade to redpanda v22.3
[Feature] Scale - work autoscale for summary-service and stream-peer to use better resources

Observability

[Feature] Prometheus - include the service name for every metrics.
[Feature] Intake service - replace UUID with table name and project name in prometheus exporter metrics
[Feature] API - Switch to JSON logging for API application