ProductPricingDocsBlog
Back to All
added

24 September 2021 - v.2.18.5

over 2 years ago by Jason Turner

Download: hdxctl-v2.18.5

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.18.5 && chmod +x hdxctl

Features

Ingest

  • Ingest: Add max_future_days limiting for tables. Enables the ability to filter date times that are presented too far in the future. Default is this is turned off.
  • Ingest (Batch): Add additional information into the log to include the the input path.
  • Ingest (Batch): Ensure Batch Peers always remove SQS entries from the queue.
  • Ingest (Kafka): Add offset for Kafka peer readers.

Query

  • Query: Enable ability to see active queries.
  • Query: Add query server metrics to Prometheus

Portal/API

  • Portal: Additional input validation and error handling.
  • Portal: Enable Shard Key in Table settings.
  • Portal: Add retry behaviour in Batch Jobs in the UI.
  • Portal: Add delete stopped jobs from the UI.
  • Portal: Add Kafka section to Cluster UI.
  • API: Change behaviour of Transforms and views PATCH to PUT.

Observability

  • Grafana: New Query Dashboard. Deployed in Grafana. Will be deployed with build later.

Bug Fixes

Ingest

  • Ingest: Intake will no longer create SQS queues if they do not exist, instead logging an error.
  • Ingest (Batch): Fix a goroutine leak when batch ingesting JSON that can exhaust memory.

Query

  • Query: Fixed segfault in query engine where unindexed partitions were queried with Indexed ones.
  • Query: Add partition option to query API (for debugging purposes).
  • Query: HTTP client bug fix where at high load the HTTP client could fail when collecting S3.
  • Query: Do not include internal queries in hdx.active_queries

Lifecycle

  • Merge: Partitions are sorted by modification date (most recent first) prior to being provided to the Merge SDK. The Merge SDK will use the first partition as the prototype for what a merged partition's schema should look like.
  • Merge: Bug fix where a non-indexed column is changed to a column caused the Merge-peer to crash.
  • Merge: Resolve an issue where merge components were unable to locate the queue when running without merge pools