Hydrolix Release Notes

v2.13.3
12 July 2021

Features

  • Updates to Streaming Dashboard in Grafana
  • Updates to Overview Dashboard in Grafana
  • Updates to Batch Dashboard in Grafana
  • API update to JSON Parser

Bug Fixes

  • String column invalid memory read fix.
  • LRU Cache eviction period decreased

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.13.3 && chmod +x hdxctl

v2.13.2
05 July 2021

Bug Fixes

  • Increase Connect, Receive and send connection timeouts for Query architecture.
  • Constrain primary merge partition selection to reasonable dates
  • Query Like and ILike fixes
  • Enable Query Peer failover.

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.13.2 && chmod +x hdxctl

v2.13.0
02 July 2021

Features

  • Adds multi-topic support for kafka peers
  • Adds an extended metadata column to the catalog entry
  • Adds additional metrics for S3 failure visibility
  • Continued improvements to the partition merge process
  • Upgrades Clickhouse to version 21.3.13
  • Adds API documentation details for the catalog list endpoint and pagination changes

Bug Fixes

  • Addressed an issue with Least Recently Used (LRU) cache eviction path
  • Addressed multiple issues that could lead to timeouts when a LIMIT clause was applied to certain queries
  • Changes and improvements to assist with scaling under peak network load conditions
  • Multiple fixes and changes to help with memory optimization

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.13.0 && chmod +x hdxctl

v2.12.5
28 June 2021

Features

  • Adding tempo to view open telemetry traces
  • Experimental support for window function
  • Enable spot fleet support
  • Support tunable toml file to store clientID variable such as ip whitelist and every options available in hdxctl
  • Initial work on auto-scaling
  • Support for caching partition on query-peer

Bug Fixes

  • Fixes UI scaling transform page
  • Fixes Core
    • Changes default partition per core to 10
    • Changes on indexed fields filtering UINT and INT
    • Improve memory utilization

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.12.5 && chmod +x hdxctl

v2.11.7
22 June 2021

Features

  • Adding support for OpenTelemetry on query peer and head
  • Replace lambda function with our own Host intake-misc
    • This is replacing lambda for reaper, decay, merge head, import head, autoingest
  • Replacing docker with podman for services execution
  • Allow scaling zookeeper to 0 or 3
  • Query result can now be written into s3 directly
  • Reduce memory consumption for query
  • Adding response header with query performance information
    • X-HDX-Query-Stats header response
  • Superset improvement
    • provide custom configuration via hdxctl superset set-config hdxcli-cli override.py
    • adding redis caching mechanism for dashboard
    • support async query via celery workers
    • increase number of celery workers
    • change default password to $host-rogue-derby
  • Initial work for shard key using alternate column
  • New page layout for transform management
  • Provide users the ability to set merge interval
  • Refactor batch retry

Bug Fixes

  • Fixes prometheus auto-discovery
  • Fixes Grafana dashboard calculating ingested bytes
  • Fixes for NonNull and empty function
  • Turn debug off in opentelemetry collector logs

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.11.7 && chmod +x hdxctl

v2.11.0
9 June 2021

Features

  • Monitoring cluster with built-in Grafana dashboard
  • Add initial capabilities for open telemetry reporting on query
  • Moved superset to a static EBS Volume
  • Update Grafana to version 8.0

Bug Fixes

  • HDXCTL Bug fixes.
  • Merge bug fixes.
  • Kafka bug fixes.
  • Stream bug fixes.
  • Prometheus discovery bug fixes.
  • Reaper bug fixes.

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.11.0 && chmod +x hdxctl

v2.10.12
17 May 2021

Features

  • Support bundle - provide a simple mechanism to provide Hydrolix support a “bundle” to help diagnose issues.
  • Add Kafka lag metric to Prometheus.
  • OS metrics added to Prometheus - node exporter. Added to Grafana template dashboards to follow.
  • Add support for provisioning grafana datasources
  • Add new shard_key to allow for separate partitions strategy other than just time.
  • Add initial underlying capabilities for Spot Fleet usage.
  • Move bastion to Auto Scaling Group so can be zero’d.

Bug fixes

  • Added HSTS header to force HTTPS usage.
  • Config api - fix duplicate doc ids, allow blank descriptions for projects and tables
  • update the generateschema endpoint to return correct values and the new datatype fields
  • HDXCTL Bug fixes.
  • Merge bug fixes
  • Stream bug fixes.
  • Catchall for unrecognised input fields to honour transform type but keeps JSON intact.
  • Update to default instance types.
  • Catalog performance optimisations
  • Update Reaper/Merge Head timeouts within Lambda.
  • Fix two Small memory leaks on Indexer partition creation

Security

  • Upgrade RDS instance to latest version.

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.10.12 && chmod +x hdxctl

v2.9.3
1 May 2021

Features

  • Add support to change bastion disk size
  • Add Catch-all field for non specified columns in the transform
  • Add Broadcast address for cluster configuration.
  • Add support for retrieving gateway IP from HDXCTL
  • Add new observability metrics for Stream, Kafka and Merge

Bug fixes

  • Various improvements to backend and web UI
  • Fix back tick issue when downloading CSV.
  • Increase open file limits for http (stream) intake services.
  • Fix for base64 types
  • Split large atomix writes for better Kenisis support.
  • Performance improvements to HDXCTL
  • Remove need to pass parent UUID’s in request body when using API.

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.9.3 && chmod +x hdxctl

v2.8.6
28 April 2021

Features

  • Added Kafka tuning for broker read_timeout and max_wait
  • Upgraded Zookeeper Client to 3.7.0
  • Improved network IO stability
  • Added ability for user to add SSH credentials for on-box access
  • Add ability for purely virtual primaries.
  • Add Kafka TLS support (mutual authentication)
  • Add regex filter to batch jobs form in UI
  • Pre-configured Superset server for observability
  • Add additional logging and data metrics for batch ingest.
  • Add configuration for custom S3 KMS.
  • Performance improvement for reaper.

Bug fixes

  • Various improvements to backend, web UI, and Kafka interoperability fetaures
  • Improved network IO stability for Query Head and Peers.
  • Merge fixes and SQS stability fixes
  • Various Query Peer stability fixes
  • Cache region detection calls to avoid S3’s 503 issue (slow-down)

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.8.6 && chmod +x hdxctl

v2.7.7
31 March 2021

Features

  • Added queryable metadata to tables
  • Added query.timeout_secs parameter to the Query API

Bug fixes

  • Various improvements to backend, web UI, and Kafka interoperability fetaures
  • Kafka stability fix

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.7.7 && chmod +x hdxctl

v2.6.3
19 March 2021

Features

  • Added support for array datatypes
  • Added support for int8, int16, int32 datatypes
  • Added support for uint8, uint16, uint32 datatypes
  • Added support for datatime64 (millisecond-resolution) timestamps

Bug fixes

  • Added support for SELECT CREATE TABLE output (to enable Superset interoperability)
  • Improved transform schema grammar for clarity and readability

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.6.3 && chmod +x hdxctl

v2.3.0
17 February 2021

Features

  • Added support for batch ingest of parquet files
  • Added new Cluster UI dashboard to support bulk updates

Bug fixes

  • Added support for Query API POST messages
  • Improved JSON Transform UI tools

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.3.0 && chmod +x hdxctl

v2.2.7
10 February 2021

Features

  • Added transform support for scripted columns
  • Added transform support for default datetime values
  • Transform now allows duplicate positions for scripted columns (CSV)
  • Improved Kafka performance for quicker reads from the stream
  • Improved Batch Ingest resilience

Bug fixes

  • Date verification logic updated to strip quotes
  • JSON transform no longer requires position
  • Fixed conversion of parquet INT96 timestamps to epoch
  • Added delay between partitions merged and reaped
  • Added check to prevent auto-view from generating multiple primary datatimes
  • Auto-view now set as default
  • Added auto-ingest concurrency limit (5)
  • Increased auto-ingest SQS retry and timeout values

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.2.7 && chmod +x hdxctl

v2.1.5 [2021-01-29]

Features

  • Added bulk update to cluster API endpoints
  • Updated compression codecs

Bug fixes

  • Automatically split large batch SQS file payloads
  • Transform no longer allows double-type tags
  • Batch peer db connection release on completion
  • Fixed UI intermittent login issues
  • Added UI API errors alerting improvements
  • Fixed UI Edit User permissions
  • Reduced query peer disk to 30 GB
  • Added check partitions are active before merge
  • Added clean up resources after merge failure

Release Location

wget -O hdxctl https://hdx-infrastructure.s3.amazonaws.com/hdxctl-v2.1.5 && chmod +x hdxctl