Releases
ProductPricingDocsBlog
Back to All

5 August 2024 - v4.17.0

4 months ago by Rich Vanderwal

Circuit breakers and query options enabled for tables and projects, support for Newline-Delimited JSON

Notable New Features

  • New Table and Project-Level Features added to UI
    • Enabled circuit breakers and query options on tables and projects.
  • Support for Newline-Delimited JSON Format
    • Hydrolix is now compatible with Newline-Delimited JSON format to support recent versions of Fluent Bit.

Breaking Changes and Compatibility Notices

Dropped Support for PostgreSQL < 12

As of version 4.17, Hydrolix requires PostgreSQL version 12 or higher. We recommend you use PostgreSQL 13 or higher for easy upgrading. If you are using PostgreSQL version 12, enable the ltree extension as superuser, and consider upgrading to an up-to-date version of PostgreSQL.

If you're in the unlikely scenario of running the internal Hydrolix PostgreSQL pod, apply this Kubernetes configuration to upgrade PostgreSQL from version 11 to version 12 before upgrading. Also, be sure that this job runs in your cluster's namespace, not the 'default' namespace.

Upgrade on GKE:

kubectl apply -f "https://www.hydrolix.io/operator/v4.17.0/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

Upgrade on EKS:

kubectl apply -f "https://www.hydrolix.io/operator/v4.17.0/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

General

  • API
    • New query settings hdx_query_catalog_timeout_ms and hdx_query_unlimited_cnf are available at table, project, and organization level
    • Pools refactor to support pooling more types of services
    • Updated Django to 5.0.3, requiring PostgreSQL 12 or later
    • Django-related library updates: asgiref to 3.7.2, djangorestframework to 3.15.1, drf-spectacular to 0.27.1, psycopg2 to 2.9.9, PyYAML to 6.0.1, ruamel.yaml to 0.18.6, ruamel.yaml.clib to 0.2.8, thriftpy2 to 0.4.20
    • super_admin user lockout scenarios are now prevented
  • Control
    • cluster_name is now reported in usagemeter
    • Added support for loading Storage definitions from configuration
    • Added support for Unified Authentication
    • Removed deprecated expected_tb_per_day setting, which previously controlled number of Redpanda partitions created for a table
    • Size of init containers has been reduced
    • traefik-cfg now logs last watch times to help troubleshoot Intake Head pod issues
  • Data
    • Added application/x-ndjson to the list of accepted content-type formats for event data
    • Primary column existence validation has been removed
    • Refactored and simplified more aggressive merge partition selection
    • Refactored and improved performance of WURFL client
    • Added intake_head_accept_data_timeout to specify time before intake-heads will return 429s to clients
  • UI
    • Added support for Azure and Linode credentials in S3 batch jobs
    • Credentials may now be uploaded from files
    • Enabled circuit breakers and query options on tables and projects
    • Exposed the force_operation query parameter in the edit transform UI

Bug Fixes

  • API
    • Preexisting dictionaries now have load level pre-set to ALL
    • Global policy removal API is more consistent
    • Unencrypted passwords are no longer logged
    • Response code for PUT request to /orgs/org_id/ changed from 500 to 403
    • Fixed orphaned source pool deletion bug
  • Control
    • usagemeter_table_update no longer deadlocks under high insertion rates
    • Turbine will exit if the expansion of its .ini template fails
    • Configuring IP allowlist no longer breaks ingest
    • Fix operator interference with restarts which previously prompted interrupted rollouts and multiple pod restarts
    • Fixed deadlock of the usagemeter_update_table function in the case of high insertion rates of new partitions
  • Core
    • Dictionaries with no load_level setting are now loaded
    • Warnings are now shown for misspelled SETTINGS in non-summary and non-indexer queries
    • Summary SQL settings are no longer applied to all queries
    • Fixed ZooKeeper segfault by replacing abort() with exit()
  • Data
    • catch_all transform setting no longer stores used top-level fields referenced by a JSON pointer
    • Introduced a new catalog index partition_vacuum_idx used by the partition-vacuum query
    • DB interactions for ALTER jobs are now part of a single transaction to prevent partitions locking after cancellation
    • Stack merge era times, resulting in a Merge Era II that is one hour wide
    • Partitions are now removed from the catalog when deleted even if they don't exist in object storage
    • Kinesis KCL containers now mount credentials directory
  • UI
    • Support for policy scopes that aren't within the default UI scope options
    • Logging in no longer hangs following logout from inactivity
    • Dashboard charts now display correct data