ProductPricingDocsBlog
Back to All

1 February 2024 - v4.6.5

3 months ago by Rich Vanderwal

You can download version 4.6.5 here.

NOTICE:
This version of Hydrolix introduces Unified Authentication by default, which requires an extra configuration step for existing customers using Basic Authentication. To maintain current behavior when upgrading a cluster from an earlier release, ensure that unified_auth: false is set in your cluster's configuration.

NOTICE:
This version of Hydrolix requires PostgreSQL version 11 or higher. We strongly recommend you use PostgreSQL 13 or higher for easy upgrading.

If you are using PostgreSQL version 11 or 12, enable the ltree extension as superuser, and consider upgrading to an up-to-date version of PostgreSQL.

Upgrade on GKE:

kubectl apply -f "https://www.hydrolix.io/operator/v4.6.5/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

Upgrade on EKS:

kubectl apply -f "https://www.hydrolix.io/operator/v4.6.5/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

Notable New Features

  • API, UI, Core: RBAC v2
  • API: Enabled configurable password complexity policy
  • Data: Add WURFL ScientiaMobile integration for device capabilities
  • Core: Optimized user authentication process
  • Core: Apply filtering predicates to catalog queries
  • Data: Implement new scheme for Kinesis shard balancing
  • Data: Merge-Head performance improvements

General

  • API: allow from_input_field for CSV transforms
  • API: Add support for "suppress" flag in transform columns
  • API: Create new Redpanda reset endpoint
  • API: Lower stream setting's cold data default partition width
  • API: Enforce single default bucket per cluster
  • API: Summary table: display parent_table uuid in summary settings
  • Control: Add support for Google Marketplace "Click-to-Deploy" in usagemeter
  • Control: Configuragble tunable Prometheus retention ratio
  • Control: Ability to use static credentials for multiple clouds simultaneously
  • Control: Set all "Server:" HTTP headers to "Hydrolix"
  • Control: Add stream_concurrency_limit tunable to traefik-cfg
  • Control: Adds hydro.monitor table and monitor-ingest service
  • Control: Fix prometheus_retention_size bug
  • Control: Per-container HPA scaling
  • Control: Added liveness check to query peer
  • Control: Set default Kubernetes version to 1.27
  • Control: Added tunables for maximum AST elements during a transform
  • Control: Default scale changes for traefik-cfg, validator, vector and lke nodes
  • Control: Support cookies for hdx-traefik-auth
  • Control: Support Prometheus remote write configuration
  • Control: Security: Traefik / Config-API or Keycloak Authentication
  • Control: Security: Require auth by default
  • Control: New query_peer_liveness_check_path tunable that allows disabling check
  • Core: Optimize trivial row count optimization logic
  • Core: Expose value for primary_key and partition_key in system.tables
  • Core: Improve backoff/retry intervals for cloud object storage
  • Core: Add ability to write data into secondary storage via SQL INSERT
  • Core: Apply filtering predicates to catalog queries
  • Core: Extend #.metadata to include cardinality, dict sizes and sort order of columns
  • Core: Added tunables for user permission request timeout to api
  • Core: Improved query stats
  • Core: Provide more information on SSL errors
  • Core: Add transform's description attribute to output of DESCRIBE
  • Data: Add metrics to SIEM peer
  • Data: Set the controller-runtime logger globally instead of only on the manager
  • Data: Allow ZooKeeper to bootstrap for checkpointing
  • Data: Update MegaTransform for Hydrologs to include more info from query
  • Data: Add summary support to Kafka peer
  • Data: Add SIEM limit parameter
  • Data: Record http_source_byte_count as actual bytes read from the request body before decompression
  • Data: Add table-level request sampling to HTTP
  • Data: Include project and table names in Akamai SIEM peer logs
  • Data: Move memory_coefficient to table merge settings
  • Data: Include signum in log line when catching signals in intake
  • Data: Gather redpanda metrics from /public_metrics endpoint
  • Merge: Set indexer io-perf-mode conservative for merge-peer
  • UI: Refactor/Update Transform page, adding column view
  • UI: Add SIEM peer service into health dashboard
  • UI: Make github transforms available to UI for cloning
  • UI: Improved transform column view
  • UI: Display HDX cluster version in the UI
  • UI: Add controls for max rows to UI
  • UI: Table bucket settings improvements
  • UI: System health improvements
  • UI: Monaco-editor/react: remove dependency on fetching from cdn.jsdelivr.net
  • UI: Remove external font dependencies
  • UI: Allow the user to set pool replicas without specifying a min-max range
  • UI: Make the columns of the query reference table searchable
  • UI: Replace the mem_size with raw_size in sql View Table Health and View Merge Health
  • UI: Better ordering and filtering for dropdowns and tables
  • UI: Better autocomplete for query editor and transform SQL

Bug Fixes

  • API: Delete the summary table Redpanda consumer group when deleting the table
  • API: Fix shard key validation on PATCH requests
  • API: Fix table listing all sources bug
  • API: Fix bug for RBAC list filtering of sources
  • API: Fix UI summary table updates
  • API: Allowed summary tables to have more than one type of sources at a time
  • API: Fixed error on invite URL view when list is POSTed
  • API: RBAC fixed permissions on initial read_only role
  • API: Fixed activity list endpoints
  • Control: Custom Scale Profiles work again
  • Control: Init waits for all Redpandas before creating topics
  • Control: Redpandas retention is now dependent on partition count
  • Control: Reduce cardinality for Prometheus
  • Control: fix Usagemeter segfault
  • Control: simplify pvc scaling process for Crunchydata volumes
  • Control: set default backup retention for Crunchydata to count of 1
  • Control: adds tunable for controlling Crunchydata postgres settings
  • Control: Fix log level indexer names truncation
  • Control: Fix prometheus_retention_size to no longer fail
  • Control: Prometheus no longer fails when prometheus_retention_size is set
  • Control: Fixed prometheus relabel config for Redpanda
  • Control: Fixed configurable prometheus label size limit
  • Control: Fixed some services not responding to scale_off: true
  • Control: Security: hdx-traefik-auth no longer reveals server version
  • Control: Fixed monitor ingest service when basic auth is enabled for stream-head
  • Core: Fixed ls_dir and http requests for non-aws s3 and azure storages
  • Core: Don't load inactive manifest configs on startup
  • Core: NOT LIKE/NOT ILIKE filter expr is now working as expected
  • Core: Fixed timerange validation logic
  • Core: Limited optimization bugfix, should not apply for multi cols in order by
  • Core: Fixed turbine table functions crashing when called without arguments
  • Core: Fixed pg_catalog segfault
  • Core: Shutdown logic no longer causes errors and segfaults
  • Data: Fix string limiter returning base64-encoded strings
  • Data: Upgrade Azure SDK to v1.8.0 to avoid summary-peer crashes
  • Data: Update partition rejector to use configured storage
  • Data: Collect merge pod logs prior to clean up
  • Data: Fix race conditions around datatypes
  • Data: Always ack merge jobs regardless of their outcome
  • Data: Register missing merge metric and properly use defer timers
  • Data: Kinesis stream name or ARN in listShards API call is no longer included if token present
  • Data: Fixed logs appearing as an "unknown" table/project in Kafka/Kinesis logs
  • Data: Ensured Kinesis source increments source row count metric
  • Data: Registered missing merge metric and now properly using defer timers
  • Data: Storage settings are now being used during uploader creation
  • Data: Various merge behavior fixes, new metrics
  • Data: Partition vacuum uses StartAfter instead of ContinuationToken
  • Data: Project name shows in logs rather than project id
  • Data: Does not increment both merge_success and merge_failure on failure
  • Data: Rejects location is fixed by using pather based on default storage
  • Merge: Indexer is now set to io_perf_mode for all buckets, not just the default
  • UI: Fix UI dropdowns to allow for custom option
  • UI: Fix duplicate key console error in single table UI.
  • UI: Fix transform error message in case of malformed JSON input
  • UI: Apply filterOption() to the DropDown component in UI
  • UI: single table page: fix duplicate key errors
  • UI: Send a null value rather than empty string if shard_key input is empty
  • UI: Security: Use relative paths instead of prevUrl
  • UI: Corrected validation for sort_key and shard_key
  • UI: Set HYDROLIX_TOKEN attributes
  • UI: Clickjacking protection