ProductPricingDocsBlog
Back to All

19 March 2024 - v4.8.10

about 1 month ago by Rich Vanderwal

Summary v2 and a new DNS resolver

Upgrade on GKE:

kubectl apply -f "https://www.hydrolix.io/operator/v4.8.10/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"

Upgrade on EKS:

kubectl apply -f "https://www.hydrolix.io/operator/v4.8.10/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"

Notable New Features

  • Summary v2: New architecture reads from source only once per parent table, supporting diverse summary rollups and groupings without the need for additional infrastructure

General

  • API: Added sample_rate streaming parameter to tables
  • API: Better validation for dictionary file delete endpoint
  • Control: Added skip_init_turbine_api tunable for skipping the database migration pieces during init-turbine-api
  • Control: Ports for HTTP API and ClickHouse are now modifiable
  • Control: Added /config-version endpoint to version service
  • Control: Priority class always schedules daemonset pods
  • Control: Kubernetes annotations flow from Hydrolix cluster configuration to deployments
  • Control: New query_peer_liveness_check_path tunable that allows disabling check
  • Core: Redesigned DNS resolver
  • Core: Added postgres response time in query stats
  • Core: Added hdx_query_unlimited_cnf flag to uncap CNFs
  • Data: Speed up summary shutdown
  • Data: Fixed summary enabled field
  • Data: Better logging for suspicious timestamps
  • Data: Set default indexer count to 1 rather than number of CPUs
  • Data: Added better reject logging with full pathnames of reject files
  • Data: New metrics created for merge subsystem
  • Data: Prioritized smaller partitions (by memory requirement) when creating candidates
  • Data: New tunables for candidate creation by era
  • Data: Implement Akamai mPulse JSON ingest subtype
  • UI: Added autocomplete for columns in summary tables
  • UI: Akamai SIEM sources is now a multi-input field

Bug Fixes

  • API: Summary table parents are now validated
  • API: Disallowed duplicate storage names
  • API: Added io_perf_mode field to storage settings
  • API: force_operation value must be "True" or 1 to take effect
  • API: Activities API now captures dictionary CRUD events
  • API: Disallowed UUID change during storage patch
  • API: Project activity endpoint restored
  • API: Resolved activity API returning 404 and 500 errors
  • Control: Delayed indexer termination during pod shutdown
  • Control: Fixed error creating vector due to insufficient quota on GCP
  • Control: Extended grace period for most data services
  • Control: traefik-cfg no longer exits when the Kubernetes API is slow
  • Control: Fix enabled_vector: false making all pods unschedulable when targeting
  • Control: turbine-api service no longer crashes when log_level is set
  • Control: traefik-cfg no longer exits when many pods are restarting
  • Core: Enable query on summary tables even if summary.settings.enable is false
  • Core: Query peers aren't registered in ZooKeeper until configuration is loaded
  • Core: Metadata queries no longer segfault when validating time filters
  • Core: Fixed pg_catalog segfault in query head
  • Core: Addressed segfault during shutdown
  • Core: Fixed filters on columns and LIKEs in #.catalog queries
  • Core: Queries are now reliably cancelled when timeout expires
  • Core: Reduced candidate queries for trivial count optimization
  • Core: Grafana queries no longer fail when Hydrolix API bearer token expires
  • Data: Removed high-cardinality 'path' label from S3 metrics for Prometheus stability
  • Data: Muffled error and allow ingest process during shutdown
  • Data: More graceful batch peer shutdown by closing a queue
  • Data: Merge won't attempt to merge partitions from different storages
  • Data: Intake ALTER client checks turbine health to avoid dictionary error
  • Data: Reaper respects cloud provider defined for a storage
  • Data: Stream head metric labels are now consistently populated
  • Data: Various bugfixes, including restore "active" filter on merge query
  • Data: Unified Kinesis checkpoint and source credentials
  • Data: Kinesis coordinator is scaled to 0 if Kinesis peers are scaled to 0
  • Data: Nested null values and index settings are indexed properly for complex types
  • Data: WURFL integration no longer uses ERROR level on high-volume paths
  • Data: Prefix db/hdx is included in path for all partition files
  • UI: User profile and settings shows settings
  • UI: Update links in RBAC policies sidebar
  • UI: Fix raw_size error messages in table stats and merge stats