19 March 2024 - v4.8.10
7 months ago by Rich Vanderwal
Summary v2 and a new DNS resolver
Upgrade on GKE:
kubectl apply -f "https://www.hydrolix.io/operator/v4.8.10/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&gcp-storage-sa=${GCP_STORAGE_SA}"
Upgrade on EKS:
kubectl apply -f "https://www.hydrolix.io/operator/v4.8.10/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}&aws-storage-role=${AWS_STORAGE_ROLE}"
Notable New Features
- Summary v2: New architecture reads from source only once per parent table, supporting diverse summary rollups and groupings without the need for additional infrastructure
General
- API: Added
sample_rate
streaming parameter to tables - API: Better validation for dictionary file delete endpoint
- Control: Added
skip_init_turbine_api
tunable for skipping the database migration pieces during init-turbine-api - Control: Ports for HTTP API and ClickHouse are now modifiable
- Control: Added
/config-version
endpoint to version service - Control: Priority class always schedules daemonset pods
- Control: Kubernetes annotations flow from Hydrolix cluster configuration to deployments
- Control: New
query_peer_liveness_check_path
tunable that allows disabling check - Core: Redesigned DNS resolver
- Core: Added postgres response time in query stats
- Core: Added
hdx_query_unlimited_cnf
flag to uncap CNFs - Data: Speed up summary shutdown
- Data: Fixed summary
enabled
field - Data: Better logging for suspicious timestamps
- Data: Set default indexer count to 1 rather than number of CPUs
- Data: Added better reject logging with full pathnames of reject files
- Data: New metrics created for merge subsystem
- Data: Prioritized smaller partitions (by memory requirement) when creating candidates
- Data: New tunables for candidate creation by era
- Data: Implement Akamai mPulse JSON ingest subtype
- UI: Added autocomplete for columns in summary tables
- UI: Akamai SIEM sources is now a multi-input field
Bug Fixes
- API: Summary table parents are now validated
- API: Disallowed duplicate storage names
- API: Added
io_perf_mode
field to storage settings - API:
force_operation
value must be "True" or 1 to take effect - API: Activities API now captures dictionary CRUD events
- API: Disallowed UUID change during storage patch
- API: Project activity endpoint restored
- API: Resolved activity API returning 404 and 500 errors
- Control: Delayed indexer termination during pod shutdown
- Control: Fixed error creating vector due to insufficient quota on GCP
- Control: Extended grace period for most data services
- Control:
traefik-cfg
no longer exits when the Kubernetes API is slow - Control: Fix
enabled_vector: false
making all pods unschedulable when targeting - Control:
turbine-api
service no longer crashes whenlog_level
is set - Control:
traefik-cfg
no longer exits when many pods are restarting - Core: Enable query on summary tables even if summary.settings.enable is false
- Core: Query peers aren't registered in ZooKeeper until configuration is loaded
- Core: Metadata queries no longer segfault when validating time filters
- Core: Fixed
pg_catalog
segfault in query head - Core: Addressed segfault during shutdown
- Core: Fixed filters on columns and LIKEs in #.catalog queries
- Core: Queries are now reliably cancelled when timeout expires
- Core: Reduced candidate queries for trivial count optimization
- Core: Grafana queries no longer fail when Hydrolix API bearer token expires
- Data: Removed high-cardinality 'path' label from S3 metrics for Prometheus stability
- Data: Muffled error and allow ingest process during shutdown
- Data: More graceful batch peer shutdown by closing a queue
- Data: Merge won't attempt to merge partitions from different storages
- Data: Intake ALTER client checks turbine health to avoid dictionary error
- Data: Reaper respects cloud provider defined for a storage
- Data: Stream head metric labels are now consistently populated
- Data: Various bugfixes, including restore "active" filter on merge query
- Data: Unified Kinesis checkpoint and source credentials
- Data: Kinesis coordinator is scaled to 0 if Kinesis peers are scaled to 0
- Data: Nested null values and index settings are indexed properly for complex types
- Data: WURFL integration no longer uses ERROR level on high-volume paths
- Data: Prefix
db/hdx
is included in path for all partition files - UI: User profile and settings shows settings
- UI: Update links in RBAC policies sidebar
- UI: Fix
raw_size
error messages in table stats and merge stats