Hydrolix Tunables List
A listing of Hydrolix tunables
A listing of HTN tunables used by Hydrolix. These tunables are set in the hydrolixcluster.yaml
configuration file, under spec:
.
Tunable Name | Description | Default | Examples | Added | Removed |
---|---|---|---|---|---|
acme_enabled | Automatically generate and renew SSL certs for your hydrolix domain. Will override any existing kubernetes secret named traefik-tls | False | v4.20 | ||
admin_email | The email address of a person who will administer the Hydrolix cluster. | v4.20 | |||
argus_fleet_table | Hydrolix table to send fleet info to, in project.table format. | fleet_inventory.fleet_table | v5.4 | ||
argus_fleet_transform | Hydrolix transform name or UUID for fleet reporting. | fleet_transform | v5.4 | ||
argus_fleet_url | URL to send fleet data to. | https: //prometheus-us.trafficpeak.live/ingest | v5.4 | ||
auth_http_read_timeout_ms | Maximum time to wait for a socket read for user-permission data from auth endpoint (turbine-api) | 2000 | v4.20 | ||
auth_http_response_timeout_ms | Maximum time to wait for receiving http headers from auth endpoint (turbine-api) in response to user permission requests | 2000 | v4.20 | ||
autoingest_unique_file_paths | Enable unique file paths from object store by ignoring duplicate paths. | False | v4.22 | ||
aws_credentials_method | DEPRECATED: use db_bucket_credentials_method | View examples | v4.20 | ||
aws_load_balancer_subnets | Subnets to assign to the load balancer of the traefik service when running in EKS | ["subnet-xxxx,mySubnet"] | v4.20 | ||
aws_load_balancer_tags | Additional tags to be added to the load balancer of the traefik service when running in EKS. | ["Environment=dev,Team=test"] | v4.20 | ||
azure_blob_storage_account | The storage account to use when accesing an Azure blob storage container. | v4.20 | |||
basic_auth | A list of Hydrolix services that should be protected with basic auth when accessed over HTTP | [] | v4.20 | ||
batch_peer_heartbeat_period | How frequently a batch peer should heartbeat any task it's working on as a duration string | 5m | v4.20 | ||
bucket | DEPRECATED: use db_bucket_url | v4.20 | |||
catalog_db_admin_db | The default db of the admin user of the Postgres server where Hydrolix metadata is stored. | turbine | v4.20 | ||
catalog_db_admin_user | The admin user of the Postgres server where Hydrolix metadata is stored. | turbine | v4.20 | ||
catalog_db_host | The Postgres server where Hydrolix metadata is stored. | postgres | v4.20 | ||
catalog_db_port | The Postgres server's port where Hydrolix metadata is stored. | 5432 | v4.21 | ||
catalog_intake_connections | Connection pool settings for intake services that connect to the Postgres server where Hydrolix metadata is stored. Available options (1) 'max_lifetime' - the max duration that a connection can live before being recycled. (2) 'max_idle_time' - the max duration that a connection can be idle before being closed. (3) 'max' - the max number of connections that can be opened by each intake service that connects to the Postgres server. (4) 'min' - the minimum number of connections to keep open to the Postgres server. (5) 'check_writable' - if set to true, then when a connection is opened to the Postgres server, ensure the Postgres server the connection is for can handle writes. | {"max_lifetime": "10m", "max_idle_time": "1m"} | v4.20 | ||
clickhouse_http_port | The port to dedicate to the Clickhouse http interface. | 8088 | v4.20 | ||
client_id | DEPRECATED: use hydrolix_name and db_bucket_url | v4.20 | |||
data_service_termination_grace_period | Termination grace period for most data services. | 120 | v4.20 | ||
data_visualization_tools | List of data visualization tools to deploy. Eg. Grafana, Kibana. | [] | v4.23 | ||
db_bucket_credentials_method | The method Hydrolix uses to acquire credentials for connecting to cloud storage. | web_identity | View examples | v4.20 | |
db_bucket_endpoint | The endpoint url for S3 compatible object storage services. Not required if using AWS S3 or if db_bucket_url is provided. | v4.20 | |||
db_bucket_name | The name of the bucket you would like Hydrolix to store data in. Not required if db_bucket_url is provided. | v4.20 | |||
db_bucket_region | Not required if it can be inferred from db_bucket_url . | View examples | v4.20 | ||
db_bucket_type | The object storage type of the bucket you would like Hydrolix to store data in. Not required if db_bucket_url is provided. | View examples | v4.20 | ||
db_bucket_url | The URL of the cloud storage bucket you would like Hydrolix to store data in. | View examples | v4.20 | ||
db_bucket_use_https | If true use https when connecting to the cloud storage service. Inferred from db_bucket_url if possible. | True | v4.20 | ||
decay_batch_size | Number of entries to fetch for each request to the catalog. | 5000 | v4.20 | ||
decay_enabled | Whether or not the Decay Cron Job should run. | True | v4.20 | ||
decay_max_deactivate_iterations | Maximum number of deactivation iterations to execute per table. | v4.20 | |||
decay_max_reap_iterations | Maximum number of reap iterations to execute per table. | v4.20 | |||
decay_reap_batch_size | Number of entries to fetch for each request when locating entries for reaping | 5000 | v4.20 | ||
decay_schedule | CRON schedule for Decay Cron Job | 0 0 * * * | v4.20 | ||
default_query_pool | A name for the default query pool. | query-peer | v4.20 | ||
disable_disk_cache | If true, query peers will immediately delete partition metadata from disk after use | False | v4.20 | ||
disable_traefik_clickhouse_http_port | If true the load balancer will not forward to traefik on port 8088. This port provides a Clickhouse compatible query interface rooted at the root of the service rather than at a subpath. | False | v4.20 | ||
disable_traefik_http_port | If true the load balancer will not forward to traefik on port 80. When tls is enabled, this port is only used to redirect to https. Otherwise this is the main way to access all services. | False | v4.20 | ||
disable_traefik_https_port | If true the load balancer will not forward to traefik on port 443. Only relevant if tls is enabled | False | v4.20 | ||
disable_traefik_mysql_port | If true the load balancer will not forward to traefik on the Clickhouse My SQL interface port. This is port 9004. | False | v5.0 | ||
disable_traefik_native_port | If true the load balancer will not forward to traefik on the Clickhouse native protocol port. This is port 9440 when TLS is enabled or 9000 if not. | False | v4.20 | ||
disable_vector_bucket_logging | Prevent vector from sending logs to the bucket. | False | v4.20 | ||
disable_vector_kafka_logging | Prevent vector from emitting logs to Redpanda. | False | v4.20 | ||
disk_cache_cull_start_perc | Percentage of cache disk space used before starting to remove files. | 75 | v4.20 | ||
disk_cache_cull_stop_perc | Percentage of cache disk space used before stopping removing files. | 65 | v4.20 | ||
disk_cache_entry_max_ttl_minutes | Max TTL for a cache disk entry. It is the longest period of time for which the LRU disk cache can save an entry before it expires. | 360 | v4.20 | ||
disk_cache_redzone_start_perc | Minimum percentage of cache disk space used to be considered as redzone. | 90 | v4.20 | ||
dns_aws_max_resolution_attempts | Maximum number of attempts made by the Dns Resolver for aws and all s3 compatible storages in a given dns refresh cycle | 1 | v4.20 | ||
dns_aws_max_ttl_secs | Max DNS TTL for aws and s3-compaitible storages. It is the longest period of time for which the DNS resolver can cache a DNS record before it expires and needs to be refreshed. max_ttl=0 means dns cache strictly respects the ttl from the dns query response | 0 | v4.20 | ||
dns_azure_max_resolution_attempts | Maximum number of attempts made by the Dns Resolver for azure storage in a given dns refresh cycle | 1 | v4.20 | ||
dns_azure_max_ttl_secs | Max DNS TTL for azure storage. It is the longest period of time for which the DNS resolver can cache a DNS record before it expires and needs to be refreshed. max_ttl=0 means dns cache strictly respects the ttl from the dns query response | 0 | v4.20 | ||
dns_gcs_max_resolution_attempts | Maximum number of attempts made by the Dns Resolver for gcs storage in a given dns refresh cycle | 1 | v4.20 | ||
dns_gcs_max_ttl_secs | Max DNS TTL for gcs storage. It is the longest period of time for which the DNS resolver can cache a DNS record before it expires and needs to be refreshed. max_ttl=0 means dns cache strictly respects the ttl from the dns query response | 0 | v4.20 | ||
dns_server_ip | The ip address of a DNS server used for performance critical purposes. | v4.20 | |||
domain | DEPRECATED: use hydrolix_url. | v4.20 | |||
eks_product_code | EKS product code for use with Amazon Marketplace. | 6ae46hfauzadikp9f8npdbh9v | v4.20 | ||
enable_password_complexity_policy | If set to true, the default password policy defined in hydrolixdocs will apply. These are: Minimum Length 8 characters Uppercase Characters: 1Lowercase Characters: 1Digits: 1Special Characters: 1Not Recently Used: the past 24 passwords Expire Password: 90 days Not Username Not Email | False | v4.20 | ||
enable_query_auth | When enabled requests to the query service, url paths starting with /query will require authentication. | False | v4.20 | ||
enable_traefik_access_logging | If set to true, Traefik will log all access requests. WARNING: This will produce a very high and potentially unmanageable amount of logs | False | v4.20 | ||
enable_traefik_hsts | If set to true, Traefik will enforce HSTS on all its connections WARNING: This may lead to hard-to-diagnose persistent SSL failures if there areany errors in SSL configuration, and cannot be turned off later. | False | v4.20 | ||
enable_vector | Run vector to send kubernetes pod logs to json files in a bucket and to the internal logs topic. Default inferred from the value of scale_off . | v4.20 | |||
env | Environment variables to set on all Kubernetes pods that are part of the Hydrolix cluster. | {} | v4.20 | ||
exp_backoff_additive_jitter | True: (growthfactor)(1 + jitter), False: growthfactor(jitter) | True | v4.20 | ||
exp_backoff_growth_factor_ms | Every sleep will use this as multiplicative factor, ie 2^i * (growth_factor)ms | 50 | v4.20 | ||
force_container_user_root | Set the initial user for all containers to 0 (root). | False | v4.20 | ||
grafana_config | Grafana configuration. NOTE: To enable Grafana deployment, include grafana in the data_visualization_tools tunable. admin_user : Grafana admin username. admin_email : Grafana admin user email. allow_embedding : Prevents embedding Grafana in frames to mitigate clickjacking risks. db_user : Grafana database username. alert_eval_timeout : Timeout for alert evaluation when fetching data from a source. smtp_enabled : Enables email server settings. Requires the GRAFANA_SMTP_PASSWORD secret. smtp_host : Email server host. smtp_user : Email server authentication username. rendering_timeout : Timeout for rendering reports (PDFs, embedded images, or CSV attachments). is_enterprise : Enables Grafana Enterprise. Requires the GRAFANA_LICENSE secret. google_auth_enabled : Enables Google OAuth authentication. Requires the GOOGLE_CLIENT_SECRET secret. google_client_id : Client ID of the Google Auth app. inactive_timeout : Maximum inactive duration before requiring login again. allow_sign_up : Controls Grafana user creation through OAuth. If false, only existing users can log in. | View default | v5.1 | ||
grafana_image | Definition of Grafana image: tag to be used. | grafana/grafana-enterprise: 11.5.0 | v5.1 | ||
hdx_anomaly_detection | Anomaly Detection configuration parameters. The configs key should be map of tenants to their associated config map(s). The config maps must be created outside of the hdx.spec. | View default | View examples | v5.3 | |
hdx_node | Whether or not enable hdx-node daemonset. | False | v4.21 | v5.1 | |
hdx_node_config | HDX-node YAML configuration. | {} | v5.1 | ||
hdx_node_enabled | Whether or not enable hdx-node daemonset. | False | v5.1 | ||
hdx_node_envs | Env vars for hdx-node configuration. Accepted key-value pairs: - LOG_LEVEL: (str) Defines the logging level for the application. Default: 'INFO'. - DRY_RUN: (bool) Whether to enable dry-run mode for the node, performing actions without making changes. Default: True. - NODE_HEALTHCHECK_PORT: (int) Port used for node health checks. Default: 9019. - NODE_NAME: (str) Name of the node for identification. Default: 'unknown-node'. - NAMESPACE: (str) Kubernetes namespace where the node operates. Default: 'default'. - OBSERVATIONS_CONFIGMAP_NAME: (str) Name of the Config Map for node observations. Default: 'hdx-node-observations'. - LEADER_CONFIGMAP_NAME: (str) Name of the Config Map used for leader election. Default: 'hdx-node-leader-election'. - NODE_DOWN_CONFIRMATION_PERIOD_SECONDS: (int) Time in seconds before confirming a node is down. Default: 300. - LEASE_DURATION_SECONDS: (int) Duration of the lease for leader election in seconds. Default: 120. - RETRY_PERIOD_SECONDS: (int) Period in seconds to retry leader election actions. Default: 60. - PROMETHEUS_SERVER_PORT: (int) Port used by the Prometheus server for scraping metrics. Default: 9009. | {} | v4.21 | v5.1 | |
hdx_query_max_memory_usage_perc | Maximum amount of memory to use for running a query on a single server as a percentage of the total available memory | 80 | v4.20 | ||
hdx_query_max_perc_before_external_group_by | Maximum amount of memory to use for running a summary merge query as a percentage of the total available memory. Zero deactivates the restriction. | 0 | v5.2 | ||
hdx_traefik_auth_workers | Number of async workers gunicorn will create for services requests. Default - 1 | 1 | v5.1 | ||
host | DEPRECATED: use hydrolix_url | v4.20 | |||
http_connect_timeout_ms | Maximum time to wait for socket connection to cloud storage to complete | 300 | v4.20 | ||
http_port | The port to serve Hydrolix plain http on. | v4.20 | |||
http_proxy | HTTP-proxy configuraiton parameters. | View default | v5.2 | ||
http_read_timeout_ms | Maximum time to wait between a socket read and cloud storage having data ready to be read | 1000 | v4.20 | ||
http_response_timeout_ms | Maximum time to wait for receiving http headers to complete while reading from cloud storage | 1000 | v4.20 | ||
http_ssl_connect_timeout_ms | Maximum time to wait for ssl handshake during connection to cloud storage to complete | 1000 | v4.20 | ||
http_write_timeout_ms | Maximum time to wait before uploading partition to cloud is complete | 10000 | v4.20 | ||
https_port | The port to serve Hydrolix https on. | v4.20 | |||
hydrolix_name | The name you would like to assign your Hydrolix cluster. Will be the same as the namespace name if not specified | v4.20 | |||
hydrolix_url | The url you would like to use to access your Hydrolix cluster. | View examples | v4.20 | ||
initial_exp_backoff_ms | Sleep time starts from this value and exponentially grows with retry count | 0 | v4.20 | ||
intake_head_accept_data_timeout | Configures the maximum duration that intake-head will wait for a request to be accepted into the partition creation pipeline. If the timeout is reached, the request will be rejected with a 429 status code response. If not configured or set to 0, intake-head pods will not timeout. | 0s | v4.20 | ||
intake_head_catalog_spill_config | Provides configuration of the spill functionality for catalog adds in intake-head whereby catalog adds are spilled to object storage when catalog interactions are slowed or fail on a particular intake-head pod. Supported keys are 'enabled', 'max_concurrent_fetch', 'fetch_lock_expire_duration', 'max_concurrent_spill', 'max_attempts_spill', 'num_partitions' and 'empty_fetch_pause_duration' | View default | v4.20 | ||
intake_head_index_backlog_enabled | Whether to absorb received buckets in a backlog prior to indexing in intake-head to allow for more buffer for absorption in the face of spikes of traffic or throughput disruptions in indexing or uploading of partitions. If enabled, the newest data received will indexed ahead of older data when the backlog grows. | False | v4.20 | ||
intake_head_index_backlog_max_accept_batch_size | Controls the maximum number of buckets accepted from ingestion and added to the backlog at a time. Only applicable if intake_head_index_backlog_enabled is true. | 50 | v4.20 | ||
intake_head_index_backlog_max_mb | Controls the maximum size in MB that the indexing backlog on intake-head is allowed to grow before either dropping data or slowing new entries depending on the configured value of intake_head_index_backlog_trim_enabled. Only applicable if intake_head_index_backlog_enabled is true. | 256 | v4.20 | ||
intake_head_index_backlog_purge_concurrency | Controls the number of workers used to purge buckets from the intake-head backlog when the max size is breached. Only applicable if intake_head_index_backlog_enabled is true. | 1 | v4.20 | ||
intake_head_max_outstanding_requests | Configures the maximum number of requests that an intake-head pod will allow to be outstanding and in process before rejecting new requests with a 429 status code response. If not configured or set to 0, intake-head pods will never reject new requests. | 0 | v4.20 | ||
intake_head_raw_data_spill_config | Provides configuration of the spill functionality for raw data in intake-head whereby ingested data is spilled to object storage when partition generation is slowed on a particular intake-head pod. Supported keys are 'enabled', 'max_concurrent_fetch', 'fetch_lock_expire_duration', 'max_concurrent_spill', 'max_attempts_spill', 'num_partitions', and 'empty_fetch_pause_duration' | View default | v4.20 | ||
io_perf_mappings | Internally used presets for io_perf_mode. Parsed as JSON Array(Array(Int)) | [[2097152, 256, 256], [6291456, 128, 128], [12582912, 64, 64]] | v4.20 | ||
ip_allowlist | A list of CIDR ranges that should be allowed to connect to the Hydrolix cluster load balancer. | ["127.0.0.1/32"] | v4.20 | ||
issue_wildcard_cert | Whether to issue wildcard TLS certificate. NOTE: DNS Challenge will be used. Route53 credentals need to be provided in ROUTE53_AWS_ACCESS_KEY_ID and ROUTE53_AWS_SECRET_ACCESS_KEY via curated secret. | False | v5.3 | ||
job_purge_age | How old a terminal job must be before it's deleted expressed as a duration string | 2160h | v4.20 | ||
job_purge_enabled | Whether or not the Job Purge Cron Job should run. | True | v4.20 | ||
job_purge_schedule | CRON schedule for Job Purge Cron Job | 0 2 * * * | v4.20 | ||
kafka_careful_mode | False | v4.20 | |||
kafka_tls_ca | A CA certificate used by the kafka_peer to authenticate Kafka servers it connects to. | v4.20 | |||
kafka_tls_cert | The PEM format certificate the kafka_peer will use to authenticate itself to a Kafka server. | v4.20 | |||
kafka_tls_key | The PEM format key the kafka_peer will use to authenticate itself to a Kafka server. | v4.20 | |||
kibana_security_enabled | Enable Kibana auth and RBAC via Elasticsearch static credentials. User credentials can be found in hdx-elastic-user Kubernetes secret. | False | v5.4 | ||
kinesis_coordinate_period | For Kinesis sources, how often the coordination process runs which checks for the available shards and peers and distributes consuming amongst available peers | 10s | v4.20 | ||
kinesis_coordinate_strategy | The strategy to use for coordinating Kinesis peers for a Kinesis source. Possible values are EXTERNAL_COORDINATOR or ZOOKEEPER | EXTERNAL_COORDINATOR | v4.20 | ||
kubernetes_cloud | DEPRECATED: use kubernetes_profile. | View examples | v4.20 | ||
kubernetes_premium_storage_class | The storage class to use with persistent volumes created in Kubernetes for parts of a Hydrolix cluster where throughput is most critical. | v4.20 | |||
kubernetes_profile | Use default settings appropriate to this type of Kubernetes deployment. | generic | View examples | v4.20 | |
kubernetes_storage_class | The storage class to use with persistent volumes created in Kubernetes as part of a Hydrolix cluster. | v4.20 | |||
limit_cpu | If set, container cpu limits are set to match cpu requests in Kubernetes. | True | v4.20 | ||
log_level | A dictionary to specify logging verbosity. Keys are service names with the special value of '*' controlling the default. | {} | v4.20 | ||
log_vacuum_concurrency | Number of concurrent log deletion processes | 8 | v4.20 | v5.1 | |
log_vacuum_dry_run | If true, Log Vacuum will only log it's intentions and take no action. | False | v4.20 | v5.1 | |
log_vacuum_enabled | Whether or not the Log Vacuum Cron Job should run. | True | v4.20 | v5.1 | |
log_vacuum_max_age | Maximum age of a log file before it is removed expressed as a duration string. | 168h | v4.20 | v5.1 | |
log_vacuum_schedule | CRON schedule for Log Vacuum Cron Job | 0 4 * * * | v4.20 | v5.1 | |
logs_http_remote_table | An existing Hydrolix <project.table> where the data should land in remote cluster. | hydro.logs | v4.20 | ||
logs_http_remote_transform | A transform schema for ingest in remote cluster. | mega Transform | v4.20 | ||
logs_http_table | An existing Hydrolix <project.table> where the data should land. | hydro.logs | v4.20 | ||
logs_http_transform | A transform schema for ingest. | mega Transform | v4.20 | ||
logs_kafka_bootstrap_servers | A comma separated list of kafka bootstrap servers to send logs to. | redpanda | v4.20 | ||
logs_kafka_topic | A Kafka topic to send logs to. | logs | v4.20 | ||
logs_sink_local_url | The full URI to make local HTTP request to. | http: //stream-head: 8089/ingest/event | v4.20 | ||
logs_sink_remote_auth_enabled | When enabled, remote HTTP will use basic auth from curated secret. | False | v4.20 | ||
logs_sink_remote_url | The full URI to make remote HTTP request to. | v4.20 | |||
logs_sink_type | Type of logs sink. | kafka | v4.20 | ||
logs_topic_partition_count | The number of partitions to assign to the logs topic for stream processing. | 81 | v4.20 | ||
max_concurrent_queries | Max limit on total number of concurrently executed queries. Zero means unlimited. | 0 | v4.20 | ||
max_exp_backoff_seconds | Cap for exponentially back off sleep time | 20 | v4.20 | ||
max_http_retries | Maximum times to retry any query-related http requests that fail | 3 | v4.20 | ||
max_server_memory_usage_perc | Max % of total system memory that server can use/allocate for its operation. | 0 | v4.20 | ||
merge_candidate_concurrency | Number of concurrent Merge Candidate construction queries to run. | 6 | v4.20 | ||
merge_cleanup_batch_size | Number of entries to fetch for each request to the catalog. | 5000 | v4.20 | ||
merge_cleanup_delay | How long before a merged partition should be deleted expressed as a duration string | 15m | v4.20 | ||
merge_cleanup_enabled | Whether or not the Merge Clean-up Cron Job should run. | True | v4.20 | ||
merge_cleanup_schedule | CRON schedule for Merge Clean-up Cron Job | /5 * * * | v4.20 | ||
merge_controller_enabled | Whether or not the next generation merge controller is enabled. | False | v5.0 | ||
merge_dispatch_frequency | How often a slot should be checked for exeeding max_idle. Expressed as duration string (e.g. 5s) | 5s | v4.20 | ||
merge_download_partitions_enabled | Whether or not merge-peer should download partitions locally for processing. | False | v5.3 | ||
merge_first_era_frequency | How often merge candidates should be constructed for the first era. | 10s | v4.20 | ||
merge_head_batch_size | Number of records to pull from the catalog per request by the merge head. | 10000 | v4.20 | ||
merge_interval | The time the merge process waits between checking for mergeable partitions. | 15s | v4.20 | ||
merge_lock_bad_partitions_enabled | Whether or not merge-peer should lock partitions which cannot be read by turbine. | False | v5.3 | ||
merge_max_candidates | Number of candidates to produce per merge target each cycle. | 100 | v4.20 | ||
merge_max_partitions | Number of partitions to be buffered awaiting candidate construction. | 10000 | v5.4 | ||
merge_max_partitions_per_candidate | The maximum number of partitions per merge candidate. | 100 | v4.20 | ||
merge_min_mb | Size in megabytes of the smallest merge tier. All other merge tiers are multiples of this value. | 1024 | v4.20 | ||
merge_primary_window_width | Specifies the interval used to further filter partition selection queries. Smaller values limit the number of records the database needs to produce, but can increase query count. | 1080h | v4.20 | ||
merge_second_era_frequency | How often merge candidates should be constructed for the second era. | 60s | v4.20 | ||
merge_streaming_selector | Whether or not to use the Streaming Candidate Selector | True | v4.20 | ||
merge_third_era_frequency | How often merge candidates should be constructed for the third era. | 60m | v4.20 | ||
metadata | Custom kubernetes labels and annotations to propagate to hydrolix workloads. Changing this value will trigger restarts for all services | {} | View examples | v5.2 | |
monitor_ingest | If enabled, deploy a service to ingest a timestamp into the hydro.monitor table every second | False | v4.20 | ||
monitor_ingest_request_timeout | The number in seconds for HTTP timeout in HTTP POST from monitor_ingest | 1 | v4.21 | ||
monitor_ingest_retry_timeout | The deadline for one submission by monitor ingest including all retries. | 1 | v4.21 | ||
monitor_ingest_timeout | The number in seconds for HTTP timeout in HTTP POST from monitor_ingest | 1 | v4.20 | ||
mysql_port | The port to serve the Clickhouse My SQL interface on if applicable. | 9004 | v5.0 | ||
mysql_port_disable_tls | When True, Traefik will not use TLS configuration on My SQL TCP route. | True | v5.0 | ||
native_port | The port to serve the Clickhouse plaintext native protocol on if applicable. | 9000 | v4.20 | ||
native_tls_port | The port to serve the Clickhouse TLS native protocol on if applicable. | 9440 | v4.20 | ||
oom_detection | Configuration options for detecting indexing OOM scenarios and retry with smaller data sizes if possible for services that perform ingest. Outer keys are names of the ingest services. The supported services are 'intake-head', 'kafka-peer', 'kinesis-peer', and 'akamai-siem-peer'. Available keys under each service are 'k8s_oom_kill_detection_enabled', 'k8s_oom_kill_detection_max_attempts', 'circuit_break_oom_detection_enabled', and 'preemptive_splitting_enabled' | v4.21 | |||
otel_endpoint | Send otlp data to the http server at this URL. | v4.20 | |||
overcommit | When true , removes all requests and limitsfrom Kubernetes containers. Useful when running on a single node Kubernetes cluster with constrained resources. When set to requests , only turns off requests. Similarly, limits removes just the limits.Not being set is the same as false . Note that removing either a memory or CPU limit or request from any container on a pod removes the Guaranteed quality of service class from that pod. | False | v4.20 | ||
overrides | Temp overrides that patch HDX spec while their schedule is active. Override key should be a unique name and value must contain: timezone (string, _required): IANA TZ to evaluate the schedule; defaults to UTC. Offsets (Z, -05: 00, …) are ignored to avoid ambiguity. _ Exactly ONE schedule block: – weekly: { days: [Sun.. Sat], start: "HH: MM", end: "HH: MM" } – window: { start: ISO-8601 datetime, end: ISO-8601 datetime } – cron: { expression: 5-field cron, duration: "4h" | "30m" } - patch (dict, required): partial HDX spec to apply while active.Multiple overrides can be active at the same time; later items in the list win when key conflicts. The HDX custom resource itself is never mutated. Overrides live only in operator memory. The operator writes the list of currently active override names to .status.active Overrides . | {} | View examples | v5.4 | |
owner | DEPRECATED: this was previously used internally by Hydrolix. | v4.20 | |||
partition_cleaner_dry_run | If true, Partition Cleaner will only log it's intentions and take no action | True | v4.21 | ||
partition_cleaner_enabled | Whether or not the partition cleaner cronjob should run. | False | v4.21 | v5.0 | |
partition_cleaner_grace_period | Minimum age of a partition before it is considered for deactivation or deletion expressed as a duration string. | 24h | v4.22 | ||
partition_cleaner_schedule | Crontab style schedule for when partition cleaner should run. | 0 0 * * * | v4.21 | v5.0 | |
partition_vacuum_batch_size | Number of entries to fetch from partition providers on each request. | 10000 | v4.20 | v4.23 | |
partition_vacuum_concurrency | Number of concurrent vacuum operations to run. Each vacuum operation covers a single table. | 5 | v4.20 | v4.23 | |
partition_vacuum_dry_run | If true, Partition Vacuum will only log it's intentions and take no action | True | v4.20 | v4.23 | |
partition_vacuum_enabled | Whether or not the Partition Vacuum Cron Job should run. | False | v4.20 | v4.23 | |
partition_vacuum_grace_period | Minimum age of a partition before it is considered for deactivation or deletion expressed as a duration string. | 24h | v4.20 | v4.23 | |
partition_vacuum_schedule | CRON schedule for Partition Vacuum Cron Job | 0 1 * * * | v4.20 | v4.23 | |
password_expiration_policy | Number of days to expire password | v4.20 | |||
pg_ssl_mode | Determines whether and with what priority an SSL connection will be negotiated when connecting to a Postgres server. See https: //bit.ly/3U9ao8O. | disable | View examples | v4.20 | |
pgbouncer_client_ca_secret | The secret that holds the ca certificates | catalog-ca | v5.4 | ||
pgbouncer_client_tls_secret | The secret that holds the client tls certificates | catalog-server | v5.4 | ||
pgbouncer_enabled | The tunable installs pgbouncer. | False | v5.4 | ||
pgbouncer_max_client_conn | Maximum number of client connections allowed. | 1000 | v5.4 | ||
pgbouncer_metrics_port | The port on which pgbouncer metrics can be scraped. | 9127 | v5.4 | ||
pgbouncer_pool_mode | The pooling method to use for connecting to backend postgres | session | View examples | v5.4 | |
pgbouncer_pool_size | Number of server connections to allow per user/database pair. | 20 | v5.4 | ||
pgbouncer_port | The port on which pgbouncer starts. | 6432 | v5.4 | ||
pools | A list of dictionaries describing pools to deploy as part of the Hydrolix cluster. | v4.20 | |||
postgrescluster_backup_retention | Depends on postgrescluster_backup_retention_type. Either the count, or the number of days to retain full postgres backups before expiring them. All differential and incremental backups that depend on full backups are also expired. Only considered when use_crunchydata_postgres is true | 1 | v4.20 | v5.2 | |
postgrescluster_backup_retention_type | Allowed values are 'count' (default) and 'time'. If set to 'count', then postgrescluster_backup_retention represents the number of full backups to retain before expiring them. If set to 'time', then postgrescluster_backup_retention represents a number of days for retention. Only considered when use_crunchydata_postgres is true . | count | View examples | v4.20 | v5.2 |
postgrescluster_backup_schedules | Backup schedules for postgres. Keys are the type of backup to perform, either 'full', 'differential' or 'incremental'. Values are cron expressions representing the schedule for that backup type. Only considered when use_crunchydata_postgres is true . | {"full": "0 16 _ 0", "differential": "0 16 _ 1-6"} | v4.20 | v5.2 | |
postgrescluster_metrics_enabled | If true, collect prometheus metrics from the postgres cluster. Changing this value will trigger a rolling restart of the postgres pods. Only considered when use_crunchydata_postgres is true | False | [] | v4.20 | v5.2 |
postgrescluster_settings | Settings to apply to postgres. Changes to this value will be automatically reloaded without validation. Changes to certain parameters will cause postgres to restart | {} | [] | v4.20 | v5.2 |
prometheus_curated_configmap | Custom curated prometheus configmap that will be mounted onto the prometheus pod. | [] | v5.0 | ||
prometheus_enabled | This tunable controls prometheus installation | True | v5.3 | ||
prometheus_ignored_apps | A comma delimited list of app labels to ignore when determining scrape targets for prometheus | View examples | v5.2 | ||
prometheus_label_value_length_limit | If a label value is larger than the value configured, Prometheus will discard the entire scrape. | 512 | [] | v4.20 | |
prometheus_namespace | The namespace where kube-prometheus service is running which is used when prometheus operator is used instead of the defacto prometheus installed with hydrolix. | v5.3 | |||
prometheus_operator_installed | This tunable indicates if prometheus-operator is installed. It does not perform installation if the flag is set to False | False | v5.3 | ||
prometheus_remote_write_url | A URL you wish to use to configure Prometheus's remote-write functionality. | [] | v4.20 | ||
prometheus_remote_write_username | The username for Prometheus to use with basic auth to connect to a remote-write endpoint. Ignored if prometheus_remote_write_url is not set. | hdx | [] | v4.20 | |
prometheus_retention_ratio | The amount of the volume to reserve for prometheus data. Example: 0.7 | 0.7 | v4.20 | ||
prometheus_retention_size | The maximum number of bytes of prometheus data to retain. Overrides prometheus_retention_ratio. Units supported: B, KB, MB, GB, TB, PB, EB | v4.20 | |||
prometheus_retention_time | When to remove old prometheus data. Example: 15d | v4.20 | |||
prometheus_scrape_interval | How frequently to scrape targets by default. | 15s | [] | v4.20 | |
prometheus_service_name | The kube-prometheus service name which is used when prometheus operator is used instead of the defacto prometheus installed with hydrolix. | v5.3 | |||
prometheus_service_port | The kube-prometheus service port which is used when prometheus operator is used instead of the defacto prometheus installed with hydrolix. | 9090 | v5.3 | ||
prometheus_servicemonitor_selector | Prometheus custom resource(CR) uses this tunable to identify the servicemonitors to be scraped for metrics | {} | {"hydrolix": "true"} | v5.3 | |
prune_locks_enabled | Whether or not the Prune Locks Cron Job should run. | True | v4.20 | ||
prune_locks_grace_period | Minimum age of a lock before it is considered for removal expressed as a duration string. | 24h | v4.20 | ||
prune_locks_schedule | CRON schedule for Prune Locks Cron Job | 30 0 * * * | v4.20 | ||
query_peer_liveness_check_path | The http path used to configure a kubernetes liveness check for query-peers. Set to 'none' to disable. | View default | v4.20 | ||
query_peer_liveness_failure_threshold | How many times query liveness check can fail. | 5 | v4.20 | ||
query_peer_liveness_initial_delay | Time in seconds to wait before starting query liveness checks. | 300 | v4.23 | ||
query_peer_liveness_period_seconds | How often should query liveness check run, in seconds. | 60 | v4.20 | ||
query_peer_liveness_probe_timeout | Number of seconds after which the liveness probe times out | 10 | v4.23 | ||
query_readiness_initial_delay | Time in seconds to wait before starting query readiness checks. | 0 | v4.20 | ||
quesma_config | Quesma config for Hydrolix data source parameters. | {"project": "hydro", "table": "logs"} | v4.23 | ||
refresh_job_statuses_enabled | Whether or not the Refresh Job Statuses Cron Job should run. | True | v4.20 | ||
refresh_job_statuses_schedule | CRON schedule for Refresh Job Statuses Cron Job | *** | v4.20 | ||
registry | A docker registry to pull Hydrolix containers from. | v4.20 | |||
rejects_vacuum_dry_run | If enabled, the Rejects Vacuum Cron Job will not delete files, but instead log its intentions. | False | v4.20 | v5.1 | |
rejects_vacuum_enabled | Whether or not the Rejects Vacuum Cron Job should run. | True | v4.20 | v5.1 | |
rejects_vacuum_max_age | How old a rejects file should be before deleted, expressed as a duration string (e.g. 1h5m4s}. | 168h | v4.20 | v5.1 | |
rejects_vacuum_schedule | CRON schedule for Reject Vacuum Cron Job | 0 0 * * * | v4.20 | v5.1 | |
rollout_strategy_max_surge | This tunable configures the number of pods (represented as percentage) that can be created above the desired amount of pods during deployment rollout update. Default: 25% | 25 | v4.23 | ||
rollout_strategy_max_unavailable | This tunable ensures the number of pods (represented as integer) that can be unavailable during deployment rollout update. Default: 0 | 0 | v4.23 | ||
sample_data_url | The storage bucket url to use to load sample data. | v4.20 | |||
scale | A list of dictionaries describing overrides for scale related configuration for Hydrolix services. | v4.20 | |||
scale_min | When true, similar to scale_off but keeps API, UI and their dependencies running. | False | v5.3 | ||
scale_off | When true, override all deployment and statefulset replica counts with a value of 0 and disable vector. | False | v4.20 | ||
scale_profile | Selects from a set of predefined defaults for scale | eval | v4.20 | ||
sdk_timeout_sec | How many seconds the Merge SDK should be given to run before it is killed. | 300 | v4.20 | ||
siem_backoff_duration | Backoff duration when SIEM limit not hit, for politeness. | 1s | v4.20 | ||
silence_linode_alerts | If true will run a daemonset that turns off Linode alerts for LKE nodes. | False | v4.20 | ||
skip_init_turbine_api | Skips running database migrations in the init-turbine-api job. Set to true when running multiple clusters with a shared database | False | v4.20 | ||
spill_locks_cleanup_enabled | Whether or not the Spill Locks Clean-up Cron Job should run. | True | v4.20 | v5.2 | |
spill_locks_cleanup_schedule | CRON schedule for Spill Locks Clean-up Cron Job | 0 12 * * * | v4.20 | v5.2 | |
sql_transform_max_ast_elements | The number of AST elements an SQL transform can contain. This limits the maximum complexity of a SQL tranform. | View examples | v4.20 | ||
sql_transform_max_expanded_ast_elements | The number of expanded AST elements an SQL transform can contain. This limits the maximum complexity of a SQL tranform. | View examples | v4.20 | ||
stale_job_monitor_batch_size | How many jobs to probe in a single request. | 300 | v4.20 | ||
stale_job_monitor_enabled | Whether or not the Statel Job Monitor Cron Job should run. | True | v4.20 | ||
stale_job_monitor_limit | How many jobs in total Stale Job will process per cycle. | 3000 | v4.20 | ||
stale_job_monitor_schedule | CRON schedule for Stale Job Monitor | /5 * * * | v4.20 | ||
str_dict_enabled | Enable/disable multi-threaded string dictionary decoding. | True | v4.20 | ||
str_dict_min_dict_size | Controls the number of entries in each string dictionary block. | 32768 | v4.20 | ||
str_dict_nr_threads | Sets the maximum number of concurrent v CPU used for decoding. | 8 | v4.20 | ||
stream_concurrency_limit | The number of concurrent stream requests per cpu allocated across all pods beyond which traefik will return 429 busy error responses. If not set or set to null no limit is enforced. | v4.20 | |||
stream_load_balancer_algorithm | Controls how Traefik balances streaming traffic across intake-head and stream-head pods. | rr | View examples | v4.20 | |
stream_partition_block | The number of partitions to use on a non-default redpanda stream topic per TB/day of usage. | 6 | v4.20 | ||
stream_partition_count | The number of partitions to use on the default redpanda topic for stream service. | 50 | v4.20 | ||
stream_replication_factor | The replication factor for the internal Redpanda topic used by the Stream service it must always be less than the number of Redpanda replicas. If it is not, the configuration will not change. | 3 | v4.20 | ||
targeting | A dictionary to pass targeting related Kubernetes settings to resources according to what Hydrolix servivce they are part of. | {} | v4.20 | ||
task_monitor_enabled | Whether or not the Task Monitor Cron Job should run. | True | v4.20 | ||
task_monitor_heartbeat_timeout | How old a tasks heartbeat should be (in seconds) before it is timed out. | 600 | v4.20 | ||
task_monitor_schedule | CRON schedule for Task Montior | /2 * * * | v4.20 | ||
task_monitor_start_timeout | How old a ready task should be (in seconds) before it is considered lost and timed out. | 21600 | v4.20 | ||
terminate_tls_at_lb | This flag controls if traefik accepts http or https traffic from load balancer. This is required if TLS termination needs to be done at the load balancer and plain traffic is sent to traefik backend | False | v5.4 | ||
traefik_external_ips | Traffic that ingresses into the cluster with one of these IP's gets directed to the traefik service. Useful in particular when deploying all on one node. | View examples | v4.20 | ||
traefik_hsts_expire_time | Expiration time for HSTS caching in seconds. | 315360000 | v4.20 | ||
traefik_keep_alive_max_time | The number of seconds a client HTTTP connection can be reused before recieving a Connection: close response from the server. Zero means no limit. | 26 | v4.20 | ||
traefik_service_allowed_headers | Header keys used in intake-head route configuration for pools. The entries should match the keys being passed in the pool annotations. The traefik daemon checks if pool annotations are in this list and if so crafts the traefik router rule accordingly. If left empty, operator checks if x-hdx-table, x-hdx-transform annotations are set for the pool and if so, uses them. | [] | View examples | v5.4 | |
traefik_service_allowed_query_params | Query params used in intake-head route configuration for pools. The entries should match the keys being passed in the pool query_params. The traefik daemon checks if pool query_params are in this list and if so crafts the traefik router rule accordingly. If left empty, operator checks if table, transform params are set for the pool and if so, uses them. | [] | View examples | v5.4 | |
traefik_service_annotations | Additional annotations for traefik service | {} | v4.22 | ||
traefik_service_cors_headers | Optional key values pairs of CORS headers | {} | v5.2 | ||
traefik_service_custom_response_headers | Optional key value pairs of custom headers that will be applied to the response | {} | v5.2 | ||
traefik_service_type | The type of service to use for traefik, the entry point to the cluster. | public_lb | View examples | v4.20 | |
turbine_api_init_pools | If enabled, the turbine-api component initializes some pools. | False | v4.20 | ||
turbine_api_require_table_default_storage | If enabled, turbine-api will require tables to have their storage_map be populated with a default_storage_id. Useful when use of the cluster's default bucket should be discouraged. | False | v5.0 | ||
unified_auth | Use the same auth used with the API for all services. | True | v4.20 | ||
usagemeter_enabled | Whether or not the usage meter cron job should run. | True | v4.20 | ||
usagemeter_preserve | Duration to hang onto old, already-reported usagemeter data on local clusters. | 1440h | v4.23 | ||
usagemeter_query_timeout | Maximum time to wait for query against catalog to complete. | 4m | v4.20 | ||
usagemeter_reporting_table | Hydrolix table to send usage to (in project.table format). | metering_project.metering_table | v4.20 | ||
usagemeter_reporting_transform | Hydrolix transform name or UUID to use for usage reporting. | metering_transform | v4.20 | ||
usagemeter_reporting_url | URL to send usage data to. | https: //prometheus-us.trafficpeak.live/ingest | v4.20 | ||
usagemeter_request_timeout | Maximum time to wait for reporting HTTP request to complete. | 1m | v4.20 | ||
usagemeter_schedule | CRON schedule for usage meter cron job (defaults to every sixth of an hour). | /10 * * * | v4.20 | ||
use_crunchydata_postgres | Use a postgres managed by Crunchydata's postgres operator instead of the default dev mode postgres. | False | v4.20 | v5.2 | |
use_https_with_s3 | DEPRECATED: use db_bucket_url or db_bucket_http_enabled. | v4.20 | |||
use_hydrolix_dns_resolver | If true, use Hydrolix Dns Resolver. If false, use system resolver | True | v4.20 | ||
use_tls | DEPRECATED: inferred from hydrolix_url. | False | v4.20 | ||
user_acl_refresh_interval_secs | Frequency at which user acl permissions are refresed (in secs) | 30 | v4.20 | ||
user_token_expiration_secs | user token expiration period (in secs) | 1800 | v4.20 | ||
user_token_refresh_interval_secs | Frequency at which user tokens are refresed (in secs) | 240 | v4.20 | ||
vector_bucket | Bucket where Vector should save json format pod logs. | v4.20 | |||
vector_bucket_path | Prefix under which vector will save pod logs. | logs | v4.20 | ||
vector_extra_namespaces | List of additional namespaces that vector should scrape pod logs from | [] | View examples | v5.4 |
aws_credentials_method – examples
aws_credentials_method – examples
Example |
---|
static |
instance_profile |
db_bucket_credentials_method – examples
db_bucket_credentials_method – examples
Example |
---|
static |
ec2_profile |
web_identity |
db_bucket_region – examples
db_bucket_region – examples
Example |
---|
us-east-2 |
us-central1 |
db_bucket_type – examples
db_bucket_type – examples
Example |
---|
gs |
s3 |
db_bucket_url – examples
db_bucket_url – examples
grafana_config – default
grafana_config – default
Key | Value |
---|---|
admin_user | admin |
admin_email | admin@localhost |
allow_embedding | False |
db_user | grafana |
alert_eval_timeout | 30s |
smtp_enabled | False |
smtp_host | smtp.sendgrid.net:587 |
smtp_user | apikey |
rendering_timeout | 120s |
is_enterprise | False |
google_auth_enabled | False |
google_client_id | None |
inactive_timeout | 7d |
allow_sign_up | False |
hdx_anomaly_detection – default
hdx_anomaly_detection – default
{
"enabled": false,
"configs": {}
}
hdx_anomaly_detection – examples
hdx_anomaly_detection – examples
Example |
---|
|
{
"enabled": True,
"configs": {"tenant1": "ad-tenant1", "tenant2": ["ad-tenant2-a", "ad-tenant2-b"]},
} |
http_proxy – default
http_proxy – default
{
"enabled": false,
"port": 9444,
"log_debug": false,
"allow_ping": false,
"server": {
"read_timeout": "2m",
"write_timeout": "4m",
"idle_timeout": "8m"
},
"users": {
"max_execution_time": "2m"
},
"heartbeat": {
"interval": "5s",
"timeout": "3s",
"request": "/query?query=SELECT%201&hdx_query_output_format=TSV",
"response": "1\n"
},
"cache": {
"dir": "/tmp/http-proxy/cache",
"max_size": "150M",
"expire": "1m"
}
}
hydrolix_url – examples
hydrolix_url – examples
intake_head_catalog_spill_config – default
intake_head_catalog_spill_config – default
Key | Value |
---|---|
enabled | false |
max_concurrent_fetch | 1 |
fetch_lock_expire_duration | 10m |
max_concurrent_spill | 20 |
max_attempts_spill | 5 |
num_partitions | 10 |
empty_fetch_pause_duration | 30s |
intake_head_raw_data_spill_config – default
intake_head_raw_data_spill_config – default
Key | Value |
---|---|
enabled | false |
max_concurrent_fetch | 1 |
fetch_lock_expire_duration | 10m |
max_concurrent_spill | 20 |
max_attempts_spill | 5 |
num_partitions | 10 |
empty_fetch_pause_duration | 30s |
kubernetes_cloud – examples
kubernetes_cloud – examples
Example |
---|
aws |
gcp |
kubernetes_profile – examples
kubernetes_profile – examples
Example |
---|
gke |
eks |
lke |
metadata – examples
metadata – examples
{
"annotations": {
"example.com/owner": "hdx"
},
"labels": {
"env": "dev"
}
}
overrides – examples
overrides – examples
Example |
---|
|
postgres-sunday-gameday:
timezone: America/New_York
weekly: {"days": ["Sun"], "start": "16:00", "end": "22:00"},
patch: {"scale": {"postgres": {"replicas": 999}}},
winter-freeze:
window: {
"start": "1999-12-24T18:00:00Z",
"end": "2000-01-02T09:00:00Z",
}
patch: {"scale_off": "true", "celebrate": True},
nightly-shrink:
cron: {"expression": "0 2 * * 1-5", "duration": "4h"},
patch: {
"pools": {"hydrologs-intake-head": {"replicas": "0"}},
}
|
pg_ssl_mode – examples
pg_ssl_mode – examples
Example |
---|
disable |
require |
verify-ca |
verify-full |
pgbouncer_pool_mode – examples
pgbouncer_pool_mode – examples
Example |
---|
session |
statement |
postgrescluster_backup_retention_type – examples
postgrescluster_backup_retention_type – examples
Example |
---|
count |
time |
prometheus_ignored_apps – examples
prometheus_ignored_apps – examples
Example |
---|
batch-head |
stream-peer,vector |
query_peer_liveness_check_path – default
query_peer_liveness_check_path – default
?query=select%20count%28id%29%20from%20hdx.liveliness%20SETTINGS%20hdx_log_query=false%2Chdx_query_timerange_required=0
sql_transform_max_ast_elements – examples
sql_transform_max_ast_elements – examples
Example |
---|
100000 |
150000 |
sql_transform_max_expanded_ast_elements – examples
sql_transform_max_expanded_ast_elements – examples
Example |
---|
100000 |
150000 |
stream_load_balancer_algorithm – examples
stream_load_balancer_algorithm – examples
Options:
Value | Description |
---|---|
rr | Round robin (default): Each server takes a turn |
p2c | Power of Two Choices (P2C): Picks two servers and chooses the least busy |
P2C improves request distribution by randomly selecting two backend pods and routing traffic to the one with fewer active connections. This can reduce 429 errors during spikes and help run services at a lower replica count.
traefik_external_ips – examples
traefik_external_ips – examples
[
"192.168.1.5",
"192.16.1.4"
]
[
"172.16.0.8"
]
traefik_service_allowed_headers – examples
traefik_service_allowed_headers – examples
Example |
---|
x-hdx-table |
x-hdx-transform |
traefik_service_allowed_query_params – examples
traefik_service_allowed_query_params – examples
Example |
---|
table |
transform |
traefik_service_type – examples
traefik_service_type – examples
Example |
---|
public_lb |
private_lb |
node_port |
cluster_ip |
vector_extra_namespaces – examples
vector_extra_namespaces – examples
[
"kube-system",
"default"
]
Updated 11 days ago