Hydrolix Tunables List

A listing of Hydrolix tunables

A listing of HTN tunables used by Hydrolix. These tunables are set in the hydrolixcluster.yaml configuration file, under spec:.


Tunable NameDescriptionDefaultExamplesAddedRemoved
acme_enabledAutomatically generate and renew SSL certs for your hydrolix domain. Will override any existing kubernetes secret named traefik-tlsFalsev4.20
admin_emailThe email address of a person who will administer the Hydrolix cluster.v4.20
argus_fleet_tableHydrolix table to send fleet info to, in project.table format.fleet_inventory.fleet_tablev5.4
argus_fleet_transformHydrolix transform name or UUID for fleet reporting.fleet_transformv5.4
argus_fleet_urlURL to send fleet data to.https: //prometheus-us.trafficpeak.live/ingestv5.4
auth_http_read_timeout_msMaximum time to wait for a socket read for user-permission data from auth endpoint (turbine-api)2000v4.20
auth_http_response_timeout_msMaximum time to wait for receiving http headers from auth endpoint (turbine-api) in response to user permission requests2000v4.20
autoingest_unique_file_pathsEnable unique file paths from object store by ignoring duplicate paths.Falsev4.22
aws_credentials_methodDEPRECATED: use db_bucket_credentials_methodView examplesv4.20
aws_load_balancer_subnetsSubnets to assign to the load balancer of the traefik service when running in EKS["subnet-xxxx,mySubnet"]v4.20
aws_load_balancer_tagsAdditional tags to be added to the load balancer of the traefik service when running in EKS.["Environment=dev,Team=test"]v4.20
azure_blob_storage_accountThe storage account to use when accesing an Azure blob storage container.v4.20
basic_authA list of Hydrolix services that should be protected with basic auth when accessed over HTTP[]v4.20
batch_peer_heartbeat_periodHow frequently a batch peer should heartbeat any task it's working on as a duration string5mv4.20
bucketDEPRECATED: use db_bucket_urlv4.20
catalog_db_admin_dbThe default db of the admin user of the Postgres server where Hydrolix metadata is stored.turbinev4.20
catalog_db_admin_userThe admin user of the Postgres server where Hydrolix metadata is stored.turbinev4.20
catalog_db_hostThe Postgres server where Hydrolix metadata is stored.postgresv4.20
catalog_db_portThe Postgres server's port where Hydrolix metadata is stored.5432v4.21
catalog_intake_connectionsConnection pool settings for intake services that connect to the Postgres server where Hydrolix metadata is stored. Available options (1) 'max_lifetime' - the max duration that a connection can live before being recycled. (2) 'max_idle_time' - the max duration that a connection can be idle before being closed. (3) 'max' - the max number of connections that can be opened by each intake service that connects to the Postgres server. (4) 'min' - the minimum number of connections to keep open to the Postgres server. (5) 'check_writable' - if set to true, then when a connection is opened to the Postgres server, ensure the Postgres server the connection is for can handle writes.{"max_lifetime": "10m", "max_idle_time": "1m"}v4.20
clickhouse_http_portThe port to dedicate to the Clickhouse http interface.8088v4.20
client_idDEPRECATED: use hydrolix_name and db_bucket_urlv4.20
data_service_termination_grace_periodTermination grace period for most data services.120v4.20
data_visualization_toolsList of data visualization tools to deploy. Eg. Grafana, Kibana.[]v4.23
db_bucket_credentials_methodThe method Hydrolix uses to acquire credentials for connecting to cloud storage.web_identityView examplesv4.20
db_bucket_endpointThe endpoint url for S3 compatible object storage services. Not required if using AWS S3 or if db_bucket_url is provided.v4.20
db_bucket_nameThe name of the bucket you would like Hydrolix to store data in. Not required if db_bucket_url is provided.v4.20
db_bucket_regionNot required if it can be inferred from db_bucket_url.View examplesv4.20
db_bucket_typeThe object storage type of the bucket you would like Hydrolix to store data in. Not required if db_bucket_url is provided.View examplesv4.20
db_bucket_urlThe URL of the cloud storage bucket you would like Hydrolix to store data in.View examplesv4.20
db_bucket_use_httpsIf true use https when connecting to the cloud storage service. Inferred from db_bucket_url if possible.Truev4.20
decay_batch_sizeNumber of entries to fetch for each request to the catalog.5000v4.20
decay_enabledWhether or not the Decay Cron Job should run.Truev4.20
decay_max_deactivate_iterationsMaximum number of deactivation iterations to execute per table.v4.20
decay_max_reap_iterationsMaximum number of reap iterations to execute per table.v4.20
decay_reap_batch_sizeNumber of entries to fetch for each request when locating entries for reaping5000v4.20
decay_scheduleCRON schedule for Decay Cron Job0 0 * * *v4.20
default_query_poolA name for the default query pool.query-peerv4.20
disable_disk_cacheIf true, query peers will immediately delete partition metadata from disk after useFalsev4.20
disable_traefik_clickhouse_http_portIf true the load balancer will not forward to traefik on port
8088. This port provides a Clickhouse compatible query interface rooted at the root of the service rather than at a subpath.
Falsev4.20
disable_traefik_http_portIf true the load balancer will not forward to traefik on port
80. When tls is enabled, this port is only used to redirect to https. Otherwise this is the main way to access all services.
Falsev4.20
disable_traefik_https_portIf true the load balancer will not forward to traefik on port
443. Only relevant if tls is enabled
Falsev4.20
disable_traefik_mysql_portIf true the load balancer will not forward to traefik on the Clickhouse My SQL interface port. This is port 9004.Falsev5.0
disable_traefik_native_portIf true the load balancer will not forward to traefik on the Clickhouse native protocol port. This is port 9440 when TLS is enabled or 9000 if not.Falsev4.20
disable_vector_bucket_loggingPrevent vector from sending logs to the bucket.Falsev4.20
disable_vector_kafka_loggingPrevent vector from emitting logs to Redpanda.Falsev4.20
disk_cache_cull_start_percPercentage of cache disk space used before starting to remove files.75v4.20
disk_cache_cull_stop_percPercentage of cache disk space used before stopping removing files.65v4.20
disk_cache_entry_max_ttl_minutesMax TTL for a cache disk entry. It is the longest period of time for which the LRU disk cache can save an entry before it expires.360v4.20
disk_cache_redzone_start_percMinimum percentage of cache disk space used to be considered as redzone.90v4.20
dns_aws_max_resolution_attemptsMaximum number of attempts made by the Dns Resolver for aws and all s3 compatible storages in a given dns refresh cycle1v4.20
dns_aws_max_ttl_secsMax DNS TTL for aws and s3-compaitible storages. It is the longest period of time for which the DNS resolver can cache a DNS record before it expires and needs to be refreshed. max_ttl=0 means dns cache strictly respects the ttl from the dns query response0v4.20
dns_azure_max_resolution_attemptsMaximum number of attempts made by the Dns Resolver for azure storage in a given dns refresh cycle1v4.20
dns_azure_max_ttl_secsMax DNS TTL for azure storage. It is the longest period of time for which the DNS resolver can cache a DNS record before it expires and needs to be refreshed. max_ttl=0 means dns cache strictly respects the ttl from the dns query response0v4.20
dns_gcs_max_resolution_attemptsMaximum number of attempts made by the Dns Resolver for gcs storage in a given dns refresh cycle1v4.20
dns_gcs_max_ttl_secsMax DNS TTL for gcs storage. It is the longest period of time for which the DNS resolver can cache a DNS record before it expires and needs to be refreshed. max_ttl=0 means dns cache strictly respects the ttl from the dns query response0v4.20
dns_server_ipThe ip address of a DNS server used for performance critical purposes.v4.20
domainDEPRECATED: use hydrolix_url.v4.20
eks_product_codeEKS product code for use with Amazon Marketplace.6ae46hfauzadikp9f8npdbh9vv4.20
enable_password_complexity_policyIf set to true, the default password policy defined in hydrolixdocs will apply. These are: Minimum Length 8 characters Uppercase Characters: 1Lowercase Characters: 1Digits: 1Special Characters: 1Not Recently Used: the past 24 passwords Expire Password: 90 days Not Username Not EmailFalsev4.20
enable_query_authWhen enabled requests to the query service, url paths starting with /query will require authentication.Falsev4.20
enable_traefik_access_loggingIf set to true, Traefik will log all access requests. WARNING: This will produce a very high and potentially unmanageable amount of logsFalsev4.20
enable_traefik_hstsIf set to true, Traefik will enforce HSTS on all its connections WARNING: This may lead to hard-to-diagnose persistent SSL failures if there areany errors in SSL configuration, and cannot be turned off later.Falsev4.20
enable_vectorRun vector to send kubernetes pod logs to json files in a bucket and to the internal logs topic. Default inferred from the value of scale_off.v4.20
envEnvironment variables to set on all Kubernetes pods that are part of the Hydrolix cluster.{}v4.20
exp_backoff_additive_jitterTrue: (growthfactor)(1 + jitter), False: growthfactor(jitter)Truev4.20
exp_backoff_growth_factor_msEvery sleep will use this as multiplicative factor, ie 2^i * (growth_factor)ms50v4.20
force_container_user_rootSet the initial user for all containers to 0 (root).Falsev4.20
grafana_configGrafana configuration. NOTE: To enable Grafana deployment, include grafana in the data_visualization_tools tunable. admin_user: Grafana admin username. admin_email: Grafana admin user email. allow_embedding: Prevents embedding Grafana in frames to mitigate clickjacking risks. db_user: Grafana database username. alert_eval_timeout: Timeout for alert evaluation when fetching data from a source. smtp_enabled: Enables email server settings. Requires the GRAFANA_SMTP_PASSWORD secret. smtp_host: Email server host. smtp_user: Email server authentication username. rendering_timeout: Timeout for rendering reports (PDFs, embedded images, or CSV attachments). is_enterprise: Enables Grafana Enterprise. Requires the GRAFANA_LICENSE secret. google_auth_enabled: Enables Google OAuth authentication. Requires the GOOGLE_CLIENT_SECRET secret. google_client_id: Client ID of the Google Auth app. inactive_timeout: Maximum inactive duration before requiring login again. allow_sign_up: Controls Grafana user creation through OAuth. If false, only existing users can log in.View defaultv5.1
grafana_imageDefinition of Grafana image: tag to be used.grafana/grafana-enterprise: 11.5.0v5.1
hdx_anomaly_detectionAnomaly Detection configuration parameters. The configs key should be map of tenants to their associated config map(s). The config maps must be created outside of the hdx.spec.View defaultView examplesv5.3
hdx_nodeWhether or not enable hdx-node daemonset.Falsev4.21v5.1
hdx_node_configHDX-node YAML configuration.{}v5.1
hdx_node_enabledWhether or not enable hdx-node daemonset.Falsev5.1
hdx_node_envsEnv vars for hdx-node configuration. Accepted key-value pairs:

- LOG_LEVEL: (str) Defines the logging level for the application. Default: 'INFO'.
- DRY_RUN: (bool) Whether to enable dry-run mode for the node, performing actions without making changes. Default: True.
- NODE_HEALTHCHECK_PORT: (int) Port used for node health checks. Default: 9019.
- NODE_NAME: (str) Name of the node for identification. Default: 'unknown-node'.
- NAMESPACE: (str) Kubernetes namespace where the node operates. Default: 'default'.
- OBSERVATIONS_CONFIGMAP_NAME: (str) Name of the Config Map for node observations. Default: 'hdx-node-observations'.
- LEADER_CONFIGMAP_NAME: (str) Name of the Config Map used for leader election. Default: 'hdx-node-leader-election'.
- NODE_DOWN_CONFIRMATION_PERIOD_SECONDS: (int) Time in seconds before confirming a node is down. Default: 300.
- LEASE_DURATION_SECONDS: (int) Duration of the lease for leader election in seconds. Default: 120.
- RETRY_PERIOD_SECONDS: (int) Period in seconds to retry leader election actions. Default: 60.
- PROMETHEUS_SERVER_PORT: (int) Port used by the Prometheus server for scraping metrics. Default: 9009.
{}v4.21v5.1
hdx_query_max_memory_usage_percMaximum amount of memory to use for running a query on a single server as a percentage of the total available memory80v4.20
hdx_query_max_perc_before_external_group_byMaximum amount of memory to use for running a summary merge query as a percentage of the total available memory. Zero deactivates the restriction.0v5.2
hdx_traefik_auth_workersNumber of async workers gunicorn will create for services requests. Default - 11v5.1
hostDEPRECATED: use hydrolix_urlv4.20
http_connect_timeout_msMaximum time to wait for socket connection to cloud storage to complete300v4.20
http_portThe port to serve Hydrolix plain http on.v4.20
http_proxyHTTP-proxy configuraiton parameters.View defaultv5.2
http_read_timeout_msMaximum time to wait between a socket read and cloud storage having data ready to be read1000v4.20
http_response_timeout_msMaximum time to wait for receiving http headers to complete while reading from cloud storage1000v4.20
http_ssl_connect_timeout_msMaximum time to wait for ssl handshake during connection to cloud storage to complete1000v4.20
http_write_timeout_msMaximum time to wait before uploading partition to cloud is complete10000v4.20
https_portThe port to serve Hydrolix https on.v4.20
hydrolix_nameThe name you would like to assign your Hydrolix cluster. Will be the same as the namespace name if not specifiedv4.20
hydrolix_urlThe url you would like to use to access your Hydrolix cluster.View examplesv4.20
initial_exp_backoff_msSleep time starts from this value and exponentially grows with retry count0v4.20
intake_head_accept_data_timeoutConfigures the maximum duration that intake-head will wait for a request to be accepted into the partition creation pipeline. If the timeout is reached, the request will be rejected with a 429 status code response. If not configured or set to 0, intake-head pods will not timeout.0sv4.20
intake_head_catalog_spill_configProvides configuration of the spill functionality for catalog adds in intake-head whereby catalog adds are spilled to object storage when catalog interactions are slowed or fail on a particular intake-head pod. Supported keys are 'enabled', 'max_concurrent_fetch', 'fetch_lock_expire_duration', 'max_concurrent_spill', 'max_attempts_spill', 'num_partitions' and 'empty_fetch_pause_duration'View defaultv4.20
intake_head_index_backlog_enabledWhether to absorb received buckets in a backlog prior to indexing in intake-head to allow for more buffer for absorption in the face of spikes of traffic or throughput disruptions in indexing or uploading of partitions. If enabled, the newest data received will indexed ahead of older data when the backlog grows.Falsev4.20
intake_head_index_backlog_max_accept_batch_sizeControls the maximum number of buckets accepted from ingestion and added to the backlog at a time. Only applicable if intake_head_index_backlog_enabled is true.50v4.20
intake_head_index_backlog_max_mbControls the maximum size in MB that the indexing backlog on intake-head is allowed to grow before either dropping data or slowing new entries depending on the configured value of intake_head_index_backlog_trim_enabled. Only applicable if intake_head_index_backlog_enabled is true.256v4.20
intake_head_index_backlog_purge_concurrencyControls the number of workers used to purge buckets from the intake-head backlog when the max size is breached. Only applicable if intake_head_index_backlog_enabled is true.1v4.20
intake_head_max_outstanding_requestsConfigures the maximum number of requests that an intake-head pod will allow to be outstanding and in process before rejecting new requests with a 429 status code response. If not configured or set to 0, intake-head pods will never reject new requests.0v4.20
intake_head_raw_data_spill_configProvides configuration of the spill functionality for raw data in intake-head whereby ingested data is spilled to object storage when partition generation is slowed on a particular intake-head pod. Supported keys are 'enabled', 'max_concurrent_fetch', 'fetch_lock_expire_duration', 'max_concurrent_spill', 'max_attempts_spill', 'num_partitions', and 'empty_fetch_pause_duration'View defaultv4.20
io_perf_mappingsInternally used presets for io_perf_mode. Parsed as JSON Array(Array(Int))[[2097152, 256, 256], [6291456, 128, 128], [12582912, 64, 64]]v4.20
ip_allowlistA list of CIDR ranges that should be allowed to connect to the Hydrolix cluster load balancer.["127.0.0.1/32"]v4.20
issue_wildcard_certWhether to issue wildcard TLS certificate. NOTE: DNS Challenge will be used. Route53 credentals need to be provided in ROUTE53_AWS_ACCESS_KEY_ID and ROUTE53_AWS_SECRET_ACCESS_KEY via curated secret.Falsev5.3
job_purge_ageHow old a terminal job must be before it's deleted expressed as a duration string2160hv4.20
job_purge_enabledWhether or not the Job Purge Cron Job should run.Truev4.20
job_purge_scheduleCRON schedule for Job Purge Cron Job0 2 * * *v4.20
kafka_careful_modeFalsev4.20
kafka_tls_caA CA certificate used by the kafka_peer to authenticate Kafka servers it connects to.v4.20
kafka_tls_certThe PEM format certificate the kafka_peer will use to authenticate itself to a Kafka server.v4.20
kafka_tls_keyThe PEM format key the kafka_peer will use to authenticate itself to a Kafka server.v4.20
kibana_security_enabledEnable Kibana auth and RBAC via Elasticsearch static credentials. User credentials can be found in hdx-elastic-user Kubernetes secret.Falsev5.4
kinesis_coordinate_periodFor Kinesis sources, how often the coordination process runs which checks for the available shards and peers and distributes consuming amongst available peers10sv4.20
kinesis_coordinate_strategyThe strategy to use for coordinating Kinesis peers for a Kinesis source. Possible values are EXTERNAL_COORDINATOR or ZOOKEEPEREXTERNAL_COORDINATORv4.20
kubernetes_cloudDEPRECATED: use kubernetes_profile.View examplesv4.20
kubernetes_premium_storage_classThe storage class to use with persistent volumes created in Kubernetes for parts of a Hydrolix cluster where throughput is most critical.v4.20
kubernetes_profileUse default settings appropriate to this type of Kubernetes deployment.genericView examplesv4.20
kubernetes_storage_classThe storage class to use with persistent volumes created in Kubernetes as part of a Hydrolix cluster.v4.20
limit_cpuIf set, container cpu limits are set to match cpu requests in Kubernetes.Truev4.20
log_levelA dictionary to specify logging verbosity. Keys are service names with the special value of '*' controlling the default.{}v4.20
log_vacuum_concurrencyNumber of concurrent log deletion processes8v4.20v5.1
log_vacuum_dry_runIf true, Log Vacuum will only log it's intentions and take no action.Falsev4.20v5.1
log_vacuum_enabledWhether or not the Log Vacuum Cron Job should run.Truev4.20v5.1
log_vacuum_max_ageMaximum age of a log file before it is removed expressed as a duration string.168hv4.20v5.1
log_vacuum_scheduleCRON schedule for Log Vacuum Cron Job0 4 * * *v4.20v5.1
logs_http_remote_tableAn existing Hydrolix <project.table> where the data should land in remote cluster.hydro.logsv4.20
logs_http_remote_transformA transform schema for ingest in remote cluster.mega Transformv4.20
logs_http_tableAn existing Hydrolix <project.table> where the data should land.hydro.logsv4.20
logs_http_transformA transform schema for ingest.mega Transformv4.20
logs_kafka_bootstrap_serversA comma separated list of kafka bootstrap servers to send logs to.redpandav4.20
logs_kafka_topicA Kafka topic to send logs to.logsv4.20
logs_sink_local_urlThe full URI to make local HTTP request to.http: //stream-head: 8089/ingest/eventv4.20
logs_sink_remote_auth_enabledWhen enabled, remote HTTP will use basic auth from curated secret.Falsev4.20
logs_sink_remote_urlThe full URI to make remote HTTP request to.v4.20
logs_sink_typeType of logs sink.kafkav4.20
logs_topic_partition_countThe number of partitions to assign to the logs topic for stream processing.81v4.20
max_concurrent_queriesMax limit on total number of concurrently executed queries. Zero means unlimited.0v4.20
max_exp_backoff_secondsCap for exponentially back off sleep time20v4.20
max_http_retriesMaximum times to retry any query-related http requests that fail3v4.20
max_server_memory_usage_percMax % of total system memory that server can use/allocate for its operation.0v4.20
merge_candidate_concurrencyNumber of concurrent Merge Candidate construction queries to run.6v4.20
merge_cleanup_batch_sizeNumber of entries to fetch for each request to the catalog.5000v4.20
merge_cleanup_delayHow long before a merged partition should be deleted expressed as a duration string15mv4.20
merge_cleanup_enabledWhether or not the Merge Clean-up Cron Job should run.Truev4.20
merge_cleanup_scheduleCRON schedule for Merge Clean-up Cron Job/5 * * *v4.20
merge_controller_enabledWhether or not the next generation merge controller is enabled.Falsev5.0
merge_dispatch_frequencyHow often a slot should be checked for exeeding max_idle. Expressed as duration string (e.g. 5s)5sv4.20
merge_download_partitions_enabledWhether or not merge-peer should download partitions locally for processing.Falsev5.3
merge_first_era_frequencyHow often merge candidates should be constructed for the first era.10sv4.20
merge_head_batch_sizeNumber of records to pull from the catalog per request by the merge head.10000v4.20
merge_intervalThe time the merge process waits between checking for mergeable partitions.15sv4.20
merge_lock_bad_partitions_enabledWhether or not merge-peer should lock partitions which cannot be read by turbine.Falsev5.3
merge_max_candidatesNumber of candidates to produce per merge target each cycle.100v4.20
merge_max_partitionsNumber of partitions to be buffered awaiting candidate construction.10000v5.4
merge_max_partitions_per_candidateThe maximum number of partitions per merge candidate.100v4.20
merge_min_mbSize in megabytes of the smallest merge tier. All other merge tiers are multiples of this value.1024v4.20
merge_primary_window_widthSpecifies the interval used to further filter partition selection queries. Smaller values limit the number of records the database needs to produce, but can increase query count.1080hv4.20
merge_second_era_frequencyHow often merge candidates should be constructed for the second era.60sv4.20
merge_streaming_selectorWhether or not to use the Streaming Candidate SelectorTruev4.20
merge_third_era_frequencyHow often merge candidates should be constructed for the third era.60mv4.20
metadataCustom kubernetes labels and annotations to propagate to hydrolix workloads. Changing this value will trigger restarts for all services{}View examplesv5.2
monitor_ingestIf enabled, deploy a service to ingest a timestamp into the hydro.monitor table every secondFalsev4.20
monitor_ingest_request_timeoutThe number in seconds for HTTP timeout in HTTP POST from monitor_ingest1v4.21
monitor_ingest_retry_timeoutThe deadline for one submission by monitor ingest including all retries.1v4.21
monitor_ingest_timeoutThe number in seconds for HTTP timeout in HTTP POST from monitor_ingest1v4.20
mysql_portThe port to serve the Clickhouse My SQL interface on if applicable.9004v5.0
mysql_port_disable_tlsWhen True, Traefik will not use TLS configuration on My SQL TCP route.Truev5.0
native_portThe port to serve the Clickhouse plaintext native protocol on if applicable.9000v4.20
native_tls_portThe port to serve the Clickhouse TLS native protocol on if applicable.9440v4.20
oom_detectionConfiguration options for detecting indexing OOM scenarios and retry with smaller data sizes if possible for services that perform ingest. Outer keys are names of the ingest services. The supported services are 'intake-head', 'kafka-peer', 'kinesis-peer', and 'akamai-siem-peer'. Available keys under each service are 'k8s_oom_kill_detection_enabled', 'k8s_oom_kill_detection_max_attempts', 'circuit_break_oom_detection_enabled', and 'preemptive_splitting_enabled'v4.21
otel_endpointSend otlp data to the http server at this URL.v4.20
overcommitWhen true, removes all requests and limits
from Kubernetes containers. Useful when running on a single node Kubernetes cluster with constrained resources.
When set to requests, only turns off requests. Similarly, limits removes just the limits.
Not being set is the same as false. Note that removing either a memory or CPU limit or request from any container on a pod removes the Guaranteed quality of service class from that pod.
Falsev4.20
overridesTemp overrides that patch HDX spec while their schedule is active. Override key should be a unique name and value must contain:
timezone (string, _required): IANA TZ to evaluate the schedule; defaults to UTC. Offsets (Z, -05: 00, …) are ignored to avoid ambiguity.
_ Exactly ONE schedule block:
– weekly: { days: [Sun.. Sat], start: "HH: MM", end: "HH: MM" }
– window: { start: ISO-8601 datetime, end: ISO-8601 datetime }
– cron: { expression: 5-field cron, duration: "4h" | "30m" }

- patch (dict, required): partial HDX spec to apply while active.Multiple overrides can be active at the same time; later items in the list win when key conflicts. The HDX custom resource itself is never mutated. Overrides live only in operator memory.
The operator writes the list of currently active override names to .status.active Overrides.
{}View examplesv5.4
ownerDEPRECATED: this was previously used internally by Hydrolix.v4.20
partition_cleaner_dry_runIf true, Partition Cleaner will only log it's intentions and take no actionTruev4.21
partition_cleaner_enabledWhether or not the partition cleaner cronjob should run.Falsev4.21v5.0
partition_cleaner_grace_periodMinimum age of a partition before it is considered for deactivation or deletion expressed as a duration string.24hv4.22
partition_cleaner_scheduleCrontab style schedule for when partition cleaner should run.0 0 * * *v4.21v5.0
partition_vacuum_batch_sizeNumber of entries to fetch from partition providers on each request.10000v4.20v4.23
partition_vacuum_concurrencyNumber of concurrent vacuum operations to run. Each vacuum operation covers a single table.5v4.20v4.23
partition_vacuum_dry_runIf true, Partition Vacuum will only log it's intentions and take no actionTruev4.20v4.23
partition_vacuum_enabledWhether or not the Partition Vacuum Cron Job should run.Falsev4.20v4.23
partition_vacuum_grace_periodMinimum age of a partition before it is considered for deactivation or deletion expressed as a duration string.24hv4.20v4.23
partition_vacuum_scheduleCRON schedule for Partition Vacuum Cron Job0 1 * * *v4.20v4.23
password_expiration_policyNumber of days to expire passwordv4.20
pg_ssl_modeDetermines whether and with what priority an SSL connection will be negotiated when connecting to a Postgres server. See https: //bit.ly/3U9ao8O.disableView examplesv4.20
pgbouncer_client_ca_secretThe secret that holds the ca certificatescatalog-cav5.4
pgbouncer_client_tls_secretThe secret that holds the client tls certificatescatalog-serverv5.4
pgbouncer_enabledThe tunable installs pgbouncer.Falsev5.4
pgbouncer_max_client_connMaximum number of client connections allowed.1000v5.4
pgbouncer_metrics_portThe port on which pgbouncer metrics can be scraped.9127v5.4
pgbouncer_pool_modeThe pooling method to use for connecting to backend postgressessionView examplesv5.4
pgbouncer_pool_sizeNumber of server connections to allow per user/database pair.20v5.4
pgbouncer_portThe port on which pgbouncer starts.6432v5.4
poolsA list of dictionaries describing pools to deploy as part of the Hydrolix cluster.v4.20
postgrescluster_backup_retentionDepends on postgrescluster_backup_retention_type. Either the count, or the number of days to retain full postgres backups before expiring them. All differential and incremental backups that depend on full backups are also expired. Only considered when use_crunchydata_postgres is true1v4.20v5.2
postgrescluster_backup_retention_typeAllowed values are 'count' (default) and 'time'. If set to 'count', then postgrescluster_backup_retention represents the number of full backups to retain before expiring them. If set to 'time', then postgrescluster_backup_retention represents a number of days for retention. Only considered when use_crunchydata_postgres is true.countView examplesv4.20v5.2
postgrescluster_backup_schedulesBackup schedules for postgres. Keys are the type of backup to perform, either 'full', 'differential' or 'incremental'. Values are cron expressions representing the schedule for that backup type. Only considered when use_crunchydata_postgres is true.{"full": "0 16 _ 0", "differential": "0 16 _ 1-6"}v4.20v5.2
postgrescluster_metrics_enabledIf true, collect prometheus metrics from the postgres cluster. Changing this value will trigger a rolling restart of the postgres pods. Only considered when use_crunchydata_postgres is trueFalse[]v4.20v5.2
postgrescluster_settingsSettings to apply to postgres. Changes to this value will be automatically reloaded without validation. Changes to certain parameters will cause postgres to restart{}[]v4.20v5.2
prometheus_curated_configmapCustom curated prometheus configmap that will be mounted onto the prometheus pod.[]v5.0
prometheus_enabledThis tunable controls prometheus installationTruev5.3
prometheus_ignored_appsA comma delimited list of app labels to ignore when determining scrape targets for prometheusView examplesv5.2
prometheus_label_value_length_limitIf a label value is larger than the value configured, Prometheus will discard the entire scrape.512[]v4.20
prometheus_namespaceThe namespace where kube-prometheus service is running which is used when prometheus operator is used instead of the defacto prometheus installed with hydrolix.v5.3
prometheus_operator_installedThis tunable indicates if prometheus-operator is installed. It does not perform installation if the flag is set to FalseFalsev5.3
prometheus_remote_write_urlA URL you wish to use to configure Prometheus's remote-write functionality.[]v4.20
prometheus_remote_write_usernameThe username for Prometheus to use with basic auth to connect to a remote-write endpoint. Ignored if prometheus_remote_write_url is not set.hdx[]v4.20
prometheus_retention_ratioThe amount of the volume to reserve for prometheus data. Example: 0.70.7v4.20
prometheus_retention_sizeThe maximum number of bytes of prometheus data to retain. Overrides prometheus_retention_ratio. Units supported: B, KB, MB, GB, TB, PB, EBv4.20
prometheus_retention_timeWhen to remove old prometheus data. Example: 15dv4.20
prometheus_scrape_intervalHow frequently to scrape targets by default.15s[]v4.20
prometheus_service_nameThe kube-prometheus service name which is used when prometheus operator is used instead of the defacto prometheus installed with hydrolix.v5.3
prometheus_service_portThe kube-prometheus service port which is used when prometheus operator is used instead of the defacto prometheus installed with hydrolix.9090v5.3
prometheus_servicemonitor_selectorPrometheus custom resource(CR) uses this tunable to identify the servicemonitors to be scraped for metrics{}{"hydrolix": "true"}v5.3
prune_locks_enabledWhether or not the Prune Locks Cron Job should run.Truev4.20
prune_locks_grace_periodMinimum age of a lock before it is considered for removal expressed as a duration string.24hv4.20
prune_locks_scheduleCRON schedule for Prune Locks Cron Job30 0 * * *v4.20
query_peer_liveness_check_pathThe http path used to configure a kubernetes liveness check for query-peers. Set to 'none' to disable.View defaultv4.20
query_peer_liveness_failure_thresholdHow many times query liveness check can fail.5v4.20
query_peer_liveness_initial_delayTime in seconds to wait before starting query liveness checks.300v4.23
query_peer_liveness_period_secondsHow often should query liveness check run, in seconds.60v4.20
query_peer_liveness_probe_timeoutNumber of seconds after which the liveness probe times out10v4.23
query_readiness_initial_delayTime in seconds to wait before starting query readiness checks.0v4.20
quesma_configQuesma config for Hydrolix data source parameters.{"project": "hydro", "table": "logs"}v4.23
refresh_job_statuses_enabledWhether or not the Refresh Job Statuses Cron Job should run.Truev4.20
refresh_job_statuses_scheduleCRON schedule for Refresh Job Statuses Cron Job***v4.20
registryA docker registry to pull Hydrolix containers from.v4.20
rejects_vacuum_dry_runIf enabled, the Rejects Vacuum Cron Job will not delete files, but instead log its intentions.Falsev4.20v5.1
rejects_vacuum_enabledWhether or not the Rejects Vacuum Cron Job should run.Truev4.20v5.1
rejects_vacuum_max_ageHow old a rejects file should be before deleted, expressed as a duration string (e.g. 1h5m4s}.168hv4.20v5.1
rejects_vacuum_scheduleCRON schedule for Reject Vacuum Cron Job0 0 * * *v4.20v5.1
rollout_strategy_max_surgeThis tunable configures the number of pods (represented as percentage) that can be created above the desired amount of pods during deployment rollout update. Default: 25%25v4.23
rollout_strategy_max_unavailableThis tunable ensures the number of pods (represented as integer) that can be unavailable during deployment rollout update. Default: 00v4.23
sample_data_urlThe storage bucket url to use to load sample data.v4.20
scaleA list of dictionaries describing overrides for scale related configuration for Hydrolix services.v4.20
scale_minWhen true, similar to scale_off but keeps API, UI and their dependencies running.Falsev5.3
scale_offWhen true, override all deployment and statefulset replica counts with a value of 0 and disable vector.Falsev4.20
scale_profileSelects from a set of predefined defaults for scaleevalv4.20
sdk_timeout_secHow many seconds the Merge SDK should be given to run before it is killed.300v4.20
siem_backoff_durationBackoff duration when SIEM limit not hit, for politeness.1sv4.20
silence_linode_alertsIf true will run a daemonset that turns off Linode alerts for LKE nodes.Falsev4.20
skip_init_turbine_apiSkips running database migrations in the init-turbine-api job. Set to true when running multiple clusters with a shared databaseFalsev4.20
spill_locks_cleanup_enabledWhether or not the Spill Locks Clean-up Cron Job should run.Truev4.20v5.2
spill_locks_cleanup_scheduleCRON schedule for Spill Locks Clean-up Cron Job0 12 * * *v4.20v5.2
sql_transform_max_ast_elementsThe number of AST elements an SQL transform can contain. This limits the maximum complexity of a SQL tranform.View examplesv4.20
sql_transform_max_expanded_ast_elementsThe number of expanded AST elements an SQL transform can contain. This limits the maximum complexity of a SQL tranform.View examplesv4.20
stale_job_monitor_batch_sizeHow many jobs to probe in a single request.300v4.20
stale_job_monitor_enabledWhether or not the Statel Job Monitor Cron Job should run.Truev4.20
stale_job_monitor_limitHow many jobs in total Stale Job will process per cycle.3000v4.20
stale_job_monitor_scheduleCRON schedule for Stale Job Monitor/5 * * *v4.20
str_dict_enabledEnable/disable multi-threaded string dictionary decoding.Truev4.20
str_dict_min_dict_sizeControls the number of entries in each string dictionary block.32768v4.20
str_dict_nr_threadsSets the maximum number of concurrent v CPU used for decoding.8v4.20
stream_concurrency_limitThe number of concurrent stream requests per cpu allocated across all pods beyond which traefik will return 429 busy error responses. If not set or set to null no limit is enforced.v4.20
stream_load_balancer_algorithmControls how Traefik balances streaming traffic across intake-head and stream-head pods.rrView examplesv4.20
stream_partition_blockThe number of partitions to use on a non-default redpanda stream topic per TB/day of usage.6v4.20
stream_partition_countThe number of partitions to use on the default redpanda topic for stream service.50v4.20
stream_replication_factorThe replication factor for the internal Redpanda topic used by the Stream service it must always be less than the number of Redpanda replicas. If it is not, the configuration will not change.3v4.20
targetingA dictionary to pass targeting related Kubernetes settings to resources according to what Hydrolix servivce they are part of.{}v4.20
task_monitor_enabledWhether or not the Task Monitor Cron Job should run.Truev4.20
task_monitor_heartbeat_timeoutHow old a tasks heartbeat should be (in seconds) before it is timed out.600v4.20
task_monitor_scheduleCRON schedule for Task Montior/2 * * *v4.20
task_monitor_start_timeoutHow old a ready task should be (in seconds) before it is considered lost and timed out.21600v4.20
terminate_tls_at_lbThis flag controls if traefik accepts http or https traffic from load balancer. This is required if TLS termination needs to be done at the load balancer and plain traffic is sent to traefik backendFalsev5.4
traefik_external_ipsTraffic that ingresses into the cluster with one of these IP's gets directed to the traefik service. Useful in particular when deploying all on one node.View examplesv4.20
traefik_hsts_expire_timeExpiration time for HSTS caching in seconds.315360000v4.20
traefik_keep_alive_max_timeThe number of seconds a client HTTTP connection can be reused before recieving a Connection: close response from the server. Zero means no limit.26v4.20
traefik_service_allowed_headersHeader keys used in intake-head route configuration for pools. The entries should match the keys being passed in the pool annotations. The traefik daemon checks if pool annotations are in this list and if so crafts the traefik router rule accordingly. If left empty, operator checks if x-hdx-table, x-hdx-transform annotations are set for the pool and if so, uses them.[]View examplesv5.4
traefik_service_allowed_query_paramsQuery params used in intake-head route configuration for pools. The entries should match the keys being passed in the pool query_params. The traefik daemon checks if pool query_params are in this list and if so crafts the traefik router rule accordingly. If left empty, operator checks if table, transform params are set for the pool and if so, uses them.[]View examplesv5.4
traefik_service_annotationsAdditional annotations for traefik service{}v4.22
traefik_service_cors_headersOptional key values pairs of CORS headers{}v5.2
traefik_service_custom_response_headersOptional key value pairs of custom headers that will be applied to the response{}v5.2
traefik_service_typeThe type of service to use for traefik, the entry point to the cluster.public_lbView examplesv4.20
turbine_api_init_poolsIf enabled, the turbine-api component initializes some pools.Falsev4.20
turbine_api_require_table_default_storageIf enabled, turbine-api will require tables to have their storage_map be populated with a default_storage_id. Useful when use of the cluster's default bucket should be discouraged.Falsev5.0
unified_authUse the same auth used with the API for all services.Truev4.20
usagemeter_enabledWhether or not the usage meter cron job should run.Truev4.20
usagemeter_preserveDuration to hang onto old, already-reported usagemeter data on local clusters.1440hv4.23
usagemeter_query_timeoutMaximum time to wait for query against catalog to complete.4mv4.20
usagemeter_reporting_tableHydrolix table to send usage to (in project.table format).metering_project.metering_tablev4.20
usagemeter_reporting_transformHydrolix transform name or UUID to use for usage reporting.metering_transformv4.20
usagemeter_reporting_urlURL to send usage data to.https: //prometheus-us.trafficpeak.live/ingestv4.20
usagemeter_request_timeoutMaximum time to wait for reporting HTTP request to complete.1mv4.20
usagemeter_scheduleCRON schedule for usage meter cron job (defaults to every sixth of an hour)./10 * * *v4.20
use_crunchydata_postgresUse a postgres managed by Crunchydata's postgres operator instead of the default dev mode postgres.Falsev4.20v5.2
use_https_with_s3DEPRECATED: use db_bucket_url or db_bucket_http_enabled.v4.20
use_hydrolix_dns_resolverIf true, use Hydrolix Dns Resolver. If false, use system resolverTruev4.20
use_tlsDEPRECATED: inferred from hydrolix_url.Falsev4.20
user_acl_refresh_interval_secsFrequency at which user acl permissions are refresed (in secs)30v4.20
user_token_expiration_secsuser token expiration period (in secs)1800v4.20
user_token_refresh_interval_secsFrequency at which user tokens are refresed (in secs)240v4.20
vector_bucketBucket where Vector should save json format pod logs.v4.20
vector_bucket_pathPrefix under which vector will save pod logs.logsv4.20
vector_extra_namespacesList of additional namespaces that vector should scrape pod logs from[]View examplesv5.4

aws_credentials_method – examples

aws_credentials_method – examples
Example
static
instance_profile

db_bucket_credentials_method – examples

db_bucket_credentials_method – examples
Example
static
ec2_profile
web_identity

db_bucket_region – examples

db_bucket_region – examples
Example
us-east-2
us-central1

db_bucket_type – examples

db_bucket_type – examples
Example
gs
s3

db_bucket_url – examples

db_bucket_url – examples

grafana_config – default

grafana_config – default
KeyValue
admin_useradmin
admin_emailadmin@localhost
allow_embeddingFalse
db_usergrafana
alert_eval_timeout30s
smtp_enabledFalse
smtp_hostsmtp.sendgrid.net:587
smtp_userapikey
rendering_timeout120s
is_enterpriseFalse
google_auth_enabledFalse
google_client_idNone
inactive_timeout7d
allow_sign_upFalse

hdx_anomaly_detection – default

hdx_anomaly_detection – default
{
  "enabled": false,
  "configs": {}
}

hdx_anomaly_detection – examples

hdx_anomaly_detection – examples
Example

|
{
"enabled": True,
"configs": {"tenant1": "ad-tenant1", "tenant2": ["ad-tenant2-a", "ad-tenant2-b"]},
} |

http_proxy – default

http_proxy – default
{
  "enabled": false,
  "port": 9444,
  "log_debug": false,
  "allow_ping": false,
  "server": {
    "read_timeout": "2m",
    "write_timeout": "4m",
    "idle_timeout": "8m"
  },
  "users": {
    "max_execution_time": "2m"
  },
  "heartbeat": {
    "interval": "5s",
    "timeout": "3s",
    "request": "/query?query=SELECT%201&hdx_query_output_format=TSV",
    "response": "1\n"
  },
  "cache": {
    "dir": "/tmp/http-proxy/cache",
    "max_size": "150M",
    "expire": "1m"
  }
}

hydrolix_url – examples

hydrolix_url – examples

intake_head_catalog_spill_config – default

intake_head_catalog_spill_config – default
KeyValue
enabledfalse
max_concurrent_fetch1
fetch_lock_expire_duration10m
max_concurrent_spill20
max_attempts_spill5
num_partitions10
empty_fetch_pause_duration30s

intake_head_raw_data_spill_config – default

intake_head_raw_data_spill_config – default
KeyValue
enabledfalse
max_concurrent_fetch1
fetch_lock_expire_duration10m
max_concurrent_spill20
max_attempts_spill5
num_partitions10
empty_fetch_pause_duration30s

kubernetes_cloud – examples

kubernetes_cloud – examples
Example
aws
gcp

kubernetes_profile – examples

kubernetes_profile – examples
Example
gke
eks
lke

metadata – examples

metadata – examples
{
  "annotations": {
    "example.com/owner": "hdx"
  },
  "labels": {
    "env": "dev"
  }
}

overrides – examples

overrides – examples
Example

|
postgres-sunday-gameday:
timezone: America/New_York
weekly: {"days": ["Sun"], "start": "16:00", "end": "22:00"},
patch: {"scale": {"postgres": {"replicas": 999}}},
winter-freeze:
window: {
"start": "1999-12-24T18:00:00Z",
"end": "2000-01-02T09:00:00Z",
}
patch: {"scale_off": "true", "celebrate": True},
nightly-shrink:
cron: {"expression": "0 2 * * 1-5", "duration": "4h"},
patch: {
"pools": {"hydrologs-intake-head": {"replicas": "0"}},
}
|

pg_ssl_mode – examples

pg_ssl_mode – examples
Example
disable
require
verify-ca
verify-full

pgbouncer_pool_mode – examples

pgbouncer_pool_mode – examples
Example
session
statement

postgrescluster_backup_retention_type – examples

postgrescluster_backup_retention_type – examples
Example
count
time

prometheus_ignored_apps – examples

prometheus_ignored_apps – examples
Example
batch-head
stream-peer,vector

query_peer_liveness_check_path – default

query_peer_liveness_check_path – default
?query=select%20count%28id%29%20from%20hdx.liveliness%20SETTINGS%20hdx_log_query=false%2Chdx_query_timerange_required=0

sql_transform_max_ast_elements – examples

sql_transform_max_ast_elements – examples
Example
100000
150000

sql_transform_max_expanded_ast_elements – examples

sql_transform_max_expanded_ast_elements – examples
Example
100000
150000

stream_load_balancer_algorithm – examples

stream_load_balancer_algorithm – examples

Options:

ValueDescription
rrRound robin (default): Each server takes a turn
p2cPower of Two Choices (P2C): Picks two servers and chooses the least busy

P2C improves request distribution by randomly selecting two backend pods and routing traffic to the one with fewer active connections. This can reduce 429 errors during spikes and help run services at a lower replica count.

traefik_external_ips – examples

traefik_external_ips – examples
[
  "192.168.1.5",
  "192.16.1.4"
]
[
  "172.16.0.8"
]

traefik_service_allowed_headers – examples

traefik_service_allowed_headers – examples
Example
x-hdx-table
x-hdx-transform

traefik_service_allowed_query_params – examples

traefik_service_allowed_query_params – examples
Example
table
transform

traefik_service_type – examples

traefik_service_type – examples
Example
public_lb
private_lb
node_port
cluster_ip

vector_extra_namespaces – examples

vector_extra_namespaces – examples
[
  "kube-system",
  "default"
]

What’s Next

Learn more about Hydrolix tunables