Tunables
For the various Cloud platforms Hydrolix deploys too there are "Tunables" that can be set to alter the behaviour of the cluster. AWS Cloudformation "Tunables" are contained within a file, it includes a number of the configurable settings used platform wide. Examples of these settings include settings for platform services such as queue time-outs, AWS Kinesis shard counts, SSH Authorized keys and S3 bucket permissions and more.
The Hydrolix Tunables file contains a number of the configurable settings used platform wide, it also includes settings for platform services such as queue time-outs, Kinesis shard counts, SSH Authorized keys and bucket permissions.
These settings can be updated via two methods:
-
Using the hdxctl command update
-
Using the hdxctl tunables command.
It is suggested where large numbers of items are required to be updated (for example when enabling bucket access or authorized ssh keys), the updates be completed using the Tunables command.
The Tunables file is a TOML file with a list of name value pairs. A partial example is below:
$ hdxctl tunables get hdxcli-abcdef
# autoingest_max_receive_count = 10
# autoingest_queue_timeout = 200
# aws_ssh_key_name = ""
# batch_bucket_kms_arn = []
# batch_peer_threads = 1
# bucket_allowlist = []
bucket_allowlist = [ "bucket1", "bucket2", "bucket3",]
# ec2_detailed_monitoring = true
# enable_query_auth = false
# enable_query_peer_hyperthreading = true
# enable_turbine_monitor = true
# import_max_receive_count = 1
# import_queue_timeout = 43200
# ip_allowlist = [ "111.2222.333.111/32", "222.111.333.222/32",]
ip_allowlist = [ "0.0.0.0/0",]
# kafka_tls_ca = ""
# kafka_tls_cert = ""
# listing_max_receive_count = 1
# listing_queue_timeout = 43200
Tunables Options
The Tunables options available for setting are as follows:
Tunable | Description | Default |
---|---|---|
autoingest_max_receive_count | The number of times a message is delivered to the queue before being moved to the dead-letter queue. Recommended to be kept as default. | 10 |
autoingest_queue_timeout | specify the maximum message retention period for the Autoingest Queue. | 200 |
aws_ssh_key_name | To add a public key that you have stored on AWS, provide a the key’s AWS name as an argument. Note only one can be provided. | none |
batch_bucket_kms_arn | Allow Hydrolix servers to decrypt a source bucket where a customer defined KMS key is required. Takes the ARN | none |
batch_peer_threads | Specify the number of vCPU’s a batch-peer should use for import jobs. | 1 |
bucket_allowlist | Enables the architecture to access other buckets other than the default deployed bucket. This is not additive, any update will overwrite previous configurations. | none |
ec2_detailed_monitoring | Turns off additional monitoring for Hydrolix EC2 components. | True |
enable_query_auth | Enable query authorisation for requests to the query end-point. Currently a place-holder and not in use. | False |
enable_query_peer_hyperthreading | Enable hyperthreading on the query peer. | Disabled |
enable_turbine_monitor | Allow query components to monitor the Hydrolix query engine, restarting it if it hangs. | True |
import_max_receive_count | The number of times a message is delivered to the import queue before being moved to the dead-letter queue. | 1 |
import_queue_timeout | Specify the time for an individual job to timeout on the SQS queue, in seconds. Recommended to be kept as default. | 43200 |
ip_allowlist | Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additive, any update will overwrite previous configurations. | none |
kafka_tls_ca | Allows the addition of a TLS Certificate Authority (CA) for mutual identification of Hydrolix Kafka ingest. PEM Format | none |
kafka_tls_cert | Allows the addition of a TLS Certificate for mutual identification of Hydrolix Kafka ingest. PEM format | none |
listing_max_receive_count | The number of times a message is delivered to the listing queue before being moved to the dead-letter queue. | 1 |
listing_queue_timeout | Specify the time for an import job to timeout of the SQS queue, in seconds. Recommended to be kept as default. | 43200 |
merge_interval | specify the interval for the Merge process to trigger. Recommended to be kept as default. | 1m |
merge_max_receive_count | The number of times a message is delivered to the merge queue before being moved to the dead-letter queue. | 1 |
merge_queue_timeout | specify the maximum message retention period for the Merge Queue, in seconds. Recommended to be kept as default. | 300 seconds |
reaper_queue_timeout | specify the maximum message retention period for the Reaper Queue, in seconds. Recommended to be kept as default. | 30 seconds |
ssh_authorized_keys | List of Authorized keys that are deployed to components for SSH access | none |
stream_shard_count | The number of shards AWS Kinesis is configured to use. This Kinesis stream is used between the stream-head and the stream-peers. | 2 |
tag | Tags to apply to this cluster, in the format TAG-NAME:TAG-VALUE. See also the further notes on tags. | none |
enable_grafana_cloudwatch | Enable cloudwatch metrics within Grafana. | False |
superset_workers | The number of threads for each Superset web worker. | 10 |
superset_threads | Superset web workers that are silent for more than this many seconds are killed and restarted. | 20 |
os_patch_date | When you deploy a new version it uses the latest patched version and stores that version, so that there are no unintentional patch updates. You can also specify a different date in which case you would get the latest patch older than that date. Default recommendation is to leave blank. | none |
superset_timeout | The number of workers for handling Superset requests. | 60 |
Updated 3 months ago