For the various Cloud platforms Hydrolix deploys too there are "Tunables" that can be set to alter the behaviour of the cluster. AWS Cloudformation "Tunables" are contained within a file, it includes a number of the configurable settings used platform wide. Examples of these settings include settings for platform services such as queue time-outs, AWS Kinesis shard counts, SSH Authorized keys and S3 bucket permissions and more.

The Hydrolix Tunables file contains a number of the configurable settings used platform wide, it also includes settings for platform services such as queue time-outs, Kinesis shard counts, SSH Authorized keys and bucket permissions.

These settings can be updated via two methods:

It is suggested where large numbers of items are required to be updated (for example when enabling bucket access or authorized ssh keys), the updates be completed using the Tunables command.

The Tunables file is a TOML file with a list of name value pairs. A partial example is below:

$ hdxctl tunables get hdxcli-abcdef
#  autoingest_max_receive_count = 10

#  autoingest_queue_timeout = 200

#  aws_ssh_key_name = ""

#  batch_bucket_kms_arn = []

#  batch_peer_threads = 1

#  bucket_allowlist = []
bucket_allowlist = [ "bucket1", "bucket2", "bucket3",]

#  ec2_detailed_monitoring = true

#  enable_query_auth = false

#  enable_query_peer_hyperthreading = true

#  enable_turbine_monitor = true

#  import_max_receive_count = 1

#  import_queue_timeout = 43200

#  ip_allowlist = [ "111.2222.333.111/32", "222.111.333.222/32",]
ip_allowlist = [ "0.0.0.0/0",]

#  kafka_tls_ca = ""

#  kafka_tls_cert = ""

#  listing_max_receive_count = 1

#  listing_queue_timeout = 43200

Tunables Options

The Tunables options available for setting are as follows:

TunableDescriptionDefault
autoingest_max_receive_countThe number of times a message is delivered to the queue before being moved to the dead-letter queue. Recommended to be kept as default.10
autoingest_queue_timeoutspecify the maximum message retention period for the Autoingest Queue.200
aws_ssh_key_nameTo add a public key that you have stored on AWS, provide a the key’s AWS name as an argument. Note only one can be provided.none
batch_bucket_kms_arnAllow Hydrolix servers to decrypt a source bucket where a customer defined KMS key is required. Takes the ARNnone
batch_peer_threadsSpecify the number of vCPU’s a batch-peer should use for import jobs.1
bucket_allowlistEnables the architecture to access other buckets other than the default deployed bucket.
This is not additive, any update will overwrite previous configurations.
none
ec2_detailed_monitoringTurns off additional monitoring for Hydrolix EC2 components.True
enable_query_authEnable query authorisation for requests to the query end-point. Currently a place-holder and not in use.False
enable_query_peer_hyperthreadingEnable hyperthreading on the query peer.Disabled
enable_turbine_monitorAllow query components to monitor the Hydrolix query engine, restarting it if it hangs.True
import_max_receive_countThe number of times a message is delivered to the import queue before being moved to the dead-letter queue.1
import_queue_timeoutSpecify the time for an individual job to timeout on the SQS queue, in seconds. Recommended to be kept as default.43200
ip_allowlistSets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additive, any update will overwrite previous configurations.none
kafka_tls_caAllows the addition of a TLS Certificate Authority (CA) for mutual identification of Hydrolix Kafka ingest. PEM Formatnone
kafka_tls_certAllows the addition of a TLS Certificate for mutual identification of Hydrolix Kafka ingest. PEM formatnone
listing_max_receive_countThe number of times a message is delivered to the listing queue before being moved to the dead-letter queue.1
listing_queue_timeoutSpecify the time for an import job to timeout of the SQS queue, in seconds. Recommended to be kept as default.43200
merge_intervalspecify the interval for the Merge process to trigger. Recommended to be kept as default.1m
merge_max_receive_countThe number of times a message is delivered to the merge queue before being moved to the dead-letter queue.1
merge_queue_timeoutspecify the maximum message retention period for the Merge Queue, in seconds. Recommended to be kept as default.300 seconds
reaper_queue_timeoutspecify the maximum message retention period for the Reaper Queue, in seconds. Recommended to be kept as default.30 seconds
ssh_authorized_keysList of Authorized keys that are deployed to components for SSH accessnone
stream_shard_countThe number of shards AWS Kinesis is configured to use. This Kinesis stream is used between the stream-head and the stream-peers.2
tagTags to apply to this cluster, in the format TAG-NAME:TAG-VALUE. See also the further notes on tags.none
enable_grafana_cloudwatchEnable cloudwatch metrics within Grafana.False
superset_workersThe number of threads for each Superset web worker.10
superset_threadsSuperset web workers that are silent for more than this many seconds are killed and restarted.20
os_patch_dateWhen you deploy a new version it uses the latest patched version and stores that version, so that there are no unintentional patch updates. You can also specify a different date in which case you would get the latest patch older than that date. Default recommendation is to leave blank.none
superset_timeoutThe number of workers for handling Superset requests.60