AWS Platform Tunables

The Hydrolix Tunables file contains a number of the configurable settings used platform wide, it also includes settings for platform services such as queue time-outs, Kinesis shard counts, SSH Authorized keys and bucket permissions.

These settings can be updated via two methods:

It is suggested where large numbers of items are required to be updated (for example when enabling bucket access or authorized ssh keys), the updates be completed using the Tunables command.

The Tunables file is a TOML file with a list of name value pairs. A partial example is below:

$ hdxctl tunables get hdxcli-ekmeho6e
#  autoingest_max_receive_count = 10

#  autoingest_queue_timeout = 200

#  aws_ssh_key_name = ""

#  batch_bucket_kms_arn = []

#  batch_peer_threads = 1

#  bucket_allowlist = []
bucket_allowlist = [ "bucket1", "bucket2", "bucket3",]

#  ec2_detailed_monitoring = true

#  enable_query_auth = false

#  enable_query_peer_hyperthreading = true

#  enable_turbine_monitor = true

#  import_max_receive_count = 1

#  import_queue_timeout = 43200

#  ip_allowlist = [ "111.2222.333.111/32", "222.111.333.222/32",]
ip_allowlist = [ "0.0.0.0/0",]

#  kafka_tls_ca = ""

#  kafka_tls_cert = ""

#  listing_max_receive_count = 1

#  listing_queue_timeout = 43200

Tunables Options

The Tunables options available for setting are as follows:

Tunable

Description

Default

autoingest_max_receive_count

The number of times a message is delivered to the queue before being moved to the dead-letter queue. Recommended to be kept as default.

10

autoingest_queue_timeout

specify the maximum message retention period for the Autoingest Queue.

200

aws_ssh_key_name

To add a public key that you have stored on AWS, provide a the key’s AWS name as an argument. Note only one can be provided.

none

batch_bucket_kms_arn

Allow Hydrolix servers to decrypt a source bucket where a customer defined KMS key is required. Takes the ARN

none

batch_peer_threads

Specify the number of vCPU’s a batch-peer should use for import jobs.

1

bucket_allowlist

Enables the architecture to access other buckets other than the default deployed bucket.
This is not additive, any update will overwrite previous configurations.

none

ec2_detailed_monitoring

Turns off additional monitoring for Hydrolix EC2 components.

True

enable_query_auth

Enable query authorisation for requests to the query end-point. Currently a place-holder and not in use.

False

enable_query_peer_hyperthreading

Enable hyperthreading on the query peer.

Disabled

enable_turbine_monitor

Allow query components to monitor the Hydrolix query engine, restarting it if it hangs.

True

import_max_receive_count

The number of times a message is delivered to the import queue before being moved to the dead-letter queue.

1

import_queue_timeout

Specify the time for an individual job to timeout on the SQS queue, in seconds. Recommended to be kept as default.

43200

ip_allowlist

Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additive, any update will overwrite previous configurations.

none

kafka_tls_ca

Allows the addition of a TLS Certificate Authority (CA) for mutual identification of Hydrolix Kafka ingest. PEM Format

none

kafka_tls_cert

Allows the addition of a TLS Certificate for mutual identification of Hydrolix Kafka ingest. PEM format

none

listing_max_receive_count

The number of times a message is delivered to the listing queue before being moved to the dead-letter queue.

1

listing_queue_timeout

Specify the time for an import job to timeout of the SQS queue, in seconds. Recommended to be kept as default.

43200

merge_interval

specify the interval for the Merge process to trigger. Recommended to be kept as default.

1m

merge_max_receive_count

The number of times a message is delivered to the merge queue before being moved to the dead-letter queue.

1

merge_queue_timeout

specify the maximum message retention period for the Merge Queue, in seconds. Recommended to be kept as default.

300 seconds

reaper_queue_timeout

specify the maximum message retention period for the Reaper Queue, in seconds. Recommended to be kept as default.

30 seconds

ssh_authorized_keys

List of Authorized keys that are deployed to components for SSH access

none

stream_shard_count

The number of shards AWS Kinesis is configured to use. This Kinesis stream is used between the stream-head and the stream-peers.

2

tag

Tags to apply to this cluster, in the format TAG-NAME:TAG-VALUE. See also the further notes on tags.

none

enable_grafana_cloudwatch

Enable cloudwatch metrics within Grafana.

False

superset_workers

The number of threads for each Superset web worker.

10

superset_threads

Superset web workers that are silent for more than this many seconds are killed and restarted.

20

os_patch_date

When you deploy a new version it uses the latest patched version and stores that version, so that there are no unintentional patch updates. You can also specify a different date in which case you would get the latest patch older than that date. Default recommendation is to leave blank.

none

superset_timeout

The number of workers for handling Superset requests.

60


Did this page help you?