Configuration with HDXCTL

You can adjust the count, type, and size of the compute instances within a deployment. The hdxctl scale command lets you view or change these settings for individual components, and provides shortcuts that quickly set most of a cluster's components to a minimal or inactive state.

Viewing Current Scale

Running hdxctl scale CLIENT_ID CLUSTER_ID without any further options displays a table summarizing given cluster's scale settings, with one row per component:




The name of the component this row describes.


The number of this instances this component contains.


The type of instance this component contains.


The size of this component's compute instances.


The size of this component's EBS storage volumes, in GB.


The type of node deployed, e.g. bastion or head.

For example:

$ hdxctl scale hdxcli-example123  hdx-example321

SERVICE/POOL                                      COUNT  FAMILY    SIZE       DISK  TYPE
----------------------------------------------  -------  --------  -------  ------  -----------
alter-peer0                                           1  r5        large        30  alter-peer
bastion                                               1  t2        micro        30  bastion
batch-peer0                                           1  r5        xlarge       30  batch-peer
grafana                                               1  t2        small        30  grafana
head                                                  1  c5n       9xlarge      30  head
intake-misc                                           1  t2        medium       30  intake-misc
kafkatest-82bc011b-a098-4a46-89a7-c0f82908b6f4        1  r5        xlarge       30  kafka-peer
merge-peer0                                           1  r5        xlarge       30  merge-peer
prometheus                                            1  t2        2xlarge      50  prometheus
query-peer0                                           1  c5n       9xlarge      30  peer
stream-head                                           1  r5        xlarge       30  stream-head
stream-peer0                                          1  r5        xlarge       30  stream-peer
superset                                              1  t3        large        30  superset
query_pool                                            1  c5n       9xlarge      30  peer
web                                                   1  t3        small        30  web
zookeeper                                             3  t2        micro        30  zookeeper

Basic Component Scaling

To adjust the count, type, or size of any component, run hdxctl scale CLIENT_ID CLUSTER_ID while supplying one or more of the many command options that refer to specific scalable attributes of your cluster. Follow each option with the desired new value.

For example, to set a cluster's query head and peer counts to five each:

$ hdxctl scale --head-count 5 --query-peer-count 5 hdxcli-example123  hdx-example321

Scaling instance type, disk and cache-disk

In addition to scaling the number of instances you are also able to scale the type of the instance, the volume sized for the root disk and where relevant the cache-disk volume.

--<service/pool>-instance-typechange the instance type or size of the server - e.g. c5n.4xlarge to c5n.9xlarge
--<service/pool>-diskThe size in GB for the root volume.
--<service/pool>-cache-diskThe size in GB for the cache-disk volume.


Scaling Pools

If you are looking to scale a Pool, this can be done either through the Portal, API or using TOML configuration files.


Two special options, --minimal and --off, let you quickly scale most or all of a cluster's components to one or zero instances, respectively.


The --minimal option sets every component of the given cluster to a single instance. If a component has more than one instance, this will scale them back to one; if it has zero, then it launches a single instance.

Use this option when you wish to activate a whole cluster at minimal cost, ready for further fine-tuning as needed.

For example:

$ hdxctl scale --minimal hdxcli-example123  hdx-example321


The --off option scales most of the given cluster's components down to zero instances. This option essentially operates as a "sleep mode" for a cluster, retaining its configuration but quiescing most of its operations.

For example:

$ hdxctl scale --off hdxcli-example123  hdx-example321

Running hdxctl scale --off does not affect your cluster's Bastion or Zookeeper instances, which help coordinate the basic operation of your cluster. It also does not affect any components that you manually added to the cluster after its creation, such as Kafka listeners.

To conveniently reactivate a cluster after --off, consider using running hdxctl scale again with the --minimal option, as described in the previous section.

Advanced Scaling

Where more advanced configurations are needed with multiple pools of servers Hydrolix also provides the option to configure the deployment using a TOML configuration file..


For the various Cloud platforms Hydrolix deploys too there are "Tunables" that can be set to alter the behaviour of the cluster. Contained within a file, it includes a number of the configurable settings used platform wide, some settings are common across the different cloud platforms, but there are others that are specific to AWS, Google etc. Examples of these settings include settings for platform services such as queue time-outs, AWS Kinesis shard counts, SSH Authorized keys and S3 bucket permissions and more.


To learn more about Tunables for your platform

AWS Platform Tunables

Did this page help you?