Scaling the Stack with hdxctl

Each component deployed in a Hydrolix cluster is configurable in terms of scale: number and instance type. We have a few build in group scale options: --off everything spun down, --minimal a single instance of all services spun up.

You can issue multiple options at the same time. For example, I can change the query-peer count and instance type in one command.

hdxctl scale --query-peer-count 10 --query-peer-instance-type c5n.xlarge CLIENT_ID CLUSTER_ID

To see the full list of options hdxctl scale --help

Usage: hdxctl scale [OPTIONS] CLIENT_ID [CLUSTER_ID]

Options:
  --edit / --no-edit
  --from-file TEXT
  --off / --no-off
  --minimal / --no-minimal
  --emit-toml / --no-emit-toml
  --update-ok
  --config-count INTEGER
  --config-instance-type TEXT
  --config-disk INTEGER
  --query-peer-count INTEGER
  --query-peer-instance-type TEXT
  --query-peer-disk INTEGER
  --ui-count INTEGER
  --ui-instance-type TEXT
  --ui-disk INTEGER
  --grafana-count INTEGER
  --grafana-instance-type TEXT
  --grafana-disk INTEGER
  --batch-peer-count INTEGER
  --batch-peer-instance-type TEXT
  --batch-peer-disk INTEGER
  --superset-count INTEGER
  --superset-instance-type TEXT
  --superset-disk INTEGER
  --prometheus-count INTEGER
  --prometheus-instance-type TEXT
  --prometheus-disk INTEGER
  --stream-peer-count INTEGER
  --stream-peer-instance-type TEXT
  --stream-peer-disk INTEGER
  --merge-peer-count INTEGER
  --merge-peer-instance-type TEXT
  --merge-peer-disk INTEGER
  --stream-head-count INTEGER
  --stream-head-instance-type TEXT
  --stream-head-disk INTEGER
  --head-count INTEGER
  --head-instance-type TEXT
  --head-disk INTEGER
  --help                          Show this message and exit.

Current Settings

Using the command without any options will supply a view of the cluster settings including the count, the family of the compute instance, the size of the instance and the size of the EBS volume attributed to it.

hdxctl scale <CLIENT_ID> <CLUSTER_ID>

Example:

 hdxctl scale hdxcli-pmpudpqe hdx-mv6hiw2t
SERVICE/POOL      COUNT  FAMILY    SIZE       DISK
--------------  -------  --------  -------  ------
batch-peer0           1  r5        xlarge       30
config                1  t2        micro        30
grafana               1  t2        micro        30
head                  1  c5n       xlarge       30
merge-peer0           1  r5        large        30
prometheus            1  t2        micro        30
query-peer0           1  c5n       2xlarge      30
stream-head           1  m5        large        30
stream-peer0          1  m5        large        30
superset              1  t2        medium       30
ui                    1  t2        micro        30

--minimal

The --minimal setting will set each of the components to have a single instance of each component. If any components are currently scaled to zero, a single instance will be started.

Usage : hdxctl scale --minimal <CLIENT_ID> <CLUSTER_ID>

Example : hdxctl scale --minimal hdxcli-pmpudpqe hdx-ex6bdtsn

Scaling individual components

Hydrolix can scale workload compute components independently depending on the usage and behaviour required.

The following components can be individually scaled

Option Description
--batch-peer-count specify the number of batch peers, a minimum of 1 is required to use batch ingest.
--batch-peer-disk specify the amount of disk (EBS) you wish to use on the batch peer. Recommended to be kept as default.
--batch-peer-instance-type change the type and class of the batch peer (e.g. m5.large).
--config-count specify the number of config servers. Recommended to be kept as default.
--config-disk specify the amount of disk (EBS) you wish to use on the config server. Recommended to be kept as default.
--config-instance-type change the type and class of the config server. Recommended to be kept as default.
--grafana-count specify the number of Grafana servers. Recommended to be kept as default.
--grafana-disk specify the amount of disk (EBS) you wish to use on the Grafana server. Recommended to be kept as default.
--grafana-instance-type change the type and class of the Grafana server. Recommended to be kept as default.
--prometheus-count Specify the number of Prometheus servers.
--prometheus-instance-type Change the type and class of cloud instance that Prometheus uses.
--prometheus-disk specify the amount of disk (EBS) you wish to use on the Prometheus server.
--query-head-count specify the number of query heads to use, a minimum of 1 is required to query the infrastructure.
--query-head-disk specify the amount of disk (EBS) you wish to use on the query head. Note this is for caching puroposes. Recommended to be kept as default.
--query-head-instance-type change the type and class of the query head (e.g. c5n.xlarge).
--query-peer-count specify the number of query peers, a minimum of 1 is required to query the infrastructure.
--query-peer-disk specify the amount of disk (EBS) you wish to use on the query peer. Note this is for caching puroposes. Recommended to be kept as default.
--query-peer-instance-type change the type and class of the query peer (e.g. c5n.2xlarge).
--stream-head-count specify the number of stream heads to use, a minimum of 1 is required to use streaming ingest.
--stream-head-disk specify the amount of disk (EBS) you wish to use on the stream head. Recommended to be kept as default.
--stream-head-instance-type change the type and class of the stream head (e.g. m5.large).
--stream-peer-count specify the number of query peers, a minimum of 1 is required to use streaming ingest.
--stream-peer-disk specify the amount of disk (EBS) you wish to use on the stream peer. Recommended to be kept as default.
--stream-peer-instance-type change the type and class of the stream peer (e.g. m5.large).
--superset-count Specify the number of Superset servers.
--superset-instance-type Change the type and class of cloud instance that Superset uses.
--superset-disk specify the amount of disk (EBS) you wish to use on the Superset server.
--ui-count specify the number of ui servers. Recommended to be kept as default.
--ui-disk specify the amount of disk (EBS) you wish to use on the ui server. Recommended to be kept as default.
--ui-instance-type change the type and class of the ui server. Recommended to be kept as default.

Usage : hdxctl scale <COMPONENT> <INTEGER> [...<COMPONENT> <INTEGER>...] <CLIENT_ID> <CLUSTER_ID>

Example : hdxctl scale --query-head-count --query-peer-count hdxcli-pmpudpqe hdx-ex6bdtsn

--off option

The --off setting will reduce the Hydrolix stack to its absolute minimum, turning off all but necessary components to restart the system. The aim of this setting is to reduce as much as possible the footprint of the architecture. There can be a small delay in getting down to the absolute minimum. This is due to connection draining on the load-balancer and you may find that the stream-head appears to linger. This is because this component is one of the last to be shut down.

Usage : hdxctl scale --off <CLIENT_ID> <CLUSTER_ID>

Example : hdxctl scale --off hdxcli-pmpudpqe hdx-ex6bdtsn

Scaling individual components after --off

Once a deployment has been turned off it is put into effectively a sleep mode with only the minimal components running. This means should you wish to re-enable it, it is important to ensure you not only enable the components you need (e.g. query-head and query-peer) but also any supporting components such as the config cluster (this is used to control how the rest of the cluster behaves) or the UI if you wish to interact with the service via the portal. To use the service the config cluster is a dependency for running the batch, stream and query components