Scaling clusters

You can adjust the count, type, and size of the cloud instances of any cluster's components. The hdxctl scale command lets you view or change these settings for individual components, and provides shortcuts that quickly set most of a cluster's components to a minimal or inactive state.

This guide presents an overview of using hdxctl scale. For more thorough documentation, consult the hdxctl reference manual.

Viewing a cluster's current scale settings

Running hdxctl scale CLIENT_ID CLUSTER_ID without any further options displays a table summarizing given cluster's scale settings, with one row per component:

ColumnDescription
SERVICE/POOLThe name of the component this row describes.
COUNTThe number of this instances this component contains.
FAMILYThe type of instance this component contains.
SIZEThe size of this component's compute instances.
DISKThe size of this component's EBS storage volumes, in GB.

For example:

$ hdxctl scale hdxcli-example123  hdx-example321

SERVICE/POOL      COUNT  FAMILY    SIZE       DISK
--------------  -------  --------  -------  ------
batch-peer0           1  r5        xlarge       30
config                1  t2        micro        30
grafana               1  t2        micro        30
head                  1  c5n       xlarge       30
merge-peer0           1  r5        large        30
prometheus            1  t2        micro        30
query-peer0           1  c5n       2xlarge      30
stream-head           1  m5        large        30
stream-peer0          1  m5        large        30
superset              1  t2        medium       30
ui                    1  t2        micro        30

Scaling individual components

To adjust the count, type, or size of any component, run hdxctl scale CLIENT_ID CLUSTER_ID while supplying one or more of the many command options that refer to specific scalable attributes of your cluster. Follow each option with the desired new value.

For example, to set a cluster's query head and peer counts to five each:

$ hdxctl scale --head-count 5 --query-peer-count 5 hdxcli-example123  hdx-example321

Scaling a whole cluster

Two special options, --minimal and --off, let you quickly scale most or all of a cluster's components to one or zero instances, respectively.

--minimal

The --minimal option sets every component of the given cluster to a single instance. If a component has more than one instance, this will scale them back to one; if it has zero, then it launches a single instance.

Use this option when you wish to activate a whole cluster at minimal cost, ready for further fine-tuning as needed.

For example:

$ hdxctl scale --minimal hdxcli-example123  hdx-example321

--off

The --off option scales most of the given cluster's components down to zero instances. This option essentially operates as a "sleep mode" for a cluster, retaining its configuration but quiescing most of its operations.

For example:

$ hdxctl scale --off hdxcli-example123  hdx-example321

Running hdxctl scale --off does not affect your cluster's Bastion or Zookeeper instances, which help coordinate the basic operation of your cluster. It also does not affect any components that you manually added to the cluster after its creation, such as Kafka listeners.

To conveniently reactivate a cluster after --off, consider using running hdxctl scale again with the --minimal option, as described in the previous section.

Advanced Scaling options

For ease of use where multiple components may need to be updated or to allow for "profiling" of a platforms scale, Hydrolix also provides Scaling configuration file.


Did this page help you?