Scaling via the Command Line

You can adjust the count, type, and size of the compute instances within a deployment. The hdxctl scale command lets you view or change these settings for individual components, and provides shortcuts that quickly set most of a cluster's components to a minimal or inactive state.

Viewing Current Scale

Running hdxctl scale CLIENT_ID CLUSTER_ID without any further options displays a table summarizing given cluster's scale settings, with one row per component:

ColumnDescription
SERVICE/POOLThe name of the component this row describes.
COUNTThe number of this instances this component contains.
FAMILYThe type of instance this component contains.
SIZEThe size of this component's compute instances.
DISKThe size of this component's EBS storage volumes, in GB.
TYPEThe type of node deployed, e.g. bastion or head.

For example:

$ hdxctl scale hdxcli-example123  hdx-example321

SERVICE/POOL                                      COUNT  FAMILY    SIZE       DISK  TYPE
----------------------------------------------  -------  --------  -------  ------  -----------
alter-peer0                                           1  r5        large        30  alter-peer
bastion                                               1  t2        micro        30  bastion
batch-peer0                                           1  r5        xlarge       30  batch-peer
grafana                                               1  t2        small        30  grafana
head                                                  1  c5n       9xlarge      30  head
intake-misc                                           1  t2        medium       30  intake-misc
kafkatest-82bc011b-a098-4a46-89a7-c0f82908b6f4        1  r5        xlarge       30  kafka-peer
merge-peer0                                           1  r5        xlarge       30  merge-peer
prometheus                                            1  t2        2xlarge      50  prometheus
query-peer0                                           1  c5n       9xlarge      30  peer
stream-head                                           1  r5        xlarge       30  stream-head
stream-peer0                                          1  r5        xlarge       30  stream-peer
superset                                              1  t3        large        30  superset
query_pool                                            1  c5n       9xlarge      30  peer
web                                                   1  t3        small        30  web
zookeeper                                             3  t2        micro        30  zookeeper

Basic Component Scaling

To adjust the count, type, or size of any component, run hdxctl scale CLIENT_ID CLUSTER_ID while supplying one or more of the many command options that refer to specific scalable attributes of your cluster. Follow each option with the desired new value.

For example, to set a cluster's query head and peer counts to five each:

$ hdxctl scale --head-count 5 --query-peer-count 5 hdxcli-example123  hdx-example321

Scaling instance type, disk and cache-disk

In addition to scaling the number of instances you are also able to scale the type of the instance, the volume sized for the root disk and where relevant the cache-disk volume.

ColumnDescription
--<service/pool>-instance-typechange the instance type or size of the server - e.g. c5n.4xlarge to c5n.9xlarge
--<service/pool>-diskThe size in GB for the root volume.
--<service/pool>-cache-diskThe size in GB for the cache-disk volume.

👍

Scaling Pools

If you are looking to scale a Pool, this can be done either through the Portal, API or using TOML configuration files.

Scaling Shortcuts

Two special options, --minimal and --off, let you quickly scale most or all of a cluster's components to one or zero instances, respectively.

--minimal

The --minimal option sets every component of the given cluster to a single instance. If a component has more than one instance, this will scale them back to one; if it has zero, then it launches a single instance.

Use this option when you wish to activate a whole cluster at minimal cost, ready for further fine-tuning as needed.

For example:

$ hdxctl scale --minimal hdxcli-example123  hdx-example321

--off

The --off option scales most of the given cluster's components down to zero instances. This option essentially operates as a "sleep mode" for a cluster, retaining its configuration but quiescing most of its operations.

For example:

$ hdxctl scale --off hdxcli-example123  hdx-example321

Running hdxctl scale --off does not affect your cluster's Bastion or Zookeeper instances, which help coordinate the basic operation of your cluster. It also does not affect any components that you manually added to the cluster after its creation, such as Kafka listeners.

To conveniently reactivate a cluster after --off, consider using running hdxctl scale again with the --minimal option, as described in the previous section.

Advanced Scaling (TOML)

For more advanced usage, scaling and configuration can be managed through the use of TOML files. The usage of TOML files enables the ability to build "profiles" for the scale, structure and configuration of your cluster, it also enables the configuration of the underlying cluster to be saved for later use. There are two methods to access the TOML file, firstly using the --edit flag and secondly through using the --emit-toml flag.

Using --edit

The edit option is a good way to edit the TOML configuration "in-place". The command downloads the TOML config file from your cluster and opens a 'vi' session for the user to edit the file. Once the user has written any changes to the file and quit the session, the TOML configuration is updated automatically on the cluster .

For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --edit

❗️

--edit Be Careful

If you are updating using --edit be careful of your changes as on writing and exiting (wq!) the cluster will be updated.

Using --emit-toml / from-file

The more advanced option for platform configuration management is to use the --emit-toml flag. This allows the user to configure, manage and build underlying cluster components configuration through the use of external configuration files. Along with the --from-file command this allows the user to use multiple configurations matching their workloads at that point in time.

The --emit-toml command will print the contents of the configuration to STDOUT. For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml

This in turn can be sent to a file that you can then edit with your favourite text editor. For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml  > myconfig.toml
$ vi myconfig.toml

Once you have made you changes, you can "load" your configuration into your cluster using the --from-file command. This command will load your configuration from the file you provide. For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --from-file myconfig.toml

TOML Structure

TOML files are structured using a hierarchy where defaults are specified for Services and (worker) Pools, with each inheriting settings or overriding them with their own specific settings.

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml
[service.default.instance]
family = "t2"
size = "micro"
count = 1
disk = 30
cache_disk = 0
spot = false
on_demand_percent = 5

[service.stream-head.instance]
family = "r5"
size = "xlarge"
count = 1
disk = 30
spot = false
cache_disk = 0
on_demand_percent = 25

[pool.merge-peer0.instance]
service = "merge-peer"
count = 1
family = "r5"
size = "xlarge"
disk = 30
spot = false
cache_disk = 0
on_demand_percent = 25

.........

Components within the TOML fall into two categories, Service Components and (worker) Pool components. Pools are typically the components that are required to scale to individual workloads (e.g. where you want two sets of "query-peer" workers, one for the DevOps team and one for everyone else). Service components can not be scaled into separate pools, but can still be scaled horizontally in number and vertically in size.

alter-head
bastion
config
intake-misc
grafana
head
rds
stream-head
prometheus
superset
web
zookeeper
serviceNo
query-peer
merge-peer
kafka-peer
batch-peer
alter-peer
poolYes

Defaults

On initial build Hydrolix specifies default Service and (worker) Pool components. These defaults form the basic deployment of Hydrolix and use a hierarchy structure to override certain settings dependent on the component.

The structure of the TOML follows a hierarchical format where components inherit their settings from the parent. This works as follows:

service.default.instance
  |--service.{{service-type}}.instance
  |--pool_default.{{service-type}}.instance
       |--pool.{{pool-name}}.instance
           service: "<service-name>"
service.default.instanceTop level for both pools and services.
service.{{service-type}}.instanceDefines the specific service component, inherits from the service.default.instance.
pool_default.{{service-type}}.instanceSpecifies default settings for the type of pool.
pool.{{pool-name}}.instanceSpecifies the settings for an individual pool. Pools inherit their settinges from the pool_default.{{service-type}}.instance. The Pool setting service: is used to match the service-type of the parent.

This structure allows the user to specify different server parameters for different pools . For example, the below shows 2 different pools of query servers (rnd and devops) that pull different settings from different levels and overriding others.

[service.default.instance]
family = "t2"
size = "micro"
count = 1
disk = 30
cache_disk = 0
spot = false
on_demand_percent = 5

[pool_default.peer.instance]
family = "c5n"
size = "4xlarge"
cache_disk = 24
count=3

[pool.rnd.instance]
service = "peer"
count = 5
cache_disk = 24

[pool.devops.instance]
service = "peer"
count = 10
size = "9xlarge"
family = "c5n"
spot = true
cache_disk = 24
on_demand_percent = 25

TOML Parameters

The following settings are available for each component.

ParameterDescriptionExample
serviceThe type of service the component will be used for. (Only used with (worker) Pools)."kafka-peer", "peer", "merge-peer"
countHow many instances to be used for a service. If auto scaling is to be used (query-peer only) either a min or max can be provided (e.g. 2-5) or a min, desired and max can be specified e.g. (2-5-10).1, 2-5, 2-5-10
familyFamily for the instances to be deployed.AWS: r5, c5n, t5 etc.
sizeSize of the instance to be deployed.AWS: Small, Medium, Large, 9xlarge etc
diskThe size of the EBS volume to be utilized by the instance.30
spotThe size of the cache_disk to be utilized (query-peer only).30
on_demand_percentSpecifies the number of on-demand % that should be used when using spot instances.25

📘

Note: AWS - on_demand_percent

Note that AWS rounds up so any on_demand_percent greater than 0.

Pools

Hydrolix has the capability to create pools of servers for specific workloads. Each Pool has ring-fenced capacity so that a workload will not be affected by other processes or users querying data or ingesting data.

For example a query-pool can be created for a subset of users that is accessible only by them, with another pool being used by everyone else or a specific pool of kafka servers can be used for the ingesting of server logs, but another Kafka pool used for performance logs.

By having this capability it allows independent access and scaling of workload components without the need to overprovision a single architecture. In addition with query server components, Role Based Performance Control is also possible - so users can have access to different levels of performance based on their role within a company.

773

📘

The following components support pooling:

  • query-peer
  • merge-peer
  • kafka-peer

Adding a Pool

To create a pool using TOML simply add to the file. For example if you wanted to create a new query pool cluster with 3 servers of c5n.9xlarge for the Support team you would add the following:

[pool.support.instance]
service = "peer"
count = 3
family = c5n
size = 9xlarge

Deleting a Pool

To remove components from the infrastructure the easiest method is to set the component to 0.

Where a Pool has been created this can be removed by removing the parameters from within the TOML and the .instance descriptor.

For example to remove the following pool:

[pool.support.instance]
service = "peer"
count = 3
family = c5n
size = 9xlarge

Simply update the TOML to:

[pool.support]

This will remove this pool from the cluster.

📘

Default Services/Pools

Default pools should stay persistent and should only be set to count=0.

AutoScaling

Hydrolix supports the ability to enable Autoscaling for certain components on certain platforms.

The argument to scaling count arguments takes not just a single value but also allows specifying a range. You can either specify a single number, a min, and max or a min, desired and max separated by dashes. If you specify a range, the control plane will create Cloudwatch alerts on CPU that will trigger scaling events.

🚧

Only use for Query-Peers

Primarily used for Query-Peers, it can be set for other components, however this is not recommended.

The following works as before, no autoscaling alarms are created:

$ hdxctl scale --query-peer-count 1 hdxctl-xxxyyy hdx-111222

The following sets the min to one, the desired count to one and the max to five and creates alerts for scaling up and down based on CPU.

$ hdxctl scale --stream-peer-count 1-5 hdxctl-xxxyyy hdx-111222

The following does the same as the above but sets the desired count to 3.

$ hdxctl scale --stream-peer-count 1-3-5 hdxctl-xxxyyy hdx-111222

Alternatively Autoscaling can be set-up through the user of the scaling TOML.

[pool.query-peer0.instance]
service = "peer"
count = "1-3-5"     <--- here
family = "c5n"
size = "9xlarge"
spot = true
disk = 30
cache_disk = 24

Spot Instances

Hydrolix supports the ability to deploy spot instances in replacement of on-demand. If this setting is enabled for a pool, the autoscale group will only request one reserved node (or 5% of the count if the count is above 30) and will attempt to request spot instances for the remaining requested capacity.

There are two methods to enable spot instances:

  • There is a scaling option for each instance type that allows you to turn on the use of spot instances. For example, hdxctl scale --query-peer-spot true CLIENT_ID CLUSTER_ID, will turn on the use of spot instances for query peers

  • The setting is edited through the TOML settings using spot = true. An example is below:

[pool.query-peer0.instance]
service = "peer"
count = "1-3-5"
family = "c5n"
size = "9xlarge"
spot = true         <--- here
disk = 30
cache_disk = 24