Scaling via the Command Line
You can adjust the count, type, and size of the compute instances within a deployment. The hdxctl scale
command lets you view or change these settings for individual components, and provides shortcuts that quickly set most of a cluster's components to a minimal or inactive state.
Viewing Current Scale
Running hdxctl scale CLIENT_ID CLUSTER_ID
without any further options displays a table summarizing given cluster's scale settings, with one row per component:
Column | Description |
---|---|
SERVICE/POOL | The name of the component this row describes. |
COUNT | The number of this instances this component contains. |
FAMILY | The type of instance this component contains. |
SIZE | The size of this component's compute instances. |
DISK | The size of this component's EBS storage volumes, in GB. |
TYPE | The type of node deployed, e.g. bastion or head. |
For example:
$ hdxctl scale hdxcli-example123 hdx-example321
SERVICE/POOL COUNT FAMILY SIZE DISK TYPE
---------------------------------------------- ------- -------- ------- ------ -----------
alter-peer0 1 r5 large 30 alter-peer
bastion 1 t2 micro 30 bastion
batch-peer0 1 r5 xlarge 30 batch-peer
grafana 1 t2 small 30 grafana
head 1 c5n 9xlarge 30 head
intake-misc 1 t2 medium 30 intake-misc
kafkatest-82bc011b-a098-4a46-89a7-c0f82908b6f4 1 r5 xlarge 30 kafka-peer
merge-peer0 1 r5 xlarge 30 merge-peer
prometheus 1 t2 2xlarge 50 prometheus
query-peer0 1 c5n 9xlarge 30 peer
stream-head 1 r5 xlarge 30 stream-head
stream-peer0 1 r5 xlarge 30 stream-peer
superset 1 t3 large 30 superset
query_pool 1 c5n 9xlarge 30 peer
web 1 t3 small 30 web
zookeeper 3 t2 micro 30 zookeeper
Basic Component Scaling
To adjust the count, type, or size of any component, run hdxctl scale CLIENT_ID CLUSTER_ID
while supplying one or more of the many command options that refer to specific scalable attributes of your cluster. Follow each option with the desired new value.
For example, to set a cluster's query head and peer counts to five each:
$ hdxctl scale --head-count 5 --query-peer-count 5 hdxcli-example123 hdx-example321
Scaling instance type, disk and cache-disk
In addition to scaling the number of instances you are also able to scale the type of the instance, the volume sized for the root disk and where relevant the cache-disk volume.
Column | Description |
---|---|
--<service/pool>-instance-type | change the instance type or size of the server - e.g. c5n.4xlarge to c5n.9xlarge |
--<service/pool>-disk | The size in GB for the root volume. |
--<service/pool>-cache-disk | The size in GB for the cache-disk volume. |
Scaling Pools
If you are looking to scale a Pool, this can be done either through the Portal, API or using TOML configuration files.
Scaling Shortcuts
Two special options, --minimal
and --off
, let you quickly scale most or all of a cluster's components to one or zero instances, respectively.
--minimal
--minimal
The --minimal
option sets every component of the given cluster to a single instance. If a component has more than one instance, this will scale them back to one; if it has zero, then it launches a single instance.
Use this option when you wish to activate a whole cluster at minimal cost, ready for further fine-tuning as needed.
For example:
$ hdxctl scale --minimal hdxcli-example123 hdx-example321
--off
--off
The --off
option scales most of the given cluster's components down to zero instances. This option essentially operates as a "sleep mode" for a cluster, retaining its configuration but quiescing most of its operations.
For example:
$ hdxctl scale --off hdxcli-example123 hdx-example321
Running hdxctl scale --off
does not affect your cluster's Bastion or Zookeeper instances, which help coordinate the basic operation of your cluster. It also does not affect any components that you manually added to the cluster after its creation, such as Kafka listeners.
To conveniently reactivate a cluster after --off
, consider using running hdxctl scale
again with the --minimal
option, as described in the previous section.
Advanced Scaling (TOML)
For more advanced usage, scaling and configuration can be managed through the use of TOML files. The usage of TOML files enables the ability to build "profiles" for the scale, structure and configuration of your cluster, it also enables the configuration of the underlying cluster to be saved for later use. There are two methods to access the TOML file, firstly using the --edit
flag and secondly through using the --emit-toml
flag.
Using --edit
--edit
The edit option is a good way to edit the TOML configuration "in-place". The command downloads the TOML config file from your cluster and opens a 'vi' session for the user to edit the file. Once the user has written any changes to the file and quit the session, the TOML configuration is updated automatically on the cluster .
For example:
$ hdxctl scale hdxctl-xxxyyy hdx-111222 --edit
--edit Be Careful
If you are updating using --edit be careful of your changes as on writing and exiting (
wq!
) the cluster will be updated.
Using --emit-toml / from-file
The more advanced option for platform configuration management is to use the --emit-toml
flag. This allows the user to configure, manage and build underlying cluster components configuration through the use of external configuration files. Along with the --from-file
command this allows the user to use multiple configurations matching their workloads at that point in time.
The --emit-toml
command will print the contents of the configuration to STDOUT. For example:
$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml
This in turn can be sent to a file that you can then edit with your favourite text editor. For example:
$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml > myconfig.toml
$ vi myconfig.toml
Once you have made you changes, you can "load" your configuration into your cluster using the --from-file
command. This command will load your configuration from the file you provide. For example:
$ hdxctl scale hdxctl-xxxyyy hdx-111222 --from-file myconfig.toml
TOML Structure
TOML files are structured using a hierarchy where defaults are specified for Services
and (worker) Pools
, with each inheriting settings or overriding them with their own specific settings.
$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml
[service.default.instance]
family = "t2"
size = "micro"
count = 1
disk = 30
cache_disk = 0
spot = false
on_demand_percent = 5
[service.stream-head.instance]
family = "r5"
size = "xlarge"
count = 1
disk = 30
spot = false
cache_disk = 0
on_demand_percent = 25
[pool.merge-peer0.instance]
service = "merge-peer"
count = 1
family = "r5"
size = "xlarge"
disk = 30
spot = false
cache_disk = 0
on_demand_percent = 25
.........
Components within the TOML fall into two categories, Service Components and (worker) Pool components. Pools are typically the components that are required to scale to individual workloads (e.g. where you want two sets of "query-peer" workers, one for the DevOps team and one for everyone else). Service components can not be scaled into separate pools, but can still be scaled horizontally in number and vertically in size.
alter-head bastion config intake-misc grafana head rds stream-head prometheus superset web zookeeper | service | No |
query-peer merge-peer kafka-peer batch-peer alter-peer | pool | Yes |
Defaults
On initial build Hydrolix specifies default Service and (worker) Pool components. These defaults form the basic deployment of Hydrolix and use a hierarchy structure to override certain settings dependent on the component.
The structure of the TOML follows a hierarchical format where components inherit their settings from the parent. This works as follows:
service.default.instance
|--service.{{service-type}}.instance
|--pool_default.{{service-type}}.instance
|--pool.{{pool-name}}.instance
service: "<service-name>"
service.default.instance | Top level for both pools and services. |
service.{{service-type}}.instance | Defines the specific service component, inherits from the service.default.instance . |
pool_default.{{service-type}}.instance | Specifies default settings for the type of pool. |
pool.{{pool-name}}.instance | Specifies the settings for an individual pool. Pools inherit their settinges from the pool_default.{{service-type}}.instance . The Pool setting service: is used to match the service-type of the parent. |
This structure allows the user to specify different server parameters for different pools . For example, the below shows 2 different pools of query servers (rnd and devops) that pull different settings from different levels and overriding others.
[service.default.instance]
family = "t2"
size = "micro"
count = 1
disk = 30
cache_disk = 0
spot = false
on_demand_percent = 5
[pool_default.peer.instance]
family = "c5n"
size = "4xlarge"
cache_disk = 24
count=3
[pool.rnd.instance]
service = "peer"
count = 5
cache_disk = 24
[pool.devops.instance]
service = "peer"
count = 10
size = "9xlarge"
family = "c5n"
spot = true
cache_disk = 24
on_demand_percent = 25
TOML Parameters
The following settings are available for each component.
Parameter | Description | Example |
---|---|---|
service | The type of service the component will be used for. (Only used with (worker) Pools). | "kafka-peer", "peer", "merge-peer" |
count | How many instances to be used for a service. If auto scaling is to be used (query-peer only) either a min or max can be provided (e.g. 2-5) or a min, desired and max can be specified e.g. (2-5-10). | 1, 2-5, 2-5-10 |
family | Family for the instances to be deployed. | AWS: r5, c5n, t5 etc. |
size | Size of the instance to be deployed. | AWS: Small, Medium, Large, 9xlarge etc |
disk | The size of the EBS volume to be utilized by the instance. | 30 |
spot | The size of the cache_disk to be utilized (query-peer only). | 30 |
on_demand_percent | Specifies the number of on-demand % that should be used when using spot instances. | 25 |
Note: AWS - on_demand_percent
Note that AWS rounds up so any on_demand_percent greater than 0.
Pools
Hydrolix has the capability to create pools of servers for specific workloads. Each Pool has ring-fenced capacity so that a workload will not be affected by other processes or users querying data or ingesting data.
For example a query-pool can be created for a subset of users that is accessible only by them, with another pool being used by everyone else or a specific pool of kafka servers can be used for the ingesting of server logs, but another Kafka pool used for performance logs.
By having this capability it allows independent access and scaling of workload components without the need to overprovision a single architecture. In addition with query server components, Role Based Performance Control is also possible - so users can have access to different levels of performance based on their role within a company.
The following components support pooling:
- query-peer
- merge-peer
- kafka-peer
Adding a Pool
To create a pool using TOML simply add to the file. For example if you wanted to create a new query pool cluster with 3 servers of c5n.9xlarge for the Support team you would add the following:
[pool.support.instance]
service = "peer"
count = 3
family = c5n
size = 9xlarge
Deleting a Pool
To remove components from the infrastructure the easiest method is to set the component to 0.
Where a Pool has been created this can be removed by removing the parameters from within the TOML and the .instance
descriptor.
For example to remove the following pool:
[pool.support.instance]
service = "peer"
count = 3
family = c5n
size = 9xlarge
Simply update the TOML to:
[pool.support]
This will remove this pool from the cluster.
Default Services/Pools
Default pools should stay persistent and should only be set to
count=0
.
AutoScaling
Hydrolix supports the ability to enable Autoscaling for certain components on certain platforms.
The argument to scaling count arguments takes not just a single value but also allows specifying a range. You can either specify a single number, a min, and max or a min, desired and max separated by dashes. If you specify a range, the control plane will create Cloudwatch alerts on CPU that will trigger scaling events.
Only use for Query-Peers
Primarily used for Query-Peers, it can be set for other components, however this is not recommended.
The following works as before, no autoscaling alarms are created:
$ hdxctl scale --query-peer-count 1 hdxctl-xxxyyy hdx-111222
The following sets the min to one, the desired count to one and the max to five and creates alerts for scaling up and down based on CPU.
$ hdxctl scale --stream-peer-count 1-5 hdxctl-xxxyyy hdx-111222
The following does the same as the above but sets the desired count to 3.
$ hdxctl scale --stream-peer-count 1-3-5 hdxctl-xxxyyy hdx-111222
Alternatively Autoscaling can be set-up through the user of the scaling TOML.
[pool.query-peer0.instance]
service = "peer"
count = "1-3-5" <--- here
family = "c5n"
size = "9xlarge"
spot = true
disk = 30
cache_disk = 24
Spot Instances
Hydrolix supports the ability to deploy spot instances in replacement of on-demand. If this setting is enabled for a pool, the autoscale group will only request one reserved node (or 5% of the count if the count is above 30) and will attempt to request spot instances for the remaining requested capacity.
There are two methods to enable spot instances:
-
There is a scaling option for each instance type that allows you to turn on the use of spot instances. For example, hdxctl scale --query-peer-spot true CLIENT_ID CLUSTER_ID, will turn on the use of spot instances for query peers
-
The setting is edited through the TOML settings using
spot = true
. An example is below:
[pool.query-peer0.instance]
service = "peer"
count = "1-3-5"
family = "c5n"
size = "9xlarge"
spot = true <--- here
disk = 30
cache_disk = 24
Updated 3 months ago