Configuration using TOML

For more advanced usage, scaling and configuration can be managed through the use of TOML files. The usage of TOML files enables the ability to build "profiles" for the scale, structure and configuration of your cluster, it also enables the configuration of the underlying cluster to be saved for later use.

Accessing your TOML

There are two methods to access the TOML file, firstly using the --edit flag and secondly through using the --emit-toml flag.

Using --edit

The edit option is a good way to edit the TOML configuration "in-place". The command downloads the TOML config file from your cluster and opens a 'vi' session for the user to edit the file. Once the user has written any changes to the file and quit the session, the TOML configuration is updated automatically on the cluster .

For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --edit

❗️

--edit Be Careful

If you are updating using --edit be careful of your changes as on writing and exiting (wq!) the cluster will be updated.

Using --emit-toml / from-file

The more advanced option for platform configuration management is to use the --emit-toml flag. This allows the user to configure, manage and build underlying cluster components configuration through the use of external configuration files. Along with the --from-file command this allows the user to use multiple configurations matching their workloads at that point in time.

The --emit-toml command will print the contents of the configuration to STDOUT. For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml

This in turn can be sent to a file that you can then edit with your favourite text editor. For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml  > myconfig.toml
$ vi myconfig.toml

Once you have made you changes, you can "load" your configuration into your cluster using the --from-file command. This command will load your configuration from the file you provide. For example:

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --from-file myconfig.toml

TOML Structure

TOML files are structured using a hierarchy where defaults are specified for Services and (worker) Pools, with each inheriting settings or overriding them with their own specific settings.

$ hdxctl scale hdxctl-xxxyyy hdx-111222 --emit-toml
[service.default.instance]
family = "t2"
size = "micro"
count = 1
disk = 30
cache_disk = 0
spot = false
on_demand_percent = 5

[service.stream-head.instance]
family = "r5"
size = "xlarge"
count = 1
disk = 30
spot = false
cache_disk = 0
on_demand_percent = 25

[pool.merge-peer0.instance]
service = "merge-peer"
count = 1
family = "r5"
size = "xlarge"
disk = 30
spot = false
cache_disk = 0
on_demand_percent = 25

.........

Components within the TOML fall into two categories, Service Components and (worker) Pool components. Pools are typically the components that are required to scale to individual workloads (e.g. where you want two sets of "query-peer" workers, one for the DevOps team and one for everyone else). Service components can not be scaled into separate pools, but can still be scaled horizontally in number and vertically in size.

Component

Classification

Can use Pools

alter-head
bastion
config
intake-misc
grafana
head
rds
stream-head
prometheus
superset
web
zookeeper

service

No

query-peer
merge-peer
kafka-peer
batch-peer
alter-peer

pool

Yes

Defaults

On initial build Hydrolix specifies default Service and (worker) Pool components. These defaults form the basic deployment of Hydrolix and use a hierarchy structure to override certain settings dependent on the component.

The structure of the TOML follows a hierarchical format where components inherit their settings from the parent. This works as follows:

service.default.instance
  |--service.{{service-type}}.instance
  |--pool_default.{{service-type}}.instance
       |--pool.{{pool-name}}.instance
           service: "<service-name>"

Name

Description

service.default.instance

Top level for both pools and services.

service.{{service-type}}.instance

Defines the specific service component, inherits from the service.default.instance.

pool_default.{{service-type}}.instance

Specifies default settings for the type of pool.

pool.{{pool-name}}.instance

Specifies the settings for an individual pool. Pools inherit their settinges from the pool_default.{{service-type}}.instance. The Pool setting service: is used to match the service-type of the parent.

This structure allows the user to specify different server parameters for different pools . For example, the below shows 2 different pools of query servers (rnd and devops) that pull different settings from different levels and overriding others.

[service.default.instance]
family = "t2"
size = "micro"
count = 1
disk = 30
cache_disk = 0
spot = false
on_demand_percent = 5

[pool_default.peer.instance]
family = "c5n"
size = "4xlarge"
cache_disk = 24
count=3

[pool.rnd.instance]
service = "peer"
count = 5
cache_disk = 24

[pool.devops.instance]
service = "peer"
count = 10
size = "9xlarge"
family = "c5n"
spot = true
cache_disk = 24
on_demand_percent = 25

TOML Parameters

The following settings are available for each component.

Parameter

Description

Example

service

The type of service the component will be used for. (Only used with (worker) Pools).

"kafka-peer", "peer", "merge-peer"

count

How many instances to be used for a service. If auto scaling is to be used (query-peer only) either a min or max can be provided (e.g. 2-5) or a min, desired and max can be specified e.g. (2-5-10).

1, 2-5, 2-5-10

family

Family for the instances to be deployed.

AWS: r5, c5n, t5 etc.

size

Size of the instance to be deployed.

AWS: Small, Medium, Large, 9xlarge etc

disk

The size of the EBS volume to be utilized by the instance.

30

spot

The size of the cache_disk to be utilized (query-peer only).

30

on_demand_percent

Specifies the number of on-demand % that should be used when using spot instances.

25

📘

Note: AWS - on_demand_percent

Note that AWS rounds up so any on_demand_percent greater than 0.

Adding Pools

To create a pool using TOML simply add to the file. For example if you wanted to create a new query pool cluster with 3 servers of c5n.9xlarge for the Support team you would add the following:

[pool.support.instance]
service = "peer"
count = 3
family = c5n
size = 9xlarge

Deleting Pools.

To remove components from the infrastructure the easiest method is to set the component to 0.

Where a Pool has been created this can be removed by removing the parameters from within the TOML and the .instance descriptor.

For example to remove the following pool:

[pool.support.instance]
service = "peer"
count = 3
family = c5n
size = 9xlarge

Simply update the TOML to:

[pool.support]

This will remove this pool from the cluster.

🚧

Default Services/Pools

Default pools should stay persistent and should only be set to count=0.


Did this page help you?