Create Cluster!

It's happening!

To deploy the platform into your cloud account you will be using the hdxctl tool you installed earlier with the use of a sub-command called create-cluster.

First time use of the hdxctl create-cluster command will create all the components needed and scale the system to a basic level. The command will require the AWS Region and the Hydrolix client_id you registered for earlier.

If this is the first time you have run the command you will also be asked for an email address, this is so you can login into the Configuration API and the Portal, simply add a working email address.


hdxctl --region <region> create-cluster <client_Id> --wait

$ hdxctl --region eu-west-1 create-cluster hdxcli-12345678 --wait
creating self_deploy stack
Waiting for stack creation sleeping 5 seconds
Waiting for stack creation sleeping 5 seconds
creating hydrolix stack
initiated creation of hdx-abcdefgh
Waiting for stack update; sleeping 100 seconds
Waiting for stack update; sleeping 100 seconds
stack creation complete
[email protected]:~$

The hdxctl CLI uses predefined AWS Cloudformation templates and a lambda to build the stack. All you need to do is sit back and wait.


To learn more about what components are created take a look at the platform section.

To see the status of your cluster (if you haven't used the --wait flag) you can use the following command, this will show the status of your cluster build:


hdxctl clusters --sync

$ hdxctl clusters --sync
CLIENT_ID        CLUSTER_ID    CREATED              HOST                           STATUS           WHO      REGION
---------------  ------------  -------------------  -----------------------------  ---------------  -------  ---------
hdxcli-12345678  hdx-abcdefgh  2020-08-20 09:57:17  CREATE_COMPLETE  user1  eu-west-1

The clusters command is useful as it can provide information on the overall state of updates and scaling changes you may have implemented. In this instance, CREATE_COMPLETE means everything has been created successfully.


If you are having problems with the install, this usually comes in the form of permissions to deploy components. To see where the deployment has failed look at the Cloud Formation console under the events for the two Stacks created.

We can now take a look at the current scale of the cluster:


hdxctl scale <client_id> <cluster_id>

$ hdxctl scale hdxcli-12345678 hdx-abcdefgh
--------------  -------  --------  ------  ------
bastion               1  t2        micro       30
batch-peer0           1  r5        xlarge      30
grafana               0  t2        small       30
head                  1  c5n       xlarge      30
intake-misc           1  t3        small       30
merge-peer0           1  r5        xlarge      30
prometheus            0  t2        medium      50
query-peer0           1  c5n       xlarge      30
stream-head           1  r5        xlarge      30
stream-peer0          1  r5        xlarge      30
superset              0  t2        medium      30
web                   1  t3        small       30
zookeeper             3  t2        medium      30

If you also check in your cloud console you should see you have a bunch of EC2 instances running a d a new S3 bucket named after your client_id


If for some reason you seeing all your counts as 0 you can use this command to get it to a minimal state.

hdxctl scale --minimal <client_id> <cluster_id>

Now, we need to get Access..... onward!