Hydrolix Command Line (HDXCTL)Reference

HDXCTL provides the primary control mechanism for building and managing a Hydrolix deployment. The HDXCTL CLI is a .pyz compressed file that Python is able to read and contains the instructions, binaries and templates needed to create a Hydrolix cluster.

Usage:

hdxctl [OPTIONS] SUB-COMMAND [ARGS]


Options Meaning Example Usage
--run-as-client unused unused
--region Target region for command hdxctl --region us-east-2 create-cluster hdxcli-ze26j666
--help Display hdxctl command line help hdxctl --help

Sub-Command Meaning
cloudformation-events List the events of the Cloudformation for a Cluster
clusters List all existing clusters with compute enabled
create-client-stack Create a new Hydrolix stack’s stateful components only.
create-cluster Create a new Hydrolix stack.
delete Delete a compute
delete-bootstrap Delete the stateful components within a cluster
goto unused - legacy functionality
instances List the compute instances currently in use and their function
list-client-ids List the client-ids currently in use and their region
route Used to change the hostname of a cluster from one compute cluster to another. Used as part of the upgrade process.
scale Used to scale a system component
smoketest Complete a basic test to see if the cluster is working
update Update Hydrolix binaries
version Show the current version, optional -a to show component versions as well

hdxctl --help

Displays help for hdxctl. This can also be used for sub-commands

Example:

hdxctl --help
Usage: hdx [OPTIONS] COMMAND [ARGS]...

Options:
  --run-as-client TEXT
  --region TEXT
  --help                Show this message and exit.

Commands:
	cloudformation-events
  clusters
  create-client-stack
  create-cluster
  delete
  delete-bootstrap
  goto
  instances
  list-client-ids
  route
  scale
  smoketest
  update
  version

cloudformation-events

Lists the events for a ClientId or a ClusterId. Using just the ClientId returns the events for the Core components. Using the ClientID and the ClusterId returns the events for the cluster componets.

Usage : Usage: hdxctl cloudformation-events [OPTIONS] CLIENT_ID [CLUSTER_ID]

Option Description
--help get help information on the command

Example:

$ hdxctl cloudformation-events hdxcli-c2tpmuym
--------------------------------  ---------------------------  -----------------------------------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-11-02 17:52:54.603000+00:00  hdxcli-c2tpmuym-self-deploy  CREATE_IN_PROGRESS                   User Initiated
2020-11-02 17:52:57.766000+00:00  ClientBucket                 CREATE_IN_PROGRESS                   -
2020-11-02 17:52:57.984000+00:00  SelfDeployRole              .................

$hdxctl cloudformation-events hdxcli-c2tpmuym hdx-qngq4obs
--------------------------------  ---------------------------------  -----------------------------------  ---------------------------
2020-11-02 18:14:07.639000+00:00  hdx-qngq4obs                       CREATE_IN_PROGRESS                   User Initiated
2020-11-02 18:14:14.871000+00:00  hdx-qngq4obs                       CREATE_IN_PROGRESS                   Transformation succeeded
.........

clusters

Lists and synchronizes all available clusters and synchronize hdxctl with your existing clusters

Usage : hdxctl clusters [OPTIONS]

Option Description
--add / -no-add unused
--id unused
--sync / --no-sync synchronizes the command with the available clusters
--help get help information on the command

Example:

$ hdxctl clusters
CLIENT_ID        CLUSTER_ID    CREATED              HOST                    STATUS           WHO      REGION
---------------  ------------  -------------------  ---------------------  ---------------  -------  ---------
hdxcli-pmpudpqe  hdx-ex6bdtsn  2020-08-18 20:53:19  mysite.hydrolix.live.  UPDATE_COMPLETE  imauser  us-east-2

create-client-stack

Create a new Hydrolix stack’s stateful components only

Usage : hdxctl create-client-stack [OPTIONS] CLIENT_ID

Option Description
--vpc-cidr Specify an alternate CIDR block for the deployment
--stage-1 / --no-stage-1 unused
--full-hydrolix-access / --no-full-hydrolix-access unused
--enable-turbine-monitor / --no-enable-turbine-monitor unused
--batch-peer-threads specify the number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--listing-queue-timeout specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--listing-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--import-queue-timeout specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--import-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--reaper-queue-timeout unused
--tag Specify additional tags you may require on your infrastructure <tag name>:<tag value>. Note, reserved values are: aws:cloudformation:logical-id, aws:cloudformation:stack-id, HdxContact, HdxService, Name, HdxBudget, aws:cloudformation:stack-name and can not be changed, also tags are only additive and will replace any previous tags you have set e.g. if --tag a:x --tag b:y is added in a command and then later --tag b:z, the second command will change the b tag but also remove the a tag.
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--help get help information on the command

create-cluster

Creates the cluster using a supplied CLIENT_ID. Does not require create-client-stack used prior to a new install.

Usage : hdxctl --region REGION create-cluster [OPTIONS] CLIENT_ID

Note: Region is a required field for create-cluster.

Option Description
--vpc-cidr Specify an alternate CIDR block for the deployment
--wait / --no-wait Have the client watch the command execute. Information is supplied as STDOUT
--environ / --no-environ unused
--full-hydrolix-access / --no-full-hydrolix-access unused
--enable-turbine-monitor unused
--batch-peer-threads specify the number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--listing-queue-timeout specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--listing-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--import-queue-timeout specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--import-max-receive-count unused
--reaper-queue-timeout unused
--tag Specify additional tags you may require on your infrastructure <tag name>:<tag value>. Note, reserved values are: aws:cloudformation:logical-id, aws:cloudformation:stack-id, HdxContact, HdxService, Name, HdxBudget, aws:cloudformation:stack-name and can not be changed, also tags are only additive and will replace any previous tags you have set e.g. if --tag a:x --tag b:y is added in a command and then later --tag b:z, the second command will change the b tag but also remove the a tag
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--help Displays the help text for the command

Example : The following will create a cluster in us-east-2 with the command outputting where it is within the build cycle.

hdxctl --region us-east-2 create-cluster hdxcli-u7mtxhmh --wait
creating hydrolix stack
initiated creation of hdx-nglnawnx
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds

delete

Deletes the compute components of the cluster using a supplied CLIENT_ID

Usage : hdxctl delete [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--wait / --no-wait Have the client watch the command execute. Information is supplied as STDOUT
--force / --no-force unused
--help Displays the help text for the command

Example :

hdxctl delete hdxcli-k754x5zs hdx-2xoq7xei --wait

delete-bootstrap

Deletes the stateful components of the stack.

Usage : hdxctl delete-bootstrap [OPTIONS] CLIENT_ID

Option Description
--wait / --no-wait Have the client watch the command execute. Information is supplied as STDOUT
--force / --no-force unused
--help Displays the help text for the command

Example :

hdxctl delete-bootstrap hdxcli-cflmkpyl --wait

instances

Gets a list of IP’s, the current state and the service type for a deployment

Usage : hdxctl instances [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Example :

hdxctl instances hdxcli-pmpudpqe hdx-ex6bdtsn
LAUNCH_TIME          SERVICE      IP            STATE    POOL
-------------------  -----------  ------------  -------  ------
2020-08-18 20:55:48  bastion      11.22.33.444  running
2020-08-25 12:26:34  batch-peer   10.0.3.98     running
2020-08-25 09:47:17  config       10.0.2.14     running
2020-08-25 09:47:32  head         10.0.13.203   running
2020-08-25 09:47:28  peer         10.0.12.177   running
2020-08-25 12:26:29  peer         10.0.11.95    running
2020-08-25 12:26:29  peer         10.0.9.229    running
2020-08-25 09:47:31  stream-head  10.0.15.167   running
2020-08-25 09:47:25  stream-peer  10.0.13.77    running
2020-08-25 09:47:20  ui           10.0.2.249    running
2020-08-18 20:55:48  zookeeper    10.0.3.231    running

list-client-ids

lists the Client ID’s attributed to you and the region they are aligned with.

Usage : hdx list-client-ids [OPTIONS]

Option Description
--help Displays the help text for the command

Example :

hdx list-client-ids
CLIENT_ID        REGION
---------------  ---------
hdxcli-dasfd243  eu-west-2
hdxcli-cn6nad32  us-west-2
hdxcli-puwas2fa  us-east-2

route

Switch the hostname to point to a different compute cluster. See Updating your Hydrxolix deployment for usage scenario.

Usage : hdx route [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Example :

hdxctl route hdxcli-puwas2fa  hdx-zuxr7yt6

scale

Scale components of the cluster. Using the command without any options will provide a current state list of the Auto-scaling group.

Usage : hdxctl scale [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--edit / --no-edit unused
--from-file load a configuration from a file. See the Advanced HDXCTL for more information.
--off / --no-off Turn off all except required stateful components.
--emit-toml / --no-emit-toml display the toml in STDOUT
--minimal / --no-minimal scale the stack to a minimal state with all components at a minimum level.
--update-ok Allow a cluster to be updated if the HDXCTL version doesnt match the cluster version.
--head-count specify the number of query heads to use, a minimum of 1 is required to query the infrastructure.
--head-instance-type change the type and class of the query head (e.g. c5n.xlarge).
--head-disk specify the amount of disk (EBS) you wish to use on the query head. Note this is for caching puroposes. Recommended to be kept as default.
--query-peer-count specify the number of query peers, a minimum of 1 is required to query the infrastructure.
--query-peer-instance-type change the type and class of the query peer (e.g. c5n.2xlarge).
--query-peer-disk specify the amount of disk (EBS) you wish to use on the query peer. Note this is for caching puroposes. Recommended to be kept as default.
--stream-head-count specify the number of stream heads to use, a minimum of 1 is required to use streaming ingest.
--stream-head-instance-type change the type and class of the stream head (e.g. m5.large).
--stream-head-disk specify the amount of disk (EBS) you wish to use on the stream head. Recommended to be kept as default.
--stream-peer-count specify the number of query peers, a minimum of 1 is required to use streaming ingest.
--stream-peer-instance-type change the type and class of the stream peer (e.g. m5.large).
--stream-peer-disk specify the amount of disk (EBS) you wish to use on the stream peer. Recommended to be kept as default.
--batch-peer-count specify the number of batch peers, a minimum of 1 is required to use batch ingest.
--batch-peer-instance-type change the type and class of the batch peer (e.g. m5.large).
--batch-peer-disk specify the amount of disk (EBS) you wish to use on the batch peer. Recommended to be kept as default.
--ui-count specify the number of ui servers. Recommended to be kept as default.
--ui-instance-type change the type and class of the ui server. Recommended to be kept as default.
--ui-disk specify the amount of disk (EBS) you wish to use on the ui server. Recommended to be kept as default.
--config-count specify the number of config servers. Recommended to be kept as default.
--config-instance-type change the type and class of the config server. Recommended to be kept as default.
--config-disk specify the amount of disk (EBS) you wish to use on the config server. Recommended to be kept as default.
--grafana-count specify the number of grafana servers you would like to run. Deploys as 0
--grafana-instance-type specify the instance type of Grafana serverss you would like to run.
--grafana-disk specify the size of disk for the Grafana server/s you would like to run
--help Displays the help text for the command

Note: You will note that settings for the batch-head appeart to be missing. This is due to batch ingest using Lambda as the ingest head and so does not have a count, type or disk option.

Example :

hdxctl scale hdxcli-cflmkpyl hdx-zgnswsoi --query-head-count 1 --query-peer-instance-type c5n.2xlarge --query-peer-count 5

hdxctl scale hdxcli-pmpudpqe hdx-ex6bdtsn
SERVICE        COUNT  FAMILY    SIZE       DISK
-----------  -------  --------  -------  ------
batch-peer         0  r5        2xlarge      30
config             0  t2        micro        30
query-head         0  c5n       xlarge       30
query-peer         0  c5n       4xlarge     100
stream-head        0  m5        xlarge       30
stream-peer        0  m5        xlarge       30
ui                 0  t2        micro        30

smoketest

Completes a basic test of the system to ingest and query some data.

Usage : hdxctl smoketest [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Note: Currently this functionality is broken, if you have concerns on your deployment please contact your Hydrolix representative

update

Used to update the version of the Hydrolix stack being used.

Usage : hdxctl update [OPTIONS] CLIENT_ID

Option Description
--enable-turbine-monitor / --no-enable-turbine-monitor unused
--batch-peer-threads specify the number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--listing-queue-timeout specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--listing-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--import-queue-timeout specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--import-max-receive-count unused
--reaper-queue-timeout unused
--tag Specify additional tags you may require on your infrastructure <tag name>:<tag value>. Note, reserved values are: aws:cloudformation:logical-id, aws:cloudformation:stack-id, HdxContact, HdxService, Name, HdxBudget, aws:cloudformation:stack-name and can not be changed, also tags are only additive and will replace any previous tags you have set e.g. if --tag a:x --tag b:y is added in a command and then later --tag b:z, the second command will change the b tag but also remove the a tag.
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--help Displays the help text for the command

version

Displays the version of the HDXCTL command you are using.

Usage : hdxctl version [OPTIONS]

Option Description
-a, --all / -A, --no-all Displays all the version information for the stack
--help Displays the help text for the command

Example :

hdxctl version -a
-----------------------  --------
SHARED_VERSION           78bce90d
UI_VERSION               80ecc715
CONFIG_VERSION           78bce90d
INTAKE_VERSION           827f5c9b
TURBINE_VERSION          1e0a0f12
LAMBDA_VERSION           a38b446d
CLI_DNS_SERVICE_VERSION  a38b446d
SELF_DEPLOY_VERSION      41cc2d2f
MACHINE_VERSION          41cc2d2f
HUMAN_VERSION            v1.10.4
GRAFANA_VERSION          a38b446d
-----------------------  --------