logo
Product
Pricing
Docs
Blog
Try Now
Contact Us
No results found.
Getting Started
Create a Cluster
Scaling a Cluster
Start Ingesting Data
Update Hydrolix
Sizing Guidelines
Downloads
HDX Installer
HDX Reader
Release Notes
Guide
Architecture
Interoperability
Working with Statistical Aggregations
Working with Fastly logging
Transforms
Data Types
API Reference
hdxctl Reference
Tools, Tips & Tricks
Data Privacy
Data Processing Addendum
Data Subject Access Request Form
Data Subprocessors
Privacy Statement
Legal
Hydrolix Terms of Service

HDXCTL (CLI) Reference

HDXCTL provides the primary control mechanism for building and managing a Hydrolix deployment. The HDXCTL CLI is a .pyz compressed file that Python is able to read and contains the instructions, binaries and templates needed to create a Hydrolix cluster.

Usage:

hdxctl [OPTIONS] SUB-COMMAND [ARGS]


Options Meaning Example Usage
--run-as-client unused unused
--region Target region for command hdxctl --region us-east-2 create-cluster hdxcli-ze26j666
--help Display hdxctl command line help hdxctl --help

Sub-Command Meaning
cloudformation-events List the events of the Cloudformation for a Cluster
clusters List all existing clusters with compute enabled
create-client-stack Create a new Hydrolix stack’s stateful components only.
create-cluster Create a new Hydrolix stack.
delete Delete a compute
delete-bootstrap Delete the stateful components within a cluster
goto unused - legacy functionality
instances List the compute instances currently in use and their function
list-client-ids List the client-ids currently in use and their region
route Used to change the hostname of a cluster from one compute cluster to another. Used as part of the upgrade process.
scale Used to scale a system component
smoketest Complete a basic test to see if the cluster is working
update Update Hydrolix binaries
version Show the current version, optional -a to show component versions as well

hdxctl --help

Displays help for hdxctl. This can also be used for sub-commands

Example:

hdxctl --help
Usage: hdx [OPTIONS] COMMAND [ARGS]...

Options:
  --run-as-client TEXT
  --region TEXT
  --help                Show this message and exit.

Commands:
	cloudformation-events
  clusters
  create-client-stack
  create-cluster
  delete
  delete-bootstrap
  goto
  instances
  list-client-ids
  route
  scale
  smoketest
  update
  version

cloudformation-events

Lists the events for a ClientId or a ClusterId. Using just the ClientId returns the events for the Core components. Using the ClientID and the ClusterId returns the events for the cluster componets.

Usage : Usage: hdxctl cloudformation-events [OPTIONS] CLIENT_ID [CLUSTER_ID]

Option Description
--help get help information on the command

Example:

$ hdxctl cloudformation-events hdxcli-c2tpmuym
--------------------------------  ---------------------------  -----------------------------------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-11-02 17:52:54.603000+00:00  hdxcli-c2tpmuym-self-deploy  CREATE_IN_PROGRESS                   User Initiated
2020-11-02 17:52:57.766000+00:00  ClientBucket                 CREATE_IN_PROGRESS                   -
2020-11-02 17:52:57.984000+00:00  SelfDeployRole              .................

$hdxctl cloudformation-events hdxcli-c2tpmuym hdx-qngq4obs
--------------------------------  ---------------------------------  -----------------------------------  ---------------------------
2020-11-02 18:14:07.639000+00:00  hdx-qngq4obs                       CREATE_IN_PROGRESS                   User Initiated
2020-11-02 18:14:14.871000+00:00  hdx-qngq4obs                       CREATE_IN_PROGRESS                   Transformation succeeded
.........

clusters

Lists and synchronizes all available clusters and synchronize hdxctl with your existing clusters

Usage : hdxctl clusters [OPTIONS]

Option Description
--add / -no-add unused
--id unused
--sync / --no-sync synchronizes the command with the available clusters
--help get help information on the command

Example:

$ hdxctl clusters
CLIENT_ID        CLUSTER_ID    CREATED              HOST                    STATUS           WHO      REGION
---------------  ------------  -------------------  ---------------------  ---------------  -------  ---------
hdxcli-pmpudpqe  hdx-ex6bdtsn  2020-08-18 20:53:19  mysite.hydrolix.live.  UPDATE_COMPLETE  imauser  us-east-2

create-client-stack

Create a new Hydrolix stack’s stateful components only

Usage : hdxctl create-client-stack [OPTIONS] CLIENT_ID

Option Description
--vpc-cidr Specify an alternate CIDR block for the deployment
--stage-1 / --no-stage-1 unused
--full-hydrolix-access / --no-full-hydrolix-access unused
--enable-turbine-monitor / --no-enable-turbine-monitor unused
--batch-peer-threads specify the number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--listing-queue-timeout specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--listing-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--import-queue-timeout specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--import-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--reaper-queue-timeout unused
--tag Specify additional tags you may require on your infrastructure <tag name>:<tag value>. Note, reserved values are: aws:cloudformation:logical-id, aws:cloudformation:stack-id, HdxContact, HdxService, Name, HdxBudget, aws:cloudformation:stack-name and can not be changed, also tags are only additive and will replace any previous tags you have set e.g. if --tag a:x --tag b:y is added in a command and then later --tag b:z, the second command will change the b tag but also remove the a tag.
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--help get help information on the command

create-cluster

Creates the cluster using a supplied CLIENT_ID. Does not require create-client-stack used prior to a new install.

Usage : hdxctl --region REGION create-cluster [OPTIONS] CLIENT_ID

Note: Region is a required field for create-cluster.

Option Description
--vpc-cidr Specify an alternate CIDR block for the deployment
--wait / --no-wait Have the client watch the command execute. Information is supplied as STDOUT
--environ / --no-environ unused
--full-hydrolix-access / --no-full-hydrolix-access unused
--enable-turbine-monitor unused
--batch-peer-threads specify the number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--listing-queue-timeout specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--listing-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--import-queue-timeout specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--import-max-receive-count unused
--reaper-queue-timeout unused
--tag Specify additional tags you may require on your infrastructure <tag name>:<tag value>. Note, reserved values are: aws:cloudformation:logical-id, aws:cloudformation:stack-id, HdxContact, HdxService, Name, HdxBudget, aws:cloudformation:stack-name and can not be changed, also tags are only additive and will replace any previous tags you have set e.g. if --tag a:x --tag b:y is added in a command and then later --tag b:z, the second command will change the b tag but also remove the a tag
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--help Displays the help text for the command

Example : The following will create a cluster in us-east-2 with the command outputting where it is within the build cycle.

hdxctl --region us-east-2 create-cluster hdxcli-u7mtxhmh --wait
creating hydrolix stack
initiated creation of hdx-nglnawnx
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds

delete

Deletes the compute components of the cluster using a supplied CLIENT_ID

Usage : hdxctl delete [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--wait / --no-wait Have the client watch the command execute. Information is supplied as STDOUT
--force / --no-force unused
--help Displays the help text for the command

Example :

hdxctl delete hdxcli-k754x5zs hdx-2xoq7xei --wait

delete-bootstrap

Deletes the stateful components of the stack.

Usage : hdxctl delete-bootstrap [OPTIONS] CLIENT_ID

Option Description
--wait / --no-wait Have the client watch the command execute. Information is supplied as STDOUT
--force / --no-force unused
--help Displays the help text for the command

Example :

hdxctl delete-bootstrap hdxcli-cflmkpyl --wait

instances

Gets a list of IP’s, the current state and the service type for a deployment

Usage : hdxctl instances [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Example :

hdxctl instances hdxcli-pmpudpqe hdx-ex6bdtsn
LAUNCH_TIME          SERVICE      IP            STATE    POOL
-------------------  -----------  ------------  -------  ------
2020-08-18 20:55:48  bastion      11.22.33.444  running
2020-08-25 12:26:34  batch-peer   10.0.3.98     running
2020-08-25 09:47:17  config       10.0.2.14     running
2020-08-25 09:47:32  head         10.0.13.203   running
2020-08-25 09:47:28  peer         10.0.12.177   running
2020-08-25 12:26:29  peer         10.0.11.95    running
2020-08-25 12:26:29  peer         10.0.9.229    running
2020-08-25 09:47:31  stream-head  10.0.15.167   running
2020-08-25 09:47:25  stream-peer  10.0.13.77    running
2020-08-25 09:47:20  ui           10.0.2.249    running
2020-08-18 20:55:48  zookeeper    10.0.3.231    running

list-client-ids

lists the Client ID’s attributed to you and the region they are aligned with.

Usage : hdx list-client-ids [OPTIONS]

Option Description
--help Displays the help text for the command

Example :

hdx list-client-ids
CLIENT_ID        REGION
---------------  ---------
hdxcli-dasfd243  eu-west-2
hdxcli-cn6nad32  us-west-2
hdxcli-puwas2fa  us-east-2

route

Switch the hostname to point to a different compute cluster. See Updating your Hydrxolix deployment for usage scenario.

Usage : hdx route [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Example :

hdxctl route hdxcli-puwas2fa  hdx-zuxr7yt6

scale

Scale components of the cluster. Using the command without any options will provide a current state list of the Auto-scaling group.

Usage : hdxctl scale [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--edit / --no-edit unused
--from-file load a configuration from a file. See the Advanced HDXCTL for more information.
--off / --no-off Turn off all except required stateful components.
--emit-toml / --no-emit-toml display the toml in STDOUT
--minimal / --no-minimal scale the stack to a minimal state with all components at a minimum level.
--update-ok Allow a cluster to be updated if the HDXCTL version doesnt match the cluster version.
--head-count specify the number of query heads to use, a minimum of 1 is required to query the infrastructure.
--head-instance-type change the type and class of the query head (e.g. c5n.xlarge).
--head-disk specify the amount of disk (EBS) you wish to use on the query head. Note this is for caching puroposes. Recommended to be kept as default.
--query-peer-count specify the number of query peers, a minimum of 1 is required to query the infrastructure.
--query-peer-instance-type change the type and class of the query peer (e.g. c5n.2xlarge).
--query-peer-disk specify the amount of disk (EBS) you wish to use on the query peer. Note this is for caching puroposes. Recommended to be kept as default.
--stream-head-count specify the number of stream heads to use, a minimum of 1 is required to use streaming ingest.
--stream-head-instance-type change the type and class of the stream head (e.g. m5.large).
--stream-head-disk specify the amount of disk (EBS) you wish to use on the stream head. Recommended to be kept as default.
--stream-peer-count specify the number of query peers, a minimum of 1 is required to use streaming ingest.
--stream-peer-instance-type change the type and class of the stream peer (e.g. m5.large).
--stream-peer-disk specify the amount of disk (EBS) you wish to use on the stream peer. Recommended to be kept as default.
--batch-peer-count specify the number of batch peers, a minimum of 1 is required to use batch ingest.
--batch-peer-instance-type change the type and class of the batch peer (e.g. m5.large).
--batch-peer-disk specify the amount of disk (EBS) you wish to use on the batch peer. Recommended to be kept as default.
--ui-count specify the number of ui servers. Recommended to be kept as default.
--ui-instance-type change the type and class of the ui server. Recommended to be kept as default.
--ui-disk specify the amount of disk (EBS) you wish to use on the ui server. Recommended to be kept as default.
--config-count specify the number of config servers. Recommended to be kept as default.
--config-instance-type change the type and class of the config server. Recommended to be kept as default.
--config-disk specify the amount of disk (EBS) you wish to use on the config server. Recommended to be kept as default.
--grafana-count specify the number of grafana servers you would like to run. Deploys as 0
--grafana-instance-type specify the instance type of Grafana serverss you would like to run.
--grafana-disk specify the size of disk for the Grafana server/s you would like to run
--help Displays the help text for the command

Note: You will note that settings for the batch-head appeart to be missing. This is due to batch ingest using Lambda as the ingest head and so does not have a count, type or disk option.

Example :

hdxctl scale hdxcli-cflmkpyl hdx-zgnswsoi --query-head-count 1 --query-peer-instance-type c5n.2xlarge --query-peer-count 5

hdxctl scale hdxcli-pmpudpqe hdx-ex6bdtsn
SERVICE        COUNT  FAMILY    SIZE       DISK
-----------  -------  --------  -------  ------
batch-peer         0  r5        2xlarge      30
config             0  t2        micro        30
query-head         0  c5n       xlarge       30
query-peer         0  c5n       4xlarge     100
stream-head        0  m5        xlarge       30
stream-peer        0  m5        xlarge       30
ui                 0  t2        micro        30

smoketest

Completes a basic test of the system to ingest and query some data.

Usage : hdxctl smoketest [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Note: Currently this functionality is broken, if you have concerns on your deployment please contact your Hydrolix representative

update

Used to update the version of the Hydrolix stack being used.

Usage : hdxctl update [OPTIONS] CLIENT_ID

Option Description
--enable-turbine-monitor / --no-enable-turbine-monitor unused
--batch-peer-threads specify the number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--listing-queue-timeout specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--listing-max-receive-count specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--import-queue-timeout specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--import-max-receive-count unused
--reaper-queue-timeout unused
--tag Specify additional tags you may require on your infrastructure <tag name>:<tag value>. Note, reserved values are: aws:cloudformation:logical-id, aws:cloudformation:stack-id, HdxContact, HdxService, Name, HdxBudget, aws:cloudformation:stack-name and can not be changed, also tags are only additive and will replace any previous tags you have set e.g. if --tag a:x --tag b:y is added in a command and then later --tag b:z, the second command will change the b tag but also remove the a tag.
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--help Displays the help text for the command

version

Displays the version of the HDXCTL command you are using.

Usage : hdxctl version [OPTIONS]

Option Description
-a, --all / -A, --no-all Displays all the version information for the stack
--help Displays the help text for the command

Example :

hdxctl version -a
-----------------------  --------
SHARED_VERSION           78bce90d
UI_VERSION               80ecc715
CONFIG_VERSION           78bce90d
INTAKE_VERSION           827f5c9b
TURBINE_VERSION          1e0a0f12
LAMBDA_VERSION           a38b446d
CLI_DNS_SERVICE_VERSION  a38b446d
SELF_DEPLOY_VERSION      41cc2d2f
MACHINE_VERSION          41cc2d2f
HUMAN_VERSION            v1.10.4
GRAFANA_VERSION          a38b446d
-----------------------  --------