Using hdxctl, the Hydrolix CLI

The program hdxctl lets you build and manage your Hydrolix clusters from the command line.

Usage

hdxctl [OPTIONS] command [ARGS]

Options

–help

Displays summary documentation for using hdxctl and exits.

–region

Specify the cloud-service region to apply to the command.

For example, to specify to hdxctl that it should [create a new cluster] in AWS’s us-east-2 region, using the Client ID “hdxcli-example123”:

$ hdxctl --region us-east-2 create-cluster hdxcli-example123

Using the --region option sets the value you provide as a default. Commands that require you to specify a region will make use of this default if you subsequently run hdxctl without setting this option.

Commands

Summary

Command Purpose
cloudformation-events Lists the CloudFormation events recorded for a given cluster or client ID.
clusters Lists your clusters.
create-client-stack Creates a new Hyrdolix stack’s, but limited to stateful components only.
create-cluster Creates a new Hydrolix stack, in full.
delete Deletes the compute components of a given cluster.
delete-bootstrap Deletes the stateful components of a given cluster.
get-license Creates a new Hydrolix license.
goto Connects to a Hydrolix component via SSH.
instances Lists the compute instances currently in use.
list-client-ids Lists the client IDs currently in use.
nat-gateway-ip Displays your clusters’ externally visible IP address.
route Associates a given client ID’s hostname with a given cluster.
scale Displays or sets the resource use of a cluster’s components.
scale-db Scale the Catalog Database component.
smoketest Runs basic-functionality tests on a given cluster.
update Updates the Hydrolix software run by a given client ID.
version Displays the currently installed version of Hydrolix.

cloudformation-events

Lists the events for a client ID or a cluster ID. Using just the client ID returns the events for the core components. Using the client ID and the cluster ID returns the events for the cluster componets.

Usage

hdxctl cloudformation-events [OPTIONS] CLIENT_ID [CLUSTER_ID]

Options

Option Description
--help Displays summary documentation for this command, and exits.

Example

$ hdxctl cloudformation-events hdxcli-c2tpmuym
--------------------------------  ---------------------------  -----------------------------------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2020-11-02 17:52:54.603000+00:00  hdxcli-c2tpmuym-self-deploy  CREATE_IN_PROGRESS                   User Initiated
2020-11-02 17:52:57.766000+00:00  ClientBucket                 CREATE_IN_PROGRESS                   -
2020-11-02 17:52:57.984000+00:00  SelfDeployRole              .................

$hdxctl cloudformation-events hdxcli-c2tpmuym hdx-qngq4obs
--------------------------------  ---------------------------------  -----------------------------------  ---------------------------
2020-11-02 18:14:07.639000+00:00  hdx-qngq4obs                       CREATE_IN_PROGRESS                   User Initiated
2020-11-02 18:14:14.871000+00:00  hdx-qngq4obs                       CREATE_IN_PROGRESS                   Transformation succeeded
.........

clusters

Lists your clusters.

Note that this command uses a local cache for efficiency. To reload the cache with your clusters’ most recent metadata, run this command with the --sync option.

Usage

hdxctl clusters [OPTIONS]

Option Description
--sync Reloads metadata from your clusters prior to display.
--help Displays summary documentation for this command, and exits.

Example:

$ hdxctl clusters
CLIENT_ID        CLUSTER_ID    CREATED              HOST                    STATUS           WHO      REGION
---------------  ------------  -------------------  ---------------------  ---------------  -------  ---------
hdxcli-pmpudpqe  hdx-ex6bdtsn  2020-08-18 20:53:19  mysite.hydrolix.live.  UPDATE_COMPLETE  imauser  us-east-2

create-client-stack

Creates a new Hydrolix stack’s stateful components only.

Usage

hdxctl create-client-stack [OPTIONS] CLIENT_ID

Options

Option Description
--batch-peer-threads The number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
--enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
--enable-turbine-monitor Allow query components to monitor the Hydrolix query engine, restarting it if it hangs.
--help Displays summary documentation for this command, and exits.
--import-max-receive-count The maximum number of items in the SQS queue. Recommended to be kept as default.
--import-queue-timeout The time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup) for incoming connections. IPs are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--listing-max-receive-count The maximum number of items in the SQS queue. Recommended to be kept as default.
--listing-queue-timeout The time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--tag Additional tags you may require on your infrastructure, in the format TAG-NAME:TAG-VALUE. See also further notes about tags, below.
--vpc-cidr An alternate CIDR block for the deployment.

create-cluster

Creates the cluster using a supplied CLIENT_ID. Does not require create-client-stack used prior to a new install.

Usage

hdxctl --region REGION create-cluster [OPTIONS] CLIENT_ID

Option Description
--autoingest-max-receive-count specify the maximum message size for the Autoingest Queue. Default 256KB.
--autoingest-queue-timeout specify the maximum message retention period for the Autoingest Queue. Default 4 Days
--aws-ssh-key-name Add an AWS defined Key Pair to the authorized keys of a deployment. Allows on-box access.
--batch-bucket-kms-arn Allow Hydrolix servers to decrypt a source bucket where a customer defined KMS key is required. Takes the ARN
--batch-peer-threads Specify the number of vCPU’s a batch-peer should use for import jobs.
--bucket-allowlist Enables the architecture to access other buckets. For example: --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. Any update will overwrite previous configurations.
--ec2-detailed-monitoring Turns off additional monitoring for Hydrolix EC2 components. Default true.
--enable-query-auth Enable query authorisation for requests to the query end-point. Currently a place-holder and not in use.
--enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
--enable-turbine-monitor Allow query components to monitor the Hydrolix query engine, restarting it if it hangs.
--help Displays summary documentation for this command, and exits.
--import-max-receive-count specifythe maximum message size for the Recieve Queue. Recommended to be kept as default. Default 256KB
--import-queue-timeout Specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--kafka-tls-ca Allows the addition of a TLS Certificate Authority (CA) for mutual identification of Hydrolix Kafka ingest. PEM Format
--kafka-tls-cert Allows the addition of a TLS Certificate for mutual identification of Hydrolix Kafka ingest. PEM format
--kafka-tls-key Allows the addition of a TLS Key for mutual identification of Hydrolix Kafka ingest. PEM format
--listing-max-receive-count Specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--listing-queue-timeout Specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--merge-interval specify the interval for the Merge process to trigger. Recommended to be kept as default.
--merge-max-receive-count specify the maximum message size for the Merge Queue. Recommended to be kept as default. Default 256KB.
--merge-queue-timeout specify the maximum message retention period for the Merge Queue. Recommended to be kept as default. Default 4 Days.
--reaper-queue-timeout specify the maximum message retention period for the Reaper Queue. Recommended to be kept as default. Default 4 Days
--ssh-authorized-keys Allows the provision of a file in the format of .ssh/authorized_keys to be appended to all .ssh/authorized_keys files in the deployed infrastructure.
--stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--tag Tags to apply to this cluster, in the format TAG-NAME:TAG-VALUE. See also further notes about tags, below.
--vpc-cidr An alternate CIDR block for the deployment.
--wait Have the client watch the command execute. Information is supplied as STDOUT

Example

The following will create a cluster in us-east-2 with the command outputting where it is within the build cycle.

$ hdxctl --region us-east-2 create-cluster hdxcli-u7mtxhmh --wait

creating hydrolix stack
initiated creation of hdx-nglnawnx
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds
hdx-nglnawnx status: CREATE_IN_PROGRESS, sleeping 30 seconds

delete

Deletes the compute components of the cluster using a supplied CLIENT_ID

Usage

hdxctl delete [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--wait Have the client watch the command execute. Information is supplied as STDOUT
--help Displays summary documentation for this command, and exits.

Example

$ hdxctl delete hdxcli-k754x5zs hdx-2xoq7xei --wait

delete-bootstrap

Deletes the stateful components of the stack.

Usage

hdxctl delete-bootstrap [OPTIONS] CLIENT_ID

Option Description
--wait Have the client watch the command execute. Information is supplied as STDOUT
--help Displays summary documentation for this command, and exits.

Example

$ hdxctl delete-bootstrap hdxcli-cflmkpyl --wait

get-license

Creates a new Hydrolix license, with its own, new client ID. This works as a command-line alternative to obtaining a license through the hydrolix.io website.

All of this command’s arguments are required, and map to the field found on the license registration web form.

Usage

hdxctl get-license \
    --admin-email EMAIL \
    --organization ORG_NAME \
    --full-name NAME \
    --account-id AWS_ACCOUNT_ID \
    --host DESIRED_HYDROLIX_HOSTNAME \
    --region AWS_REGION \

Example

This example would attempt to create a new license whose clusters would run at the host “example.hydrolix.live”.

$ hdxctl get-license \
    --admin-email "alice@example.com" \
    --organization "Yoyodyne Propulsion Systems" \
    --full-name "Alice Nobody" \
    --account-id "123456789123" \
    --host "example" \
    --region "us-east-1" \

goto

Connects to a single component of a specified cluster via SSH.

If run with the -i option, this command instead displays the specified component’s local IP address and exits.

Connecting to components through this command requires setting SSH public keys on your client stack as a prerequisite step. For more information on this and other aspects of using this command, see Accessing components with SSH.

Usage

hdxctl goto [-i] CLIENT_ID CLUSTER_ID COMPONENT_NAME

Where COMPONENT_NAME is one of:

  • bastion
  • batch-peer
  • clickhouse
  • config
  • grafana
  • head
  • kafka-peer
  • merge-peer
  • peer
  • prometheus
  • stream-head
  • stream-peer
  • superset
  • ui
  • zookeeper

Example

Logging into a cluster’s UI component via SSH:

$ hdxctl goto client hdxcli-abc1234 hdx-xyz1234 ui

instances

Gets a list of IP’s, the current state and the service type for a deployment

Usage

hdxctl instances [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Example

$ hdxctl instances hdxcli-pmpudpqe hdx-ex6bdtsn

LAUNCH_TIME          SERVICE      IP            STATE    POOL
-------------------  -----------  ------------  -------  ------
2020-08-18 20:55:48  bastion      11.22.33.444  running
2020-08-25 12:26:34  batch-peer   10.0.3.98     running
2020-08-25 09:47:17  config       10.0.2.14     running
2020-08-25 09:47:32  head         10.0.13.203   running
2020-08-25 09:47:28  peer         10.0.12.177   running
2020-08-25 12:26:29  peer         10.0.11.95    running
2020-08-25 12:26:29  peer         10.0.9.229    running
2020-08-25 09:47:31  stream-head  10.0.15.167   running
2020-08-25 09:47:25  stream-peer  10.0.13.77    running
2020-08-25 09:47:20  ui           10.0.2.249    running
2020-08-18 20:55:48  zookeeper    10.0.3.231    running

list-client-ids

lists the Client ID’s attributed to you and the region they are aligned with.

Usage

hdxctl list-client-ids [OPTIONS]

Option Description
--help Displays the help text for the command

Example

$ hdxctl list-client-ids

CLIENT_ID        REGION
---------------  ---------
hdxcli-dasfd243  eu-west-2
hdxcli-cn6nad32  us-west-2
hdxcli-puwas2fa  us-east-2

nat-gateway-ip

Displays the single IP address that your various components present to the wider internet, from the perspective of an external recipient

Usage

hdxctl nat-gateway-ip CLIENT_ID

Example

$ hdxctl nat-gateway-ip hdxcli-example123

34.192.246.29

route

Switch the hostname to point to a different compute cluster. See Updating your Hydrxolix deployment for usage scenario.

Usage

hdxctl route [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

Example

$ hdxctl route hdxcli-puwas2fa  hdx-zuxr7yt6

scale

Scale components of the cluster. Using the command without any options will provide a current state list of the Auto-scaling group.

Usage

hdxctl scale [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--batch-peer-count Specify the number of batch peers, a minimum of 1 is required to use batch ingest.
--batch-peer-disk Specify the amount of disk (EBS) you wish to use on the batch peer. Recommended to be kept as default.
--batch-peer-instance-type Change the type and class of the batch peer (e.g. m5.large).
--config-count Specify the number of config servers. Recommended to be kept as default.
--config-disk Specify the amount of disk (EBS) you wish to use on the config server. Recommended to be kept as default.
--config-instance-type Change the type and class of the config server. Recommended to be kept as default.
--emit-toml Display the toml in STDOUT.
--from-file Load a configuration from a file. See the Advanced HDXCTL for more information.
--grafana-count Specify the number of grafana servers you would like to run. Deploys as 0
--grafana-disk Specify the size of disk for the Grafana server/s you would like to run
--grafana-instance-type Specify the instance type of Grafana serverss you would like to run.
--head-count Specify the number of query heads to use. A minimum of 1 is required to query the infrastructure.
--head-disk Specify the amount of disk (EBS) you wish to use on the query head. Note this is for caching puroposes. Recommended to be kept as default.
--head-instance-type Change the type and class of the query head (e.g. c5n.xlarge).
--help Displays the help text for the command
--minimal Scale the stack to a minimal state with all components at a minimum level.
--off Turn off all except required stateful components.
--query-peer-count Specify the number of query peers, a minimum of 1 is required to query the infrastructure.
--query-peer-disk Specify the amount of disk (EBS) you wish to use on the query peer. Note this is for caching puroposes. Recommended to be kept as default.
--query-peer-instance-type Change the type and class of the query peer (e.g. c5n.2xlarge).
--stream-head-count Specify the number of stream heads to use, a minimum of 1 is required to use streaming ingest.
--stream-head-disk Specify the amount of disk (EBS) you wish to use on the stream head. Recommended to be kept as default.
--stream-head-instance-type Change the type and class of the stream head (e.g. m5.large).
--stream-peer-count Specify the number of query peers, a minimum of 1 is required to use streaming ingest.
--stream-peer-disk Specify the amount of disk (EBS) you wish to use on the stream peer. Recommended to be kept as default.
--stream-peer-instance-type Change the type and class of the stream peer (e.g. m5.large).
--ui-count Specify the number of ui servers. Recommended to be kept as default.
--ui-disk Specify the amount of disk (EBS) you wish to use on the ui server. Recommended to be kept as default.
--ui-instance-type Change the type and class of the ui server. Recommended to be kept as default.
--update-ok Allow a cluster to be updated if the HDXCTL version doesnt match the cluster version.

Note: You will note that settings for the batch-head appeart to be missing. This is due to batch ingest using Lambda as the ingest head and so does not have a count, type or disk option.

Example

$ hdxctl scale hdxcli-cflmkpyl hdx-zgnswsoi --query-head-count 1 --query-peer-instance-type c5n.2xlarge --query-peer-count 5

hdxctl scale hdxcli-pmpudpqe hdx-ex6bdtsn
SERVICE        COUNT  FAMILY    SIZE       DISK
-----------  -------  --------  -------  ------
batch-peer         0  r5        2xlarge      30
config             0  t2        micro        30
query-head         0  c5n       xlarge       30
query-peer         0  c5n       4xlarge     100
stream-head        0  m5        xlarge       30
stream-peer        0  m5        xlarge       30
ui                 0  t2        micro        30

scale-db

Scale the RDS Catalog Database component of the cluster. Using the command without any options will provide a current state of the database. The catalog is a core component to the system. Downgrading of size should be done carefully. Any upgrade/downgrade needs to be completed off-line.

Usage

hdxctl scale-db [OPTIONS] CLIENT_ID

Option Description
--db-instance-type Update the instance type to be used.
--db-disk Update the instance’s disk size.
--help Displays the help text for the command.

Example

$ hdxctl scale-db hdxcli-ekmeho6e

Family: db.t2, Instance Size: medium, Disk Size: 30GB

smoketest

Completes a basic test of the system to ingest and query some data.

Usage

hdxctl smoketest [OPTIONS] CLIENT_ID CLUSTER_ID

Option Description
--help Displays the help text for the command

update

Used to update the version of the Hydrolix stack being used.

Usage

hdxctl update [OPTIONS] CLIENT_ID

Option Description
--autoingest-max-receive-count specify the maximum message size for the Autoingest Queue. Recommended to be kept as default. Default 256KB.
--autoingest-queue-timeout specify the maximum message retention period for the Autoingest Queue. Recommended to be kept as default. Default 4 Days
--aws-ssh-key-name Add an AWS defined Key Pair to the authorized keys of a deployment. Allows on-box access.
--batch-bucket-kms-arn Allow Hydrolix servers to decrypt a source bucket where a customer defined KMS key is required. Takes the ARN
--batch-peer-threads Specify the number of vCPU’s a batch-peer should use for import jobs. Recommended to be kept as default.
--bucket-allowlist Enables the architecture to access other buckets. Buckets names are provided as only the bucket name. For example --bucket-allowlist mybucket1 --bucket-allowlist anotherbucket. This is not additivie, any update will overwrite previous configurations.
--ec2-detailed-monitoring Turns off additional monitoring for Hydrolix EC2 components. Default true.
--enable-query-auth Enable query authorisation for requests to the query end-point. Currently a place-holder and not in use.
--enable-query-peer-hyperthreading Enable hyperthreading on the query peer. Default disabled.
--enable-turbine-monitor Allow query components to monitor the Hydrolix query engine, restarting it if it hangs.
--help Displays the help text for the command
--import-max-receive-count specifythe maximum message size for the Recieve Queue. Recommended to be kept as default. Default 256KB
--import-queue-timeout Specify the time for an individual job to timeout on the SQS queue. Recommended to be kept as default.
--ip-allowlist Sets IP allow lists on the appropriate security groups (BastionSecurityGroup and ELBSecurityGroup for incoming connections. IP’s are provided as CIDR formations. For example: --ip-allowlist 4.2.2.2/32 --ip-allowlist 8.8.8.0/24. Note: If an allow list doesn’t contain “0.0.0.0/0” then the ip /32 of the nat gateway will get added automatically. This is not additivie, any update will overwrite previous configurations.
--kafka-tls-ca Allows the addition of a TLS Certificate Authority (CA) for mutual identification of Hydrolix Kafka ingest. PEM Format
--kafka-tls-cert Allows the addition of a TLS Certificate for mutual identification of Hydrolix Kafka ingest. PEM format
--kafka-tls-key Allows the addition of a TLS Key for mutual identification of Hydrolix Kafka ingest. PEM format
--listing-max-receive-count Specify the maximum number of items in the SQS queue. Recommended to be kept as default.
--listing-queue-timeout Specify the time for an import job to timeout of the SQS queue. Recommended to be kept as default.
--merge-interval specify the interval for the Merge process to trigger. Recommended to be kept as default.
--merge-max-receive-count specify the maximum message size for the Merge Queue. Recommended to be kept as default. Default 256KB.
--merge-queue-timeout specify the maximum message retention period for the Merge Queue. Recommended to be kept as default. Default 4 Days.
--reaper-queue-timeout specify the maximum message retention period for the Reaper Queue. Recommended to be kept as default. Default 4 Days
--ssh-authorized-keys Allows the provision of a file in the format of .ssh/authorized_keys to be appended to all .ssh/authorized_keys files in the deployed infrastructure.
--stream-shard-count Alter the Ingest Streaming Shard count for Kinesis. Default 2
--tag Additional tags you may require on your infrastructure, in the format TAG-NAME:TAG-VALUE. See also further notes about tags, below.

version

Displays the version of the hdxctl program you are using.

Usage

hdxctl version [OPTIONS]

Option Description
-a, --all Displays all the version information for the stack
--help Displays the help text for the command

Example

$ hdxctl version -a

-----------------------  --------
SHARED_VERSION           b5de4e88
UI_VERSION               7605aa1e
CONFIG_VERSION           1b43fda6
KEYCLOAK_VERSION         15a561b0
INTAKE_VERSION           b5de4e88
TURBINE_VERSION          b78e7185
LAMBDA_VERSION           b5de4e88
CLI_DNS_SERVICE_VERSION  7181f171
SELF_DEPLOY_VERSION      b5de4e88
MACHINE_VERSION          b5de4e88
HUMAN_VERSION            v2.9.3
SECRETS_FROM_S3_VERSION  15a561b0
ENVIRON_VERSION          15a561b0
LAST_TAG_NAME            v2.9.3
LAST_TAG                 b5de4e88
TAG                      v2.9.3
GRAFANA_VERSION          e1f7907d
PROMETHEUS_VERSION       e1f7907d
SUPERSET_VERSION         e1f7907d
TERRAFORM_VERSION        f8011a06
-----------------------  --------

Further notes about tags

Using the --tag option with the update command will erase all tags previously set on the cluster prior to setting the newly specified ones.

Several tag names are reserved for Hydrolix’s internal use, and cannot be set via the --tag option:

  • aws:cloudformation:logical-id

  • aws:cloudformation:stack-id

  • HdxContact

  • HdxService

  • Name

  • HdxBudget

  • aws:cloudformation:stack-name