HDXCLI Commands

hdxcli

The hdxcli is a command line tool that can be used to manage certain aspects of your data platform. Primarily used as a CLI interface for the API, it can be used to script certain aspects as well as manage items such as project, table and transform settings.
The code is available on github, to install it you can use:

pip install hdxcli

Reference

hdxcli [OPTIONS] COMMAND [ARGS]

Options

OptionsPurpose
--helpDisplays summary documentation for using hdxcli and exits.
--profile PROFILENAMEPerform operation with a different profile.
(Default profile is 'default')
--project PROJECTNAMEExplicitly pass the project name. If one was
set it will be overridden.
--table TABLENAMEExplicitly pass the table name. If one was
set it will be overridden.
--transform TRANSFORMNAMEExplicitly pass the transform name. If one was
set it will be overridden, if none is given it'll use the default transform.
--job JOBNAMEPerform operation on the passed jobname
--function FUNCTIONNAMEPerform operation on the passed function
--dictionary DICTIONARYNAMEPerform operation on the passed dictionary
--passwordLogin password. If provided and the access token is expired, it will be used.

Commands

CommandPurpose
dictionaryDictionary-related operations
functionFunction-related operations
jobJob-related operations
profileProfile-related operations
projectProject-related operations
purgejobsPurge all batch jobs in your org
setSet project and or/table to apply subsequent commands on it
tableTable-related operations
transformTransform-related operations
unsetRemove any set projects/tables
versionPrint hdxcli version

Configuration file

hdxcli leverage a configuration file which reference your Hydrolix cluster environment.
The file is located here: ~/.hdx_cli/config.toml

[default]
username = "[email protected]"
hostname = "cluster.domain.net"
projectname = "hydro"
tablename = "logs"

[gcp]
username = "[email protected]"
hostname = "gcp-cluster.domain.net"

[aws]
username = "[email protected]"
hostname = "aws-cluster.domain.net"

The configuration has a default section, if nothing is set in your command hdxcli it'll use the default cluster, project and table. You can override the default by using --project or --table.
To use a specific profile table and project you can use:

hdxcli --profile gcp --project myproject --table mytable table settings

Usage

Using hdxcli command-line program

hdxcli supports multiple profiles. You can use a default profile or use the --profile option to operate on a non-default profile.

When trying to invoke a command, if a login to the server is necessary, a prompt will be shown and the token will be cached.

Command-line tool organization

The tool is organized, mostly with the general invocation form of:

hdxcli <resource> [<subresource...] <verb> [<resource_name>]

Table and project resources have defaults that depend on the profile you are working with, so they can be omitted if you used the set command.

For all other resources, you can use --transform, --dictionary, etc. Please see the command line help for more information.

Projects, tables and transforms

The basic operations you can do with these resources are:

  • list them
  • create a new resource
  • delete an existing resource
  • modify an existing resource
  • show a resource in raw json format
  • show settings from a resource
  • write a setting
  • show a single setting

Working with transforms

In order to use a transforms, you need to:

  1. create a transform
hdxcli transform create -f localfile.json transformname

Where localfile.json is a local file and transformname is the name for the transform that will be uploaded to the cluster.
Remember that a transform is applied to a table in a project, so whatever you set with the command-line tool will be the target of your transform.

If you want to override it, use:

hdxcli --project <the-project> --table <the-table> transform create -f localfile.json transformname
  1. get a specific transform settings
hdxcli --project hydro --table logs transform settings settings.format_details
settings.format_details: {'flattening': {'depth': 1, 'active': True, 'map_flattening_strategy': {'left': '.'}, 'slice_flattening_strategy': None}}

Listing and showing your profiles

Listing profiles:

hdxcli profile list

Showing default profile:

hdxcli profile show