Data and Jobs

hdxcli v1.0.83

This command sends the contents of a local file to the ingest endpoint. It automatically applies the default transform for the target table, unless a specific transform is provided through the --transform option.

Usage

hdxcli stream ingest [OPTIONS] DATA_FILE_PATH

Examples

## Ingest data from a JSON file using the default transform
hdxcli stream --project my_project --table my_table ingest /data/today/access_log.json

## Ingest CSV data using a specific transform
hdxcli stream --project my_project --table my_table --transform custom_csv_transform ingest /data/today/metrics.csv

Job⚓︎

Manage batch and alter jobs.

Usage

hdxcli job [OPTIONS] COMMAND [ARGS]...

Purgejobs⚓︎

Purge all batch jobs in the organization.

Usage

hdxcli job purgejobs [OPTIONS]

Options

Option	Description
`-y, --yes`	Skip confirmation prompt.

Alter⚓︎

Manage alter jobs.

Usage

hdxcli job alter [OPTIONS] COMMAND [ARGS]...

Options

Option	Description
`--job TEXT`	Perform an operation on the passed job name.

Cancel⚓︎

Cancel an ongoing alter job.

Usage

hdxcli job alter cancel [OPTIONS] [JOB_NAME]

Examples

## Cancel the alter job named 'alter_job_123'
hdxcli job alter cancel alter_job_123

Commit⚓︎

Commit changes made by an alter job.

Usage

hdxcli job alter commit [OPTIONS] [JOB_NAME]

Examples

## Commit the changes for the alter job named 'alter_job_123'
hdxcli job alter commit alter_job_123

Delete⚓︎

Delete a specific alter.

This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.

Usage

hdxcli job alter delete [OPTIONS] ALTER_NAME

Options

Option	Description
`--disable-confirmation-prompt`	Suppress confirmation to delete alter.

Examples

## Delete the alter named 'my_alter'
hdxcli job alter delete my_alter

List⚓︎

List all alter jobs.

Usage

hdxcli job alter list [OPTIONS]

Options

Option	Description
`--status TEXT`	Filter alter jobs by status.
`--project TEXT`	Filter alter jobs by project name.
`--table TEXT`	Filter alter jobs by table name.

Examples

## List all alter jobs with the status 'done'
hdxcli job alter list --status done

Retry⚓︎

Retry a failed alter job.

Usage

hdxcli job alter retry [OPTIONS] [JOB_NAME]

Examples

## Retry the failed job named 'alter_job_123'
hdxcli job alter retry alter_job_123

Show⚓︎

Show details for a specific alter.

Retrieves and displays the settings of a single alter. If no name is provided, the default alter will be used if exists.

Usage

hdxcli job alter show [OPTIONS] ALTER_NAME

Options

Option	Description
`-i, --indent`	Indent the output.

Examples

## Show details for the alter named 'my_alter'
hdxcli job alter show my_alter

Verify⚓︎

Verify the status of an alter job.

Usage

hdxcli job alter verify [OPTIONS] [JOB_NAME]

Examples

## Verify the status of the job named 'alter_job_123'
hdxcli job alter verify alter_job_123

Create⚓︎

Create a new alter job.

Usage

hdxcli job alter create [OPTIONS] COMMAND [ARGS]...

Delete⚓︎

Create a job to delete specific rows from a table.

Usage

hdxcli job alter create delete [OPTIONS]

Options

Option	Description
`--table TEXT`	The table to alter, for example, my_proj.my_tbl. [required]
`--where TEXT`	The WHERE clause for the delete operation. [required]

Examples

## Delete all rows older than a specific date
hdxcli job alter create delete --table my_proj.my_tbl --where "timestamp < '2024-01-01'"

Update⚓︎

Create a job to update specific rows in a table.

Usage

hdxcli job alter create update [OPTIONS]

Options

Option	Description
`--table TEXT`	The table to alter, for example, my_proj.my_tbl. [required]
`--column TEXT`	The column to update. [required]
`--value TEXT`	The new value for the column. [required]
`--where TEXT`	The WHERE clause for the update operation. [required]

Examples

## Update the 'status' column to 'hidden' for specific rows
hdxcli job alter create update --table my_proj.my_tbl --column status --value hidden --where "timestamp < '2020-10-10'"

Batch⚓︎

Manage batch jobs.

Usage

hdxcli job batch [OPTIONS] COMMAND [ARGS]...

Options

Option	Description
`--job JOB_NAME`	Perform operation on the passed job name.

Cancel⚓︎

Cancel a running batch job.

Usage

hdxcli job batch cancel [OPTIONS] JOB_NAME

Examples

## Cancel the job named 'batch-job-123'
hdxcli job batch cancel batch-job-123

Delete⚓︎

Delete a specific batch.

This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.

Usage

hdxcli job batch delete [OPTIONS] BATCH_NAME

Options

Option	Description
`--disable-confirmation-prompt`	Suppress confirmation to delete batch.

Examples

## Delete the batch named 'my_batch'
hdxcli job batch delete my_batch

Ingest⚓︎

Create an ingest job from a settings file.

This command creates a batch ingest job based on a JSON configuration file. The file defines the data source (for example, a cloud storage bucket), and other job-specific settings.

You can override the destination project, table, and transform specified within the settings file by using the --project, --table, and --transform options.

Usage

hdxcli job batch ingest [OPTIONS] JOB_NAME SETTINGS_FILE_PATH

Options

Option	Description
`--project TEXT`	Override the project for the ingest job.
`--table TEXT`	Override the table for the ingest job.
`--transform TEXT`	Override the transform to use. Defaults to the table's default transform.

Examples

## Create an ingest job from a file, overriding the destination table
hdxcli job batch ingest my-batch-job ./aws-s3-settings.json --project my_proj --table my_tabl

List⚓︎

List all batch jobs.

Usage

hdxcli job batch list [OPTIONS]

Examples

## List all batch jobs in the organization
hdxcli job batch list

Retry⚓︎

Retry a failed batch job.

Usage

hdxcli job batch retry [OPTIONS] JOB_NAME

Examples

## Retry the failed job named 'batch-job-123'
hdxcli job batch retry batch-job-123

Settings⚓︎

List, get, or set key-value settings for a specific batch.

This command operates in three modes: - LIST: Invoked with no arguments, it lists all settings. - GET: Invoked with only a KEY, it retrieves the value of that setting. - SET: Invoked with a KEY and a VALUE, it sets the value for that setting.

The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.

Usage

hdxcli job batch settings [OPTIONS] [KEY] [VALUE]

Options

Option	Description
`-F, --force`	This flag allows adding the `force_operation` parameter to the request.

Examples

## List all settings for the batch 'my_batch'
hdxcli job batch --batch my_batch settings

## Get the 'name' setting for the batch 'my_batch'
hdxcli job batch --batch my_batch settings name

## Set a new 'name' setting for the batch 'my_batch'
hdxcli job batch --batch my_batch settings name new_name

Show⚓︎

Show details for a specific batch.

Retrieves and displays the settings of a single batch. If no name is provided, the default batch will be used if exists.

Usage

hdxcli job batch show [OPTIONS] BATCH_NAME

Options

Option	Description
`-i, --indent`	Indent the output.

Examples

## Show details for the batch named 'my_batch'
hdxcli job batch show my_batch

Source⚓︎

Manage sources (Kafka, Kinesis, SIEM).

Usage

hdxcli source [OPTIONS] COMMAND [ARGS]...

Kafka⚓︎

Manage Kafka sources.

Usage

hdxcli source kafka [OPTIONS] COMMAND [ARGS]...

Options

Option	Description
`--project PROJECT_NAME`	Use or override project set in the profile.
`--table TABLE_NAME`	Use or override table set in the profile.
`--source KAFKA_SOURCE_NAME`	The name of the Kafka source.

Create⚓︎

Creates a new kafka source from a JSON configuration file.

Usage

hdxcli source kafka create [OPTIONS] KAFKA_NAME SETTINGS_FILENAME

Examples

## Create a kafka source
hdxcli source kafka --project my_project --table my_table create my_kafka path/to/kafka-settings.json

Delete⚓︎

Delete a specific kafka.

This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.

Usage

hdxcli source kafka delete [OPTIONS] KAFKA_NAME

Options

Option	Description
`--disable-confirmation-prompt`	Suppress confirmation to delete kafka.

Examples

## Delete the kafka named 'my_kafka'
hdxcli source kafka --project my_project --table my_table delete my_kafka

List⚓︎

List all available kafkas.

Retrieves a list of all kafkas you have access to. Pagination options (--page, --page-size) are available if supported by the API.

Usage

hdxcli source kafka list [OPTIONS]

Options

Option	Description
`-p, --page INTEGER`	Page number.
`-s, --page-size INTEGER`	Number of items per page.

Examples

## List the first page of kafkas
hdxcli source kafka --project my_project --table my_table list

Settings⚓︎

List, get, or set key-value settings for a specific kafka.

This command operates in three modes: - LIST: Invoked with no arguments, it lists all settings. - GET: Invoked with only a KEY, it retrieves the value of that setting. - SET: Invoked with a KEY and a VALUE, it sets the value for that setting.

The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.

Usage

hdxcli source kafka settings [OPTIONS] [KEY] [VALUE]

Options

Option	Description
`-F, --force`	This flag allows adding the `force_operation` parameter to the request.

Examples

## List all settings for the kafka 'my_kafka'
hdxcli source kafka --project my_project --table my_table --kafka my_kafka settings

## Get the 'name' setting for the kafka 'my_kafka'
hdxcli source kafka --project my_project --table my_table --kafka my_kafka settings name

## Set a new 'name' setting for the kafka 'my_kafka'
hdxcli source kafka --project my_project --table my_table --kafka my_kafka settings name new_name

Show⚓︎

Show details for a specific kafka.

Retrieves and displays the settings of a single kafka. If no name is provided, the default kafka will be used if exists.

Usage

hdxcli source kafka show [OPTIONS] KAFKA_NAME

Options

Option	Description
`-i, --indent`	Indent the output.

Examples

## Show details for the kafka named 'my_kafka'
hdxcli source kafka --project my_project --table my_table show my_kafka

Kinesis⚓︎

Manage Kinesis sources.

Usage

hdxcli source kinesis [OPTIONS] COMMAND [ARGS]...

Options

Option	Description
`--project PROJECT_NAME`	Use or override project set in the profile.
`--table TABLE_NAME`	Use or override table set in the profile.
`--source KINESIS_SOURCE_NAME`	The name of the Kinesis source.

Create⚓︎

Creates a new kinesis source from a JSON configuration file.

Usage

hdxcli source kinesis create [OPTIONS] KINESIS_NAME SETTINGS_FILENAME

Examples

## Create a kinesis source
hdxcli source kinesis --project my_project --table my_table create my_kinesis path/to/kinesis-settings.json

Delete⚓︎

Delete a specific kinesis.

This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.

Usage

hdxcli source kinesis delete [OPTIONS] KINESIS_NAME

Options

Option	Description
`--disable-confirmation-prompt`	Suppress confirmation to delete kinesis.

Examples

## Delete the kinesis named 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table delete my_kinesis

List⚓︎

List all available kineses.

Retrieves a list of all kineses you have access to. Pagination options (--page, --page-size) are available if supported by the API.

Usage

hdxcli source kinesis list [OPTIONS]

Options

Option	Description
`-p, --page INTEGER`	Page number.
`-s, --page-size INTEGER`	Number of items per page.

Examples

## List the first page of kineses
hdxcli source kinesis --project my_project --table my_table list

Settings⚓︎

List, get, or set key-value settings for a specific kinesis.

This command operates in three modes: - LIST: Invoked with no arguments, it lists all settings. - GET: Invoked with only a KEY, it retrieves the value of that setting. - SET: Invoked with a KEY and a VALUE, it sets the value for that setting.

The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.

Usage

hdxcli source kinesis settings [OPTIONS] [KEY] [VALUE]

Options

Option	Description
`-F, --force`	This flag allows adding the `force_operation` parameter to the request.

Examples

## List all settings for the kinesis 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table --kinesis my_kinesis settings

## Get the 'name' setting for the kinesis 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table --kinesis my_kinesis settings name

## Set a new 'name' setting for the kinesis 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table --kinesis my_kinesis settings name new_name

Show⚓︎

Show details for a specific kinesis.

Retrieves and displays the settings of a single kinesis. If no name is provided, the default kinesis will be used if exists.

Usage

hdxcli source kinesis show [OPTIONS] KINESIS_NAME

Options

Option	Description
`-i, --indent`	Indent the output.

Examples

## Show details for the kinesis named 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table show my_kinesis

Siem⚓︎

Manage SIEM sources.

Usage

hdxcli source siem [OPTIONS] COMMAND [ARGS]...

Options

Option	Description
`--project PROJECT_NAME`	Use or override project set in the profile.
`--table TABLE_NAME`	Use or override table set in the profile.
`--source SIEM_SOURCE_NAME`	The name of the SIEM source.

Create⚓︎

Creates a new siem source from a JSON configuration file.

Usage

hdxcli source siem create [OPTIONS] SIEM_NAME SETTINGS_FILENAME

Examples

## Create a siem source
hdxcli source siem --project my_project --table my_table create my_siem path/to/siem-settings.json

Delete⚓︎

Delete a specific siem.

This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.

Usage

hdxcli source siem delete [OPTIONS] SIEM_NAME

Options

Option	Description
`--disable-confirmation-prompt`	Suppress confirmation to delete siem.

Examples

## Delete the siem named 'my_siem'
hdxcli source siem --project my_project --table my_table delete my_siem

List⚓︎

List all available siems.

Retrieves a list of all siems you have access to. Pagination options (--page, --page-size) are available if supported by the API.

Usage

hdxcli source siem list [OPTIONS]

Options

Option	Description
`-p, --page INTEGER`	Page number.
`-s, --page-size INTEGER`	Number of items per page.

Examples

## List the first page of siems
hdxcli source siem --project my_project --table my_table list

Settings⚓︎

List, get, or set key-value settings for a specific siem.

This command operates in three modes: - LIST: Invoked with no arguments, it lists all settings. - GET: Invoked with only a KEY, it retrieves the value of that setting. - SET: Invoked with a KEY and a VALUE, it sets the value for that setting.

The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.

Usage

hdxcli source siem settings [OPTIONS] [KEY] [VALUE]

Options

Option	Description
`-F, --force`	This flag allows adding the `force_operation` parameter to the request.

Examples

## List all settings for the siem 'my_siem'
hdxcli source siem --project my_project --table my_table --siem my_siem settings

## Get the 'name' setting for the siem 'my_siem'
hdxcli source siem --project my_project --table my_table --siem my_siem settings name

## Set a new 'name' setting for the siem 'my_siem'
hdxcli source siem --project my_project --table my_table --siem my_siem settings name new_name

Show⚓︎

Show details for a specific siem.

Retrieves and displays the settings of a single siem. If no name is provided, the default siem will be used if exists.

Usage

hdxcli source siem show [OPTIONS] SIEM_NAME

Options

Option	Description
`-i, --indent`	Indent the output.

Examples

## Show details for the siem named 'my_siem'
hdxcli source siem --project my_project --table my_table show my_siem

Storage⚓︎

This group of commands allows to create, list, show, and delete storages.

Usage

hdxcli storage [OPTIONS] COMMAND [ARGS]...

Options

Option	Description
`--storage STORAGE_NAME`	Perform operation on the passed storage.

Create⚓︎

Create a new storage.

A storage can be created in two ways: - Using a JSON settings file via --settings-filename. - Providing individual settings as options (--bucket-path, --bucket-name, etc).

Usage

hdxcli storage create [OPTIONS] STORAGE_NAME

Options

Option	Description
`-f, --settings-filename PATH`	Path to a JSON file with storage configuration settings.
`-p, --bucket-path TEXT`	Path to the storage bucket.
`-n, --bucket-name TEXT`	Name of the storage bucket.
`-r, --region TEXT`	Region for the storage bucket.
`-c, --cloud TEXT`	Type of cloud storage (for example, aws, gcp).
`-E, --endpoint TEXT`	Endpoint for the storage bucket.
`-C, --credential-name TEXT`	Name of the credential to use for the storage bucket.
`-M, --io-perf-mode [aggressive\\|moderate\\|moderate]`	I/O performance mode for the storage bucket.

Examples

## Create a storage from a settings file
hdxcli storage create my_storage -f path/to/storage.json

## Create a storage using individual options
hdxcli storage create gcp-storage --bucket-path my-path --bucket-name my-bucket --region us-central1 --cloud gcp

Delete⚓︎

Delete a specific storage.

This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.

Usage

hdxcli storage delete [OPTIONS] STORAGE_NAME

Options

Option	Description
`--disable-confirmation-prompt`	Suppress confirmation to delete storage.

Examples

## Delete the storage named 'my_storage'
hdxcli storage delete my_storage

List⚓︎

List all available storages.

Retrieves a list of all storages you have access to. Pagination options (--page, --page-size) are available if supported by the API.

Usage

hdxcli storage list [OPTIONS]

Options

Option	Description
`-p, --page INTEGER`	Page number.
`-s, --page-size INTEGER`	Number of items per page.

Examples

## List the first page of storages
hdxcli storage list

Settings⚓︎

List, get, or set key-value settings for a specific storage.

This command operates in three modes: - LIST: Invoked with no arguments, it lists all settings. - GET: Invoked with only a KEY, it retrieves the value of that setting. - SET: Invoked with a KEY and a VALUE, it sets the value for that setting.

The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.

Usage

hdxcli storage settings [OPTIONS] [KEY] [VALUE]

Options

Option	Description
`-F, --force`	This flag allows adding the `force_operation` parameter to the request.

Examples

## List all settings for the storage 'my_storage'
hdxcli storage --storage my_storage settings

## Get the 'name' setting for the storage 'my_storage'
hdxcli storage --storage my_storage settings name

## Set a new 'name' setting for the storage 'my_storage'
hdxcli storage --storage my_storage settings name new_name

Show⚓︎

Show details for a specific storage.

Retrieves and displays the settings of a single storage. If no name is provided, the default storage will be used if exists.

Usage

hdxcli storage show [OPTIONS] STORAGE_NAME

Options

Option	Description
`-i, --indent`	Indent the output.

Examples

## Show details for the storage named 'my_storage'
hdxcli storage show my_storage

Data and Jobs

Stream⚓︎

Ingest⚓︎

Job⚓︎

Purgejobs⚓︎

Alter⚓︎

Cancel⚓︎

Commit⚓︎

Delete⚓︎

List⚓︎

Retry⚓︎

Show⚓︎

Verify⚓︎

Create⚓︎

Delete⚓︎

Update⚓︎

Batch⚓︎

Cancel⚓︎

Delete⚓︎

Ingest⚓︎

List⚓︎

Retry⚓︎

Settings⚓︎

Show⚓︎

Source⚓︎

Kafka⚓︎

Create⚓︎

Delete⚓︎

List⚓︎

Settings⚓︎

Show⚓︎

Kinesis⚓︎

Create⚓︎

Delete⚓︎

List⚓︎

Settings⚓︎

Show⚓︎

Siem⚓︎

Create⚓︎

Delete⚓︎

List⚓︎

Settings⚓︎

Show⚓︎

Storage⚓︎

Create⚓︎

Delete⚓︎

List⚓︎

Settings⚓︎

Show⚓︎