Data & Jobs
Commands for data ingestion and process management. This covers real-time streaming, batch jobs, and storage configuration.
hdxcli v1.0.83
Stream
Commands for streaming data ingestion into a table. A project and table context is required.
Usage
hdxcli stream [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--project PROJECT_NAME | Use or override project set in the profile. |
--table TABLE_NAME | Use or override table set in the profile. |
--transform TRANSFORM_NAME | Explicitly pass the transform name. If none is given, the default transform for the used table is used. |
Ingest
Ingest data from a file into a table.
This command sends the contents of a local file to the ingest endpoint. It automatically applies the default transform for the target table, unless a specific transform is provided via the --transform option.
Usage
hdxcli stream ingest [OPTIONS] DATA_FILE_PATH
Examples
# Ingest data from a JSON file using the default transform
hdxcli stream --project my_project --table my_table ingest /data/today/access_log.json
# Ingest CSV data using a specific transform
hdxcli stream --project my_project --table my_table --transform custom_csv_transform ingest /data/today/metrics.csv
Job
Manage batch and alter jobs.
Usage
hdxcli job [OPTIONS] COMMAND [ARGS]...
Purgejobs
Purge all batch jobs in the organization.
Usage
hdxcli job purgejobs [OPTIONS]
Options
| Option | Description |
|---|---|
-y, --yes | Skip confirmation prompt. |
Alter
Manage alter jobs.
Usage
hdxcli job alter [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--job TEXT | Perform an operation on the passed job name. |
Cancel
Cancel an ongoing alter job.
Usage
hdxcli job alter cancel [OPTIONS] [JOB_NAME]
Examples
# Cancel the alter job named 'alter_job_123'
hdxcli job alter cancel alter_job_123
Commit
Commit changes made by an alter job.
Usage
hdxcli job alter commit [OPTIONS] [JOB_NAME]
Examples
# Commit the changes for the alter job named 'alter_job_123'
hdxcli job alter commit alter_job_123
Delete
Delete a specific alter.
This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.
Usage
hdxcli job alter delete [OPTIONS] ALTER_NAME
Options
| Option | Description |
|---|---|
--disable-confirmation-prompt | Suppress confirmation to delete alter. |
Examples
# Delete the alter named 'my_alter'
hdxcli job alter delete my_alter
List
List all alter jobs.
Usage
hdxcli job alter list [OPTIONS]
Options
| Option | Description |
|---|---|
--status TEXT | Filter alter jobs by status. |
--project TEXT | Filter alter jobs by project name. |
--table TEXT | Filter alter jobs by table name. |
Examples
# List all alter jobs with the status 'done'
hdxcli job alter list --status done
Retry
Retry a failed alter job.
Usage
hdxcli job alter retry [OPTIONS] [JOB_NAME]
Examples
# Retry the failed job named 'alter_job_123'
hdxcli job alter retry alter_job_123
Show
Show details for a specific alter.
Retrieves and displays the settings of a single alter. If no name is provided, the default alter will be used if exists.
Usage
hdxcli job alter show [OPTIONS] ALTER_NAME
Options
| Option | Description |
|---|---|
-i, --indent | Indent the output. |
Examples
# Show details for the alter named 'my_alter'
hdxcli job alter show my_alter
Verify
Verify the status of an alter job.
Usage
hdxcli job alter verify [OPTIONS] [JOB_NAME]
Examples
# Verify the status of the job named 'alter_job_123'
hdxcli job alter verify alter_job_123
Create
Create a new alter job.
Usage
hdxcli job alter create [OPTIONS] COMMAND [ARGS]...
Delete
Create a job to delete specific rows from a table.
Usage
hdxcli job alter create delete [OPTIONS]
Options
| Option | Description |
|---|---|
--table TEXT | The table to alter, e.g., my_proj.my_tbl. [required] |
--where TEXT | The WHERE clause for the delete operation. [required] |
Examples
# Delete all rows older than a specific date
hdxcli job alter create delete --table my_proj.my_tbl --where "timestamp < '2024-01-01'"
Update
Create a job to update specific rows in a table.
Usage
hdxcli job alter create update [OPTIONS]
Options
| Option | Description |
|---|---|
--table TEXT | The table to alter, e.g., my_proj.my_tbl. [required] |
--column TEXT | The column to update. [required] |
--value TEXT | The new value for the column. [required] |
--where TEXT | The WHERE clause for the update operation. [required] |
Examples
# Update the 'status' column to 'hidden' for specific rows
hdxcli job alter create update --table my_proj.my_tbl --column status --value hidden --where "timestamp < '2020-10-10'"
Batch
Manage batch jobs.
Usage
hdxcli job batch [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--job JOB_NAME | Perform operation on the passed job name. |
Cancel
Cancel a running batch job.
Usage
hdxcli job batch cancel [OPTIONS] JOB_NAME
Examples
# Cancel the job named 'batch-job-123'
hdxcli job batch cancel batch-job-123
Delete
Delete a specific batch.
This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.
Usage
hdxcli job batch delete [OPTIONS] BATCH_NAME
Options
| Option | Description |
|---|---|
--disable-confirmation-prompt | Suppress confirmation to delete batch. |
Examples
# Delete the batch named 'my_batch'
hdxcli job batch delete my_batch
Ingest
Create an ingest job from a settings file.
This command creates a batch ingest job based on a JSON configuration file. The file defines the data source (e.g., a cloud storage bucket), and other job-specific settings.
You can override the destination project, table, and transform specified within the settings file by using the --project, --table, and --transform options.
Usage
hdxcli job batch ingest [OPTIONS] JOB_NAME SETTINGS_FILE_PATH
Options
| Option | Description |
|---|---|
--project TEXT | Override the project for the ingest job. |
--table TEXT | Override the table for the ingest job. |
--transform TEXT | Override the transform to use. Defaults to the table's default transform. |
Examples
# Create an ingest job from a file, overriding the destination table
hdxcli job batch ingest my-batch-job ./aws-s3-settings.json --project my_proj --table my_tabl
List
List all batch jobs.
Usage
hdxcli job batch list [OPTIONS]
Examples
# List all batch jobs in the organization
hdxcli job batch list
Retry
Retry a failed batch job.
Usage
hdxcli job batch retry [OPTIONS] JOB_NAME
Examples
# Retry the failed job named 'batch-job-123'
hdxcli job batch retry batch-job-123
Settings
List, get, or set key-value settings for a specific batch.
This command operates in three modes:
- LIST: Invoked with no arguments, it lists all settings.
- GET: Invoked with only a KEY, it retrieves the value of that setting.
- SET: Invoked with a KEY and a VALUE, it sets the value for that setting.
The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.
Usage
hdxcli job batch settings [OPTIONS] [KEY] [VALUE]
Options
| Option | Description |
|---|---|
-F, --force | This flag allows adding the force_operation parameter to the request. |
Examples
# List all settings for the batch 'my_batch'
hdxcli job batch --batch my_batch settings
# Get the 'name' setting for the batch 'my_batch'
hdxcli job batch --batch my_batch settings name
# Set a new 'name' setting for the batch 'my_batch'
hdxcli job batch --batch my_batch settings name new_name
Show
Show details for a specific batch.
Retrieves and displays the settings of a single batch. If no name is provided, the default batch will be used if exists.
Usage
hdxcli job batch show [OPTIONS] BATCH_NAME
Options
| Option | Description |
|---|---|
-i, --indent | Indent the output. |
Examples
# Show details for the batch named 'my_batch'
hdxcli job batch show my_batch
Source
Manage sources (Kafka, Kinesis, SIEM).
Usage
hdxcli source [OPTIONS] COMMAND [ARGS]...
Kafka
Manage Kafka sources.
Usage
hdxcli source kafka [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--project PROJECT_NAME | Use or override project set in the profile. |
--table TABLE_NAME | Use or override table set in the profile. |
--source KAFKA_SOURCE_NAME | The name of the Kafka source. |
Create
Creates a new kafka source from a JSON configuration file.
Usage
hdxcli source kafka create [OPTIONS] KAFKA_NAME SETTINGS_FILENAME
Examples
# Create a kafka source
hdxcli source kafka --project my_project --table my_table create my_kafka path/to/kafka-settings.json
Delete
Delete a specific kafka.
This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.
Usage
hdxcli source kafka delete [OPTIONS] KAFKA_NAME
Options
| Option | Description |
|---|---|
--disable-confirmation-prompt | Suppress confirmation to delete kafka. |
Examples
# Delete the kafka named 'my_kafka'
hdxcli source kafka --project my_project --table my_table delete my_kafka
List
List all available kafkas.
Retrieves a list of all kafkas you have access to. Pagination options (--page, --page-size) are available if supported by the API.
Usage
hdxcli source kafka list [OPTIONS]
Options
| Option | Description |
|---|---|
-p, --page INTEGER | Page number. |
-s, --page-size INTEGER | Number of items per page. |
Examples
# List the first page of kafkas
hdxcli source kafka --project my_project --table my_table list
Settings
List, get, or set key-value settings for a specific kafka.
This command operates in three modes:
- LIST: Invoked with no arguments, it lists all settings.
- GET: Invoked with only a KEY, it retrieves the value of that setting.
- SET: Invoked with a KEY and a VALUE, it sets the value for that setting.
The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.
Usage
hdxcli source kafka settings [OPTIONS] [KEY] [VALUE]
Options
| Option | Description |
|---|---|
-F, --force | This flag allows adding the force_operation parameter to the request. |
Examples
# List all settings for the kafka 'my_kafka'
hdxcli source kafka --project my_project --table my_table --kafka my_kafka settings
# Get the 'name' setting for the kafka 'my_kafka'
hdxcli source kafka --project my_project --table my_table --kafka my_kafka settings name
# Set a new 'name' setting for the kafka 'my_kafka'
hdxcli source kafka --project my_project --table my_table --kafka my_kafka settings name new_name
Show
Show details for a specific kafka.
Retrieves and displays the settings of a single kafka. If no name is provided, the default kafka will be used if exists.
Usage
hdxcli source kafka show [OPTIONS] KAFKA_NAME
Options
| Option | Description |
|---|---|
-i, --indent | Indent the output. |
Examples
# Show details for the kafka named 'my_kafka'
hdxcli source kafka --project my_project --table my_table show my_kafka
Kinesis
Manage Kinesis sources.
Usage
hdxcli source kinesis [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--project PROJECT_NAME | Use or override project set in the profile. |
--table TABLE_NAME | Use or override table set in the profile. |
--source KINESIS_SOURCE_NAME | The name of the Kinesis source. |
Create
Creates a new kinesis source from a JSON configuration file.
Usage
hdxcli source kinesis create [OPTIONS] KINESIS_NAME SETTINGS_FILENAME
Examples
# Create a kinesis source
hdxcli source kinesis --project my_project --table my_table create my_kinesis path/to/kinesis-settings.json
Delete
Delete a specific kinesis.
This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.
Usage
hdxcli source kinesis delete [OPTIONS] KINESIS_NAME
Options
| Option | Description |
|---|---|
--disable-confirmation-prompt | Suppress confirmation to delete kinesis. |
Examples
# Delete the kinesis named 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table delete my_kinesis
List
List all available kineses.
Retrieves a list of all kineses you have access to. Pagination options (--page, --page-size) are available if supported by the API.
Usage
hdxcli source kinesis list [OPTIONS]
Options
| Option | Description |
|---|---|
-p, --page INTEGER | Page number. |
-s, --page-size INTEGER | Number of items per page. |
Examples
# List the first page of kineses
hdxcli source kinesis --project my_project --table my_table list
Settings
List, get, or set key-value settings for a specific kinesis.
This command operates in three modes:
- LIST: Invoked with no arguments, it lists all settings.
- GET: Invoked with only a KEY, it retrieves the value of that setting.
- SET: Invoked with a KEY and a VALUE, it sets the value for that setting.
The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.
Usage
hdxcli source kinesis settings [OPTIONS] [KEY] [VALUE]
Options
| Option | Description |
|---|---|
-F, --force | This flag allows adding the force_operation parameter to the request. |
Examples
# List all settings for the kinesis 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table --kinesis my_kinesis settings
# Get the 'name' setting for the kinesis 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table --kinesis my_kinesis settings name
# Set a new 'name' setting for the kinesis 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table --kinesis my_kinesis settings name new_name
Show
Show details for a specific kinesis.
Retrieves and displays the settings of a single kinesis. If no name is provided, the default kinesis will be used if exists.
Usage
hdxcli source kinesis show [OPTIONS] KINESIS_NAME
Options
| Option | Description |
|---|---|
-i, --indent | Indent the output. |
Examples
# Show details for the kinesis named 'my_kinesis'
hdxcli source kinesis --project my_project --table my_table show my_kinesis
Siem
Manage SIEM sources.
Usage
hdxcli source siem [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--project PROJECT_NAME | Use or override project set in the profile. |
--table TABLE_NAME | Use or override table set in the profile. |
--source SIEM_SOURCE_NAME | The name of the SIEM source. |
Create
Creates a new siem source from a JSON configuration file.
Usage
hdxcli source siem create [OPTIONS] SIEM_NAME SETTINGS_FILENAME
Examples
# Create a siem source
hdxcli source siem --project my_project --table my_table create my_siem path/to/siem-settings.json
Delete
Delete a specific siem.
This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.
Usage
hdxcli source siem delete [OPTIONS] SIEM_NAME
Options
| Option | Description |
|---|---|
--disable-confirmation-prompt | Suppress confirmation to delete siem. |
Examples
# Delete the siem named 'my_siem'
hdxcli source siem --project my_project --table my_table delete my_siem
List
List all available siems.
Retrieves a list of all siems you have access to. Pagination options (--page, --page-size) are available if supported by the API.
Usage
hdxcli source siem list [OPTIONS]
Options
| Option | Description |
|---|---|
-p, --page INTEGER | Page number. |
-s, --page-size INTEGER | Number of items per page. |
Examples
# List the first page of siems
hdxcli source siem --project my_project --table my_table list
Settings
List, get, or set key-value settings for a specific siem.
This command operates in three modes:
- LIST: Invoked with no arguments, it lists all settings.
- GET: Invoked with only a KEY, it retrieves the value of that setting.
- SET: Invoked with a KEY and a VALUE, it sets the value for that setting.
The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.
Usage
hdxcli source siem settings [OPTIONS] [KEY] [VALUE]
Options
| Option | Description |
|---|---|
-F, --force | This flag allows adding the force_operation parameter to the request. |
Examples
# List all settings for the siem 'my_siem'
hdxcli source siem --project my_project --table my_table --siem my_siem settings
# Get the 'name' setting for the siem 'my_siem'
hdxcli source siem --project my_project --table my_table --siem my_siem settings name
# Set a new 'name' setting for the siem 'my_siem'
hdxcli source siem --project my_project --table my_table --siem my_siem settings name new_name
Show
Show details for a specific siem.
Retrieves and displays the settings of a single siem. If no name is provided, the default siem will be used if exists.
Usage
hdxcli source siem show [OPTIONS] SIEM_NAME
Options
| Option | Description |
|---|---|
-i, --indent | Indent the output. |
Examples
# Show details for the siem named 'my_siem'
hdxcli source siem --project my_project --table my_table show my_siem
Storage
This group of commands allows to create, list, show, and delete storages.
Usage
hdxcli storage [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--storage STORAGE_NAME | Perform operation on the passed storage. |
Create
Create a new storage.
A storage can be created in two ways:
- Using a JSON settings file via
--settings-filename. - Providing individual settings as options (
--bucket-path,--bucket-name, etc).
Usage
hdxcli storage create [OPTIONS] STORAGE_NAME
Options
| Option | Description |
|---|---|
-f, --settings-filename PATH | Path to a JSON file with storage configuration settings. |
-p, --bucket-path TEXT | Path to the storage bucket. |
-n, --bucket-name TEXT | Name of the storage bucket. |
-r, --region TEXT | Region for the storage bucket. |
-c, --cloud TEXT | Type of cloud storage (e.g., aws, gcp). |
-E, --endpoint TEXT | Endpoint for the storage bucket. |
-C, --credential-name TEXT | Name of the credential to use for the storage bucket. |
-M, --io-perf-mode [aggressive|moderate|moderate] | I/O performance mode for the storage bucket. |
Examples
# Create a storage from a settings file
hdxcli storage create my_storage -f path/to/storage.json
# Create a storage using individual options
hdxcli storage create gcp-storage --bucket-path my-path --bucket-name my-bucket --region us-central1 --cloud gcp
Delete
Delete a specific storage.
This is a permanent action and cannot be undone. You will be prompted for confirmation unless --disable-confirmation-prompt is used.
Usage
hdxcli storage delete [OPTIONS] STORAGE_NAME
Options
| Option | Description |
|---|---|
--disable-confirmation-prompt | Suppress confirmation to delete storage. |
Examples
# Delete the storage named 'my_storage'
hdxcli storage delete my_storage
List
List all available storages.
Retrieves a list of all storages you have access to. Pagination options (--page, --page-size) are available if supported by the API.
Usage
hdxcli storage list [OPTIONS]
Options
| Option | Description |
|---|---|
-p, --page INTEGER | Page number. |
-s, --page-size INTEGER | Number of items per page. |
Examples
# List the first page of storages
hdxcli storage list
Settings
List, get, or set key-value settings for a specific storage.
This command operates in three modes:
- LIST: Invoked with no arguments, it lists all settings.
- GET: Invoked with only a KEY, it retrieves the value of that setting.
- SET: Invoked with a KEY and a VALUE, it sets the value for that setting.
The VALUE can be a string, a number, or a JSON-formatted string for lists/objects. When setting a value, the --force-operation option may be required for certain resource.
Usage
hdxcli storage settings [OPTIONS] [KEY] [VALUE]
Options
| Option | Description |
|---|---|
-F, --force | This flag allows adding the force_operation parameter to the request. |
Examples
# List all settings for the storage 'my_storage'
hdxcli storage --storage my_storage settings
# Get the 'name' setting for the storage 'my_storage'
hdxcli storage --storage my_storage settings name
# Set a new 'name' setting for the storage 'my_storage'
hdxcli storage --storage my_storage settings name new_name
Show
Show details for a specific storage.
Retrieves and displays the settings of a single storage. If no name is provided, the default storage will be used if exists.
Usage
hdxcli storage show [OPTIONS] STORAGE_NAME
Options
| Option | Description |
|---|---|
-i, --indent | Indent the output. |
Examples
# Show details for the storage named 'my_storage'
hdxcli storage show my_storage
Updated about 10 hours ago