HKT Commands

hkt

The hkt CLI (Hydrolix Kubernetes Tools) is purely used for Kubernetes deployments and provides the templates needed to deploy and manage Hydrolix within Kubernetes.

Usage

$ hkt [OPTIONS] COMMAND [ARGS]

Options

OptionPurpose
--helpDisplays summary documentation for using hkt and exits.

Commands

Summary

The following commands are available with hkt.

CommandPurpose
get-resourcesRetrieves the YAML for the Kubernetes Cluster and components.
hydrolix-clusterCreates the Hydrolix Cluster resource definition YAML
operator-resourcesCreates the Operator resource definition YAML
scale-defaultsProvides a list of scale-defaults for replicas, cpu, memory, storage and data_strorage for the 3 predefined scale profiles default, dev and prod
versionRetrieves the version of the hkt tool.
config-reportReport all the different options specified in the cluster

get-resources

Retrieves the YAML for the Kubernetes Cluster, jobs and components.

Usage

hkt get-resources [OPTIONS] {secrets|data|init-cluster|keycloak|init-turbine-api|hydrolix|load-sample-project|ingress}

Options

OptionPurpose
--version TEXTVersion override. Default uses the value found in the --cluster-file if not set.
-c, --cluster-file TEXT [required]Required: The Hydrolix Cluster resource definition file.
--helpDisplays summary documentation for using hkt and exits.

Flags [required]

FlagDescription
secretsDescribes the secrets of the Kubernetes cluster. Outputs YAML.
dataDescribes the data components for the cluster. Outputs YAML.
init-clusterDescribes the initialisation Job that is executed on creation of a cluster. Outputs YAML.
keycloakDescribes the service and deployment of the keycloak component. Outputs YAML.
init-turbine-apiDescribes the turbine api initialisation Job that is executed on creation of a cluster. Outputs YAML.
hydrolixDescribes the bulk of the Hydrolix cluster components. Outputs YAML.
load-sample-projectDescribes a load Job for example data into the cluster that is executed on creation of a cluster. Outputs YAML.
ingressDescribes ingress (Traefik) components of the cluster. Outputs YAML.

Example

$ hkt get-resources -c hydrolixcluster.yaml init-cluster
apiVersion: batch/v1
kind: Job
metadata:
  name: init-cluster-v1-2-3
  namespace: hdxcli-abc456
spec:
  backoffLimit: 0
  template:
    spec:
      containers:
      - env: []
        envFrom:
        - configMapRef:
            name: general
        - configMapRef:
            name: curated
            optional: true
        - secretRef:
            name: general
        - secretRef:
            name: curated
            optional: true
        image: 123456.abc.def.us-east-2.amazonaws.com/turbine/init-cluster:1234-abcde
        name: init-cluster-v1-2-3
        resources:
          limits: &id001
            cpu: 500m
            ephemeral-storage: 5Gi
            memory: 1Gi
          requests: *id001
      imagePullSecrets:
      - name: hdx-registry-creds
      restartPolicy: Never

hydrolix-cluster

Provides the hydrolix cluster resources for a Hydrolix Kubernetes deployment.

Usage

$ hkt hydrolix-cluster [OPTIONS]

Options

OptionDescription
--client-id TEXT [required]Required. The Hydrolix licence/client_id.
--admin-email TEXT [required]Required. Email address to be used as an admin email.
--host TEXT [required]Required. Hostname to be used to access the Hydrolix Cluster
--domain TEXT [required]Required. Domain name to be used to access the Hydrolix Cluster
--owner TEXT [required]Required. A named owner for the system.
--cloud TEXT [required]Required. The cloud the Kubernetes cluster will be deployed upon.
--region TEXT [required]Required. The Region to be used
--env TEXTUse an Environment variable within the command
--scale-profile TEXTOptional. The default scale-profile to use in bringing up the cluster. See scale-defaults below for more information.
--ip-allowlist TEXTRequired. An IP allowlist of IP's / CIDR block to access the cluster. More detailed IP access control can be defined if needed.
--helpDisplays summary documentation for this command, and exits.

Example

hkt hydrolix-cluster --ip-allowlist 0.0.0.0/0 [email protected] --client-id=hdxcli-1234abc --host=myhost --owner=imadmin --region=us-central1 --cloud=gcp --domain=mydomain.net --scale-profile=prod
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: hdxcli-1234abc
  namespace: hdxcli-1234abc
spec:
  admin_email: [email protected]
  client_id: hdxcli-1234abc
  cloud: gcp
  domain: mydomain.net
  env:
  host: myhost
  ip_allowlist:
  - source: 0.0.0.0/0
  owner: imadmin
  region: us-central1
  scale: {}
  scale_profile: prod

operator-resources

Provides the operator YAML required for a Hydrolix Kubernetes deployment.

Usage

hkt operator-resources [OPTIONS]

Options

OptionDescription
--client-id TEXT [required]The Hydrolix licence/client_id. Required field.
--versionVersion of Hydrolix that should be used with this operator. If not set the default is the current version of the HKT Tool.
--helpDisplays summary documentation for this command, and exits.

Example

$ hkt operator-resources --client-id hdxcli-1234abc
apiVersion: v1
kind: ServiceAccount
metadata:
  name: operator
  namespace: hdxcli-1234abc
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: operator-abcd1234
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
   .....

scale-defaults

Provides a list of scale-defaults for replicas, cpu, memory, storage and data_strorage for the 3 predefined scale profiles default, dev and prod

Usage

$ hkt scale-defaults [OPTIONS]

Options

OptionDescription
-p, --profile TEXTThe scale profile to be viewed ( default, dev and prod). If none are specified the default is shown.
--helpDisplays summary documentation for this command, and exits.

Example

$ hkt scale-defaults -p dev
service                replicas    cpu  memory    storage    data_storage
-------------------  ----------  -----  --------  ---------  --------------
alter-turbine                 0    0.2  1Gi       1Gi        -
alter-head                    0    0.8  1Gi       1Gi        -
alter-peer                    0    0.8  1Gi       1Gi        -
batch-head                    1    0.5  1Gi       1Gi        -
batch-peer                    1    1    4Gi       1Gi        -
decay                         1    0.5  512Mi     1Gi        -
ui                            1    0.5  1Gi       1Gi        -
traefik                       1    0.5  1Gi       1Gi        -
init-cluster                  1    0.5  1Gi       1Gi        -
init-turbine-api              1    0.5  1Gi       1Gi        -
load-sample-project           1    0.5  1Gi       1Gi        -
intake-api                    1    0.5  1Gi       1Gi        -
kafka-peer                    1    0.5  1Gi       1Gi        -
keycloak                      1    1    1Gi       1Gi        -
kinesis-peer                  1    0.5  1Gi       1Gi        -
merge-head                    1    0.5  1Gi       1Gi        -
merge-peer                    1    0.5  1Gi       1Gi        -
autoingest                    0    0.5  512Mi     1Gi        -
operator                      1    0.5  1Gi       1Gi        -
postgres                      1    1    1Gi       1Gi        5Gi
prometheus                    0    1    1Gi       1Gi        5Gi
query-head                    1    1    1Gi       1Gi        -
query-peer                    1    1    1Gi       1Gi        -
rabbitmq                      1    0.5  1Gi       1Gi        -
reaper                        1    0.5  512Mi     1Gi        -
redpanda                      1    1    1Gi       1Gi        5Gi
stream-head                   1    0.5  1Gi       1Gi        -
stream-peer                   1    0.5  1Gi       1Gi        -
turbine-api                   1    1    1Gi       1Gi        -
version                       1    0.5  1Gi       1Gi        -
zookeeper                     1    0.5  1Gi       1Gi        -

version

Retrieves the version of the hkt tool.

Usage

hkt version [OPTIONS]

Options

OptionDescription
--helpDisplays summary documentation for this command, and exits.

Example

$ hkt version
v3.10.1

config-report

Display the different variables available in the configuration.

Usage

hkt config-report [OPTIONS]

Options

OptionDescription
-c, --cluster-fileGenerate a report of the current variable definition for the cluster

Example

hkt config-report
SETTING                             VALUE
----------------------------------  ---------------------------------------
admin_email                         "[email protected]"
batch_peer_threads                  1
client_id                           "hdxcli-12345"
cloud                               "gcp"
domain                              "example-domin.tld"
enable_manifest_cache               true
enable_query_auth                   false
heartbeat_timeout                   600
host                                "example-host"
job_purge_age                       7
job_purge_period                    120
kafka_careful_mode                  false
kafka_tls_ca                        ""
kafka_tls_cert                      ""
merge_batch_size                    10000
merge_interval                      "15s"
merge_max_partitions_per_candidate  100
merge_queue_limit                   500
namespace                           "hdxcli-12345"
overcommit                          false
owner                               "example-user"
per_pool_merge_queues               false
pools.0.name                        "query-peer"
pools.0.service                     "query-peer"
premium_storage_class               "premium-rwo"
query_otlp_exporter_enabled         false
region                              "us-west1"
sdk_timeout_sec                     300
stale_job_batch_size                300
stale_job_limit                     3000
stale_job_period                    120
standard_storage_class              "standard-rwo"
stream_partition_count              50
stream_replication_factor           1
task_start_timeout                  21600
turbine_api_init_pools              false
url                                 "http://example-host.example-domin.tld"
use_https_with_s3                   true
use_tls                             false
vacuum_rejects_dry_run              false
vacuum_rejects_max_age              "168h"
vacuum_rejects_period_mins          180

Environment Variables

It is recommended the following environment variables are used in the operation of hkt:

NameDescriptionExample
CLIENT_IDThe Hydrolix licence you would have received when signing up to the platform.hdxcli-abc456
ADMIN_EMAILAn email account that will be used as the admin account of the cluster.[email protected]
OWNERName of the owner for the clusteradmin
HYDROLIX_HOSTHostname for the Hydrolix clustermyhost
HYDROLIX_DOMAINDomain name for the Hydrolix clustermydomain.com
CLOUDThe cloud provider to deploy your cluster too.gcp
REGIONThe region in which the Hydrolix cluster will be deployedus-central1
HKT_VERSIONThe Hydrolix version to be used.v1.2.3
PROJECT_IDThe project_id used for the clusterProd-12345