Deploy Hydrolix

Hydrolix deployments follow the Kubernetes operator pattern. To deploy Hydrolix, generate an operator configuration (operator.yaml), a secrets file (secrets.yaml), and a custom resource Hydrolix configuration (hydrolixcluster.yaml). You'll use these files to deploy Hydrolix on your Kubernetes cluster.

Prerequisite: Environment Variables

These CLI commands require you to set environment variables before generating the configuration. See Prepare your AKS Cluster for more information about the required inputs.

Configure and Deploy the Hydrolix Operator⚓︎

The operator-resources command generates the Kubernetes resource definitions required for deploying the operator, service accounts, and role permissions. The operator manages all Hydrolix cluster deployments. Run the following command to generate a YAML operator configuration file for your cluster:

curl "https://www.hydrolix.io/operator/latest/operator-resources?namespace=${HDX_KUBERNETES_NAMESPACE}" > operator.yaml

Next, use the Kubernetes command line tool (kubectl) to apply the generated configuration to your Kubernetes cluster:

kubectl apply -f operator.yaml

Generate and Prepare an Azure Static Secret⚓︎

This step allows your Hydrolix cluster to use cloud storage. List the keys for your account and resource group:

az storage account keys list --resource-group $HDX_AZURE_RG --account-name $HDX_AZURE_STORAGE_ACCOUNT

Find the key1 value and place it in a file called secrets.yaml:

---
apiVersion: v1
kind: Secret
metadata:
  name: curated
  namespace: $HDX_KUBERNETES_NAMESPACE
stringData:
  AZURE_ACCOUNT_KEY: <key1>
type: Opaque

Next, use the Kubernetes command line tool (kubectl) to apply the generated secret to your Kubernetes cluster:

eval "echo \"$(cat secrets.yaml)\"" > secrets.yaml
kubectl apply -f secrets.yaml

Configure and Deploy a Hydrolix Cluster⚓︎

The hkt tool's hydrolix-cluster command generates the hydrolixcluster.yaml deployment file. Alternatively, you can use the file below as a template.

We provide scale profiles for various cloud providers and deployment sizes. You can optionally specify a profile using the scale-profile flag. By default, Hydrolix uses a minimal profile. Add the following to a file named hydrolixcluster.yaml to generate a YAML cluster configuration file for a dev scale deployment:

hydrolixcluster.yaml

apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: hdx
  namespace: ${HDX_KUBERNETES_NAMESPACE}
spec:
  admin_email: ${HDX_ADMIN_EMAIL}
  db_bucket_region: ${HDX_BUCKET_REGION}
  db_bucket_url: ${HDX_DB_BUCKET_URL}
  env: {}
  hydrolix_name: hdx
  hydrolix_url: ${HDX_HYDROLIX_URL}
  ip_allowlist:
  - 0.0.0.0/0 #TODO: Replace this with your IP address in CIDR notation, eg. 12.13.14.15/32
  overcommit: false
  scale_profile: dev

The above config will deploy, among other things, a default, internal Postgres instance that is non-HA. If you want to run a more resilient version, read our Deploy Production Postgres guide.

Use the following command to replace the environment variables above with their values:

eval "echo \"$(cat hydrolixcluster.yaml)\"" > hydrolixcluster.yaml

Don't forget to add your IP address to the allowlist. You can get your IP address by running curl -s ifconfig.me.

Manually Edit Configuration Files

You can also edit the hydrolixcluster.yaml to tune each deployment to your resource requirements.

Next, use the Kubernetes command line tool (kubectl) to apply the generated configuration to your Kubernetes cluster:

kubectl apply -f hydrolixcluster.yaml

Create Your DNS Record⚓︎

Next, create a DNS record so you can access your cluster. Run the following command to retrieve the traefik record:

kubectl get service/traefik --namespace=$HDX_KUBERNETES_NAMESPACE

You should see output similar to the following:

1 2	`NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 8089/TCP 68m traefik LoadBalancer 10.64.14.42 34.66.136.134 80:31708/TCP,9000:32344/TCP 2m50s`

Consult with your network administrator to configure DNS to map your hostname to the external IP.

Check Deployment Status⚓︎

You can now check the status of your deployment. Run the followingkubectl command to see the status of all pods in your cluster:

kubectl get pods --namespace $HDX_KUBERNETES_NAMESPACE

You should see output similar to the following:

NAME                             READY   STATUS      RESTARTS   AGE
autoingest-658f799497-czw59      1/1     Running     0          5m44s
batch-head-bcf7869bc-fm794       1/1     Running     0          5m46s
batch-peer-555df86d8-svlmw       2/2     Running     0          5m45s
decay-78775df79d-ppxpf           1/1     Running     0          5m45s
init-cluster-v3-16-0-6fcml       0/1     Completed   0          5m45s
init-turbine-api-v3-16-0-jqt4m   0/1     Completed   0          5m46s
intake-api-747cdd5d4d-vrsjm      1/1     Running     0          5m45s
keycloak-68fcff9b69-p4lt5        1/1     Running     0          5m46s
load-sample-project-nv8dl        1/1     Running     0          5m44s
merge-head-7df478d57-7qgwn       1/1     Running     0          5m44s
merge-peer-dbb68cc75-c8fl4       1/1     Running     0          5m45s
merge-peer-dbb68cc75-ntwpj       1/1     Running     0          5m45s
operator-55d4dfff6f-pktrl        1/1     Running     0          7m10s
postgres-0                       1/1     Running     0          5m46s
prometheus-0                     2/2     Running     0          5m45s
query-head-65bf688594-l9prj      1/1     Running     0          5m45s
query-peer-67dfcccb56-h6rkw      1/1     Running     0          5m44s
rabbitmq-0                       1/1     Running     0          5m46s
reaper-647d474f5-mfgww           1/1     Running     0          5m44s
redpanda-0                       2/2     Running     0          5m46s
redpanda-1                       2/2     Running     0          5m23s
redpanda-2                       2/2     Running     0          3m38s
stream-head-6ccc9779df-7jvzf     1/1     Running     0          5m43s
stream-peer-6db9464bd5-cgq6x     2/2     Running     0          5m44s
traefik-6f898fd647-lxf84         2/2     Running     0          5m43s
turbine-api-65d44c7d54-crpcm     1/1     Running     0          5m43s
ui-5b8bc9c9d4-pgjtv              1/1     Running     0          5m43s
validator-769ff76ddb-5mm5w       2/2     Running     0          5m43s
vector-557q5                     1/1     Running     0          4m58s
vector-5ttd4                     1/1     Running     0          5m46s
vector-5z8zq                     1/1     Running     0          5m46s
vector-qnpn9                     1/1     Running     0          5m46s
vector-r8pj6                     1/1     Running     0          3m4s
version-848c8c964c-j2khx         1/1     Running     0          5m43s
zookeeper-0                      1/1     Running     0          5m46s

You can also check your cluster status in the Microsoft AKS console.

Enable IP Access and TLS⚓︎

Configure IP Access control and a TLS certificate. You can find instructions in Secure a Kubernetes Cluster.

The Final Step⚓︎

You should have received an email that will now allow you to set a password and login. If you do not receive this email, please feel free to contact us at mailto:support@hydrolix.io and we'll happily assist you.