Skip to content

Deploy Hydrolix

Hydrolix deployments follow the Kubernetes operator pattern.

To deploy Hydrolix, generate an operator configuration (operator.yaml) and a custom resource Hydrolix configuration (hydrolixcluster.yaml).

You'll use these files to install and run Hydrolix on your Kubernetes cluster.

Overview⚓︎

  1. Configure and deploy the Hydrolix operator.
  2. Configure and deploy a Hydrolix cluster.
  3. (optional) Use external PostgreSQL.
  4. Create a DNS record.
  5. Verify the cluster.
  6. Log in.

Configure and deploy the Hydrolix operator⚓︎

Hydrolix publishes a service under www.hydrolix.io which provides software descriptions required to install Kubernetes resources for the operator and cluster software.

Prerequisites: Ensure environment variables are set

Instructions on this page depend on settings in environment variables decided before generating configurations. See Prepare your GKE Cluster about the required inputs.

The operator-resources command generates the Kubernetes resource definitions required for deploying the operator, service accounts, and role permissions. The operator manages all Hydrolix cluster deployments. Run the following command to generate a YAML operator configuration file for your cluster:

1
2
3
4
5
curl --get \
  --data "namespace=${HDX_KUBERNETES_NAMESPACE}" \
  --data "gcp-storage-sa=${GCP_STORAGE_SA}" \
  -- "https://www.hydrolix.io/operator/latest/operator-resources" \
  > operator.yaml

Apply the generated configuration to your Kubernetes cluster using the kubectl Kubernetes command-line tool:

1
kubectl apply -f operator.yaml

Configure and deploy a Hydrolix cluster⚓︎

Start with the following Hydrolix cluster manifest template file. This manifest file describes the HydrolixCluster Kubernetes custom resource definition (CRD). The settings in the spec key control the behavior of the cluster, especially the tunables. This example includes minimal settings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: hydrolix.io/v1
kind: HydrolixCluster
metadata:
  name: hdx
  namespace: ${HDX_KUBERNETES_NAMESPACE}
spec:
  acme_enabled: true
  admin_email: ${HDX_ADMIN_EMAIL}
  db_bucket_region: ${HDX_BUCKET_REGION}
  db_bucket_url: ${HDX_DB_BUCKET_URL}
  env: {}
  hydrolix_name: hdx
  hydrolix_url: ${HDX_HYDROLIX_URL}
  ip_allowlist:
  - 0.0.0.0/0 # -- Default, allow all. Limit by replacing with IP addresses in CIDR notation, like 12.13.14.15/32
  overcommit: false
  scale_profile: dev

Manually edit the configurable lines in the above template file or programmatically replace the variables from your environment variables with a command like envsubst.

1
envsubst < template.yaml > hydrolixcluster.yaml

Standard scale profiles size a cluster to fit different cloud providers and deployment sizes. Default installations use a dev profile. Specify an alternate profile using the scale_profile flag.

Set IP access control⚓︎

The default ip_allowlist is 0.0.0.0/0 which allows incoming network connections into the cluster's service IP.

Configure a network access control list (ACL) to limit which client IPs which can connect to the service IP of your Hydrolix cluster.

See effects of IP access controls, especially on the Hydrolix ACME client.

Enable TLS⚓︎

These instructions recommend TLS certificate provisioning with the Hydrolix ACME client. When the acme_enabled tunable is true, the cluster runs this service.

See Enable TLS for other TLS certificate provisioning options.

Certificate provisioning depends on both the DNS and running cluster

External Certification Authorities (CAs) can't complete the challenge response verification to issue a certificate before

  • the hostname is installed in public DNS
  • the CA's validators can reach the cluster over plaintext HTTP

Verify the TLS certificate in a later step.

Apply HydrolixCluster config⚓︎

Apply the configuration to your Kubernetes cluster using kubectl

1
kubectl apply -f hydrolixcluster.yaml

Use external PostgreSQL⚓︎

This step is optional.

Hydrolix provisions a single internal PostgreSQL pod by default. Skip this section unless you want to use an external PostgreSQL service.

For production deployments that need high availability or managed backups, CloudNativePG or Google Cloud SQL can be used instead.

Follow Deploy Production PostgreSQL before ingesting any data.

Create a DNS record⚓︎

To collect the public, external cluster IP for the DNS record, use this command to retrieve the traefik record:

1
2
3
$ kubectl get service/traefik --namespace=$HDX_KUBERNETES_NAMESPACE
NAME          TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                AGE
traefik       LoadBalancer   10.64.14.42    34.66.136.134   80:31708/TCP,9000:32344/TCP            2m50s
Restarting services if kubectl reports an error
Example Error Message
1
Error from server (NotFound): services "traefik" not found

If you receive an error, try restarting the operator with

1
kubectl -n $HDX_KUBERNETES_NAMESPACE rollout restart deployment operator

Use your authoritative DNS provider to create a DNS record for the hydrolix_url name corresponding to the external IP.

Verify the cluster⚓︎

Check deployment status⚓︎

You can now check the status of your deployment. Run the followingkubectl command to see the status of all pods in your cluster:

1
kubectl get pods --namespace $HDX_KUBERNETES_NAMESPACE
List running pods output

You should see output similar to the following. The pod listing changes between versions.

Sample Output Listing Running Pods
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
NAME                                  READY   STATUS      RESTARTS   AGE
check-bucket-access-v5soln3bp-p67xk   0/1     Completed   0          2m50s
hydrologs-0646fb7a-cc6649766-lk2vh    2/2     Running     0          2m48s
init-acme-509c50f0-kmjfd              0/1     Completed   3          2m49s
init-cluster-v4-17-0-b86e0119-dkxdp   0/1     Completed   0          2m50s
init-turbine-api-v4-17-0-lj45f        0/1     Completed   0          2m50s
intake-head-858bb875b-58l2b           2/2     Running     0          2m44s
intake-head-858bb875b-gvnn8           2/2     Running     0          2m44s
keycloak-575d78ff58-8kfx6             1/1     Running     0          2m48s
load-sample-project-96jdt             1/1     Running     0          2m49s
merge-cleanup-29199670-dr7q8          0/1     Completed   0          2m48s
merge-controller-b9cdfbc6b-gthf2      1/1     Running     0          2m48s
merge-peer-6cdbdf5f8d-7b8pf           2/2     Running     0          2m46s
merge-peer-ii-957b77b85-vnpsb         2/2     Running     0          2m45s
merge-peer-iii-7cb7977bf7-tkpdm       2/2     Running     0          2m46s
monitor-ingest-7fb565c97d-x8pdn       1/1     Running     0          2m46s
operator-586f67bcb6-wxfjm             1/1     Running     0          2m57s
pushgateway-f6864dd79-w6d6z           1/1     Running     0          2m48s
query-head-64cf6c7f5f-zwqzk           1/1     Running     0          2m45s
query-peer-5cd45fc74d-m25zt           1/1     Running     0          2m44s
rabbitmq-0                            1/1     Running     0          2m49s
redpanda-0                            2/2     Running     0          2m50s
refresh-job-statuses-29199674-nqqv5   0/1     Completed   0          50s
stale-job-monitor-29199670-vfxgp      0/1     Completed   0          2m48s
task-monitor-29199674-9lzvr           0/1     Completed   0          50s
traefik-6cc4c57894-7qvjr              3/3     Running     0          2m46s
traefik-6cc4c57894-q5qkl              3/3     Running     0          2m46s
turbine-api-7564c994b9-9lwjk          1/1     Running     0          2m47s
turbine-api-7564c994b9-pg7j4          1/1     Running     0          2m47s
ui-5d9d5ccb46-vhcz6                   1/1     Running     0          2m46s
usagemeter-648ff5b88c-tvmz7           1/1     Running     0          2m50s
validator-5bd656d65f-mbbvw            2/2     Running     0          2m14s
vector-2nftt                          1/1     Running     0          2m9s
vector-cw5wq                          1/1     Running     0          2m9s
vector-klj5l                          1/1     Running     0          89s
vector-p47pb                          1/1     Running     0          2m10s
vector-q4lnn                          1/1     Running     0          119s
vector-qxgbk                          1/1     Running     0          2m9s
vector-t8fpz                          1/1     Running     0          2m
vector-v2fpq                          1/1     Running     0          2m9s
vector-vtwvr                          1/1     Running     0          2m49s
vector-xwmp6                          1/1     Running     0          2m11s
version-5c69568597-tpg5k              1/1     Running     0          2m47s
zookeeper-0                           1/1     Running     0          2m49s
zookeeper-1                           1/1     Running     0          2m34s
zookeeper-2                           1/1     Running     0          107s

You can also check your cluster status in the Google Cloud console.

Verify TLS certificate⚓︎

Connect to the cluster's service IP by name using a browser or command-line utility like curl to verify the certificate.

1
2
3
$ curl --output /dev/null --fail -- "${HDX_HYDROLIX_URL}"
$ echo $?
0

1
2
3
4
5
6
7
8
9
$ curl --output /dev/null  --fail  -- "${HDX_HYDROLIX_URL}"
curl: (60) SSL: no alternative certificate subject name matches target host name 'hostname.hydrolix.live'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
$ echo $?
60

See Certificate troubleshooting for more help.

Log in⚓︎

You should have received an email with instructions to set a password and login. If you don't receive this email, or have trouble logging in, try these things:

  • Verify the email address in your hydrolixcluster.yaml file is correct and that you can receive mail sent to it.
  • Try the Forgot my password option on the login page.
  • If those two steps fail, contact us at support@hydrolix.io.

Once you are able to log in to your Hydrolix cluster, setup is complete, and you are ready to store and query data. Proceed to the next step only if you want to query your data using the Hydrolix Connector for Apache Spark.

(Hydrolix Connector for Apache Spark only) Add a credential to the storage bucket⚓︎

To query the Hydrolix Cluster using the Hydrolix Connector for Apache Spark, configure a credential for your storage bucket. The connector reads partitions through presigned URLs issued by the Hydrolix cluster. On GKE, the cluster's default workload identity credentials can't sign GCS URLs, and signing requires a service account private key. Attaching a service account credential to the storage bucket gives the cluster the key it needs to issue presigned URLs. The following steps walk you through generating a new credential and updating your storage bucket to use the credential.

Step 1: Create a credential⚓︎

Create a Google service account credential. Download the credentials.json file containing your keys from Google. If you need to create a new credential, or you're not sure where to find this file, see Google's Create credentials for a service account instructions.

Installing the credential into the Hydrolix cluster is best accomplished in the UI.

Select Add new > Credential in your Hydrolix cluster UI. Fill out the form with the following:

  • Supply a name and description for your credential
  • Select gcp_service_account_keys for Cloud Provider Type
  • Upload your Google credentials file
  • Review the fields filled in from the supplied credentials file then select Create credential

GCP credential creation form in Hydrolix UI

Required credential attributes with examples:

Example value
Name my_gcp_credential
Description A credential for the default GCP bucket
Cloud Provider Type gcp_service_account_keys (drop-down list)
Upload Credential JSON (optional): a GCP credentials file name
Type service_account (drop-down list)
Project Id hdx-cluster-docs
Private Key Id private_key_id
Private Key private_key_goes_here
Client Email user@hdx-cluster-docs.iam.gserviceaccount.com
Client Id {id}
Auth Uri https://accounts.google.com/o/oauth2/auth
Token Uri https://oauth2.googleapis.com/token
Auth Provider X509 Cert Url https://www.googleapis.com/oauth2/v1/certs
Client X509 Cert Url https://www.googleapis.com/robot/v1/metadata/x509

You can review your new credential by navigating to Security > Credentials, then selecting your credential by name. You can also do this using the API through the List credentials endpoint. You will need your credential ID for the next step.

Step 2: Attach the credential to the storage bucket⚓︎

Using the update storage endpoint, in the next steps you will attach your newly created credential to the storage bucket.

Set settings.credential_id to the ID of the credential you created in the previous step. This is the Credential ID in the UI or uuid in the API response to List credentials.

Credential ID in the UI⚓︎

Credential ID displayed in Hydrolix UI

1
2
3
4
5
Credential ID: d70d9fc4-8422-496c-98ce-f59aed82099d
Name: gcp service account credential
Description: A test credential for a gcp storage bucket
Cloud Provider Type: gcp_service_account_keys
HDX credential key: K1E4FB4F63DC14C9CAD3C1ED56D412FA0

Credential ID in the API response⚓︎

Credential ID in API response JSON

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[
    {
        "name": "gcp service account credential",
        "type": "gcp_service_account_keys",
        "cloud": "gcp",
        "org": "ae5e3698-b13a-4f8f-ab82-ad2fa391a1a8",
        "description": "A test credential for a gcp storage bucket",
        "uuid": "d70d9fc4-8422-496c-98ce-f59aed82099",
        ...
    }

Append ?force_operation=true to the URL.

The following is an example cURL request attaching a credential to the default Google storage bucket:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
curl --request PUT \
     --url https://{hdx-cluster-host}/config/v1/orgs/{org-id}/storages/{bucket-id}\?force_operation\=true \
     --header 'accept: application/json' \
     --header 'authorization: Bearer {token}' \
     --header 'content-type: application/json' \
     --data '
{
  "settings": {
    "bucket_path": "/",
    "is_default": true,
    "bucket_name": "{bucket-name}",
    "cloud": "gcp",
    "credential_id": "{credential-id-as-a-string}",
    "region": "{region}"
  },
  "name": "hdx_primary",
  "uuid": "{bucket-id}",
  "description": "The default google storage bucket"
}
'

Once you've completed these steps, your cluster can receive queries from the Hydrolix Connector for Apache Spark.