Prepare a Cluster
To prepare a Linode environment for a Hydrolix deployment, use the Linode Cloud console. You'll need to create a new object storage bucket with an access key, a Kubernetes cluster, and a PostgreSQL instance.
Variables
You'll need seven pieces of information to create environment variables for easy management of your Hydrolix Kubernetes cluster.
Info | Description | Example |
---|---|---|
Hydrolix URL | The URL of your Hydrolix installation. You will use this for the config, data, and query APIs as well as the web user interface. | https://my.domain.com |
Kubernetes Namespace | The namespace of the Linode Kubernetes Engine you create in the Create the Kubernetes Cluster section below. | my-production-service |
Bucket Region | The Linode geographical region in which your bucket resides. You will determine this in the Create Your Bucket section below. | us-southeast-1 |
Bucket URL | The Linode bucket URL from Create Your Bucket, below. | https://hdx-company.us-southeast-1.linodeobjects.com |
Bucket Endpoint | The Linode bucket endpoint from Create Your Bucket, below | https://us-southeast-1.linodeobjects.com |
Access Key | A bucket access key you create in the Object Storage section of the Linode Cloud Console | QJN31Z84ED71HS2BEP5D |
Secret Key | A secret key you create in the Object Storage section of the Linode Cloud Console | wtJ7smrtB5pEWYuLiNr0rBAEaDUTq1kXhFsyFr99 |
Administrative E-mail | An e-mail address used for verification during the setup process | [email protected] |
Collect this information as you set up services so you can put them in an env.sh
script for easy access later in the process.
Create Your Bucket
- Begin by logging in to the Linode Cloud console. Click the Create button. From the resulting list, select the "Bucket" option:
- Enter a label (also known in the Linode UI as a "name") for the bucket. Hydrolix will store all of your row data in this bucket. Choose a region, preferring the region where you intend to configure your Kubernetes cluster:
- Record the bucket URL for later use. You can find it in the "Buckets" list under the title of the bucket.
- Record the bucket region identifier for later use in the Set Environment Variables section. A full list of region codes can be found in the Linode documentation. For example, Atlanta, GA's identifier is
us-southeast-1
. - Record the bucket endpoint. This is simply your bucket URL minus the first part of the bucket name. For example, if your bucket URL is
https://hdx-company.us-southeast-1.linodeobjects.com
, then the bucket endpoint ishttps://us-southeast-1.linodeobjects.com
Create a Bucket Access Key and Secret Key
Next, create an access key so Hydrolix can securely connect to your storage bucket. Within the Linode Cloud console, navigate to the "Object Storage" page. Click on "Access Keys" to view the access keys tab:
Click the "Create Access Key" button. Enter a name for the key. Enable the "Limited Access" toggle. Grant the key "Read/Write" access to the bucket you created in the previous step. Click the "Create Access Key" button to create the keys.
A dialog box will appear in the Linode Cloud console, displaying your access key and your secret key.
Store your secret key somewhere secure, like a password manager: Linode will only show you this key once. You can view the access key at any time, but it's best to save it now since you'll need it to create your cluster. Once you've recorded the keys, click the "I Have Saved My Secret Key" button to dismiss the dialog box.
Create the Kubernetes Cluster
Now it's time to create the Kubernetes cluster where you'll deploy Hydrolix services. In the Linode Cloud console, click the Create button. From the resulting list, select the "Kubernetes" option:
- Enter a name for your cluster.
- Select the same region used to create your bucket.
- Use the latest version of Kubernetes available in the dropdown list. You can reference the Hydrolix release notes if you want to verify whether a version of Kubernetes is supported.
- Choose the HA option, for high availability.
- Use three Dedicated 32 GB nodes in the list below by clicking its "Add" button. (See the last line in the screenshot below.)
- Click "Create Cluster" to instantiate your cluster.
A few minutes later, you'll see the cluster summary page show you that your three nodes are up and running.
Set Environment Variables
Command-line access is much easier if you set up the below environment variables using an env.sh
script.
export HDX_HYDROLIX_URL=<Hydrolix URL>
export HDX_KUBERNETES_NAMESPACE=<Kubernetes Namespace>
export HDX_BUCKET_REGION=<Bucket Region>
export HDX_DB_BUCKET_URL=<Bucket URL>
export HDX_DB_BUCKET_ENDPOINT=<Bucket Endpoint>
export HDX_ADMIN_EMAIL=<Administrative E-mail>
For instance, following the values from the examples above:
export HDX_HYDROLIX_URL=https://my.domain.com
export HDX_KUBERNETES_NAMESPACE=hdx-company
export HDX_BUCKET_REGION=us-southeast-1
export HDX_DB_BUCKET_URL=https://hdx-company.us-southeast-1.linodeobjects.com
export HDX_DB_BUCKET_ENDPOINT=https://us-southeast-1.linodeobjects.com
export [email protected]
Load these environment variables with the following command:
source env.sh
Configure the Kubernetes Cluster
After creating the cluster, you should see the following cluster summary page:
First, download your kubeconfig.yaml
file. Save it to your local computer. You'll use this file to connect to your cluster on the command line.
Next, configure autoscaling for your cluster:
- Click "Autoscale Pool" to open a dialog where you can configure autoscaling.
- Enable the "Autoscaler" toggle.
- For the "Max" setting, enter a value of "20".
- Click the "Save Changes" button to update your cluster configuration.
You can now connect to your cluster with kubectl
:
kubectl --kubeconfig <configfile> get nodes
For easier access, export the configuration filename in the KUBECONFIG environment variable. Following our example where you downloaded a hdx-company-kubeconfig.yaml file:
export KUBECONFIG=${PWD}/hdx-company-kubeconfig.yaml
kubectl get nodes
Create a k8s Namespace
Create a dedicated namespace
in your Kubernetes Cluster:
kubectl create namespace $HDX_KUBERNETES_NAMESPACE
For ease of use, set your new namespace as a default:
kubectl config set-context --current --namespace="$HDX_KUBERNETES_NAMESPACE"
Set Bucket Access Key and Secret within Kubernetes
Run the following using the bucket access key and secret key you obtained from the corresponding step in which you generated them.
kubectl create secret generic curated \
--from-literal=AWS_ACCESS_KEY_ID='{YOUR_ACCESS_KEY_ID}' \
--from-literal=AWS_SECRET_ACCESS_KEY='{YOUR_SECRET_ACCESS_KEY}'
This sets the bucket access key and secret key in a Kubernetes secret. The operator
pod will aggregate all the values from the curated
Kubernetes secret and merge them into the dynamically generated general
secret. This general
secret is accessible for all resources within your given namespace, giving your Hydrolix cluster's pods and their containers access to the bucket credentials.
Updated about 9 hours ago