Preparing your EKS Cluster

Setting up your AWS environment

This guide provides all setup steps required for creating an EKS cluster in order to deploy Hydrolix. We will be using the cloud providers cli tools to build the cluster and configure it, but you could just as easily do this though AWS CloudShell or Terraform/Helm charts.

❗️

AWS Fargate

Hydrolix cannot be deployed in an AWS Fargate environment since we require high performance EBS storage for a number of our stateful infrastructure components.

Preparing an EKS managed cluster

Setup

Configuring your cluster

Configuring your cluster autoscaling

Setup

Prepare your aws, eksctl, and kubectl command line tools

Review the AWS documentation to ensure the following are configured on your local machine

  • aws to interact with the cloud providers many services
  • eksctl to build the EKS cluster
  • kubectl to interact with the cluster

Create a local environment variables file

Environment variables are used throughout the remaining steps as a simple templating mechanism. We recommend putting them into a file that can be brought into scope within your terminal shell.

Replace <> with your values. When working with different AWS accounts, adding your AWS_PROFILE in the file helps to prevent mistakes :)

cat <<<<EOT >>>> env.sh
export AWS_PROFILE=<i.e dev/staging/prod>
export HKT_VERSION=<i.e. v3.22.0>                                                        
export HDX_BUCKET_REGION=<i.e. us-east-2>                                                                   
export HDX_HYDROLIX_URL=<i.e https://my.domain.com>                                                       
export HDX_KUBERNETES_NAMESPACE=<i.e. production-service> 
export HDX_DB_BUCKET_URL=s3://$HDX_KUBERNETES_NAMESPACE
export HDX_ADMIN_EMAIL=<i.e. [email protected]>
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query "Account" --output text)"
export AWS_STORAGE_ROLE="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${HDX_KUBERNETES_NAMESPACE}-bucket"
EOT

Now run source env.sh to bring the variables into scope. Note, if you switch terminal windows these variables will not be available until you run source env.sh again. We have all made that mistake!

Configuring your cluster

Create an EKS cluster

We use eksctl to build the EKS cluster. It takes a simple yaml file which defines the scale and node group definitions along with meta-data. Hydrolix contains a few StatefulSet deployments, therefore we need to add EBS driver in the addons section.

📘

Selecting an instanceType

The choice of instanceType depends on your needs. We strongly recommend the c5n compute range due to its network bandwidth guarantees. You could choose c5n.2xlarge for a small trial deployment, c5n.4xlarge for POC/development, through to c5n.9xlarge in production.

cat > eksctl.yaml << EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: $HDX_KUBERNETES_NAMESPACE 
  region: $HDX_BUCKET_REGION 

addons:
  - name: aws-ebs-csi-driver

iam:
  withOIDC: true

managedNodeGroups:
  - name: nodegroup0 
    instanceType: c5n.2xlarge
    minSize: 3
    maxSize: 30
    desiredCapacity: 7
    volumeSize: 128
    privateNetworking: true
EOF

# lets go ahead and create the cluster based on your settings
eksctl create cluster -f eksctl.yaml

This step can take a while to complete. Thankfully, it provides a lot of progress updates in the terminal

Create a S3 bucket & policy

Hydrolix will be storing all your compressed & indexed data on cloud storage. It is in the same region as the cluster you created in the previous step and shares the same name as your namespace. Easy to remember.

aws s3 mb --region $HDX_BUCKET_REGION  $HDX_DB_BUCKET_URL

Before you can access the bucket you'll need an IAM policy. Here we are specify the Hydrolix permissions required.

read -r -d '' POLICY_DOC << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::${HDX_KUBERNETES_NAMESPACE}",
                "arn:aws:s3:::hdx-public"
            ]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": [
                "arn:aws:s3:::${HDX_KUBERNETES_NAMESPACE}/*",
                "arn:aws:s3:::hdx-public/*"
            ]
        }
    ]
}
EOF

# create the IAM policy
aws iam create-policy --policy-name "${HDX_KUBERNETES_NAMESPACE}-bucket" --policy-document $POLICY_DOC

Create an IAM policy for service accounts

Hydrolix service accounts interact with the cluster using AssumeRoleWithWebIdentity. This is a session token based mechanism managed by an OIDC provider.

When you created the cluster via eksctl it enables an IAM OICD provide in the process. You'll need this information for the service account policy.

export OIDC_PROVIDER="$(aws --region ${HDX_BUCKET_REGION} eks describe-cluster --name ${HDX_KUBERNETES_NAMESPACE} \
    --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")"
    
# check the variable has been set correctly
echo $OIDC_PROVIDER

Now you can create the OIDC managed service account policy and the role

read -r -d '' SA_POLICY_DOC << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${HDX_KUBERNETES_NAMESPACE}:hydrolix"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${HDX_KUBERNETES_NAMESPACE}:turbine-api"
        }
      }
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${HDX_KUBERNETES_NAMESPACE}:vector"
        }
      }
    }
  ]
}
EOF

aws iam create-role --role-name "${HDX_KUBERNETES_NAMESPACE}-bucket" \
    --assume-role-policy-document $SA_POLICY_DOC \
    --description "${HDX_KUBERNETES_NAMESPACE}-bucket"

Finally, attach the service account IAM policy to the service account IAM role.

aws iam attach-role-policy --role-name "${HDX_KUBERNETES_NAMESPACE}-bucket" \
    --policy-arn="arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${HDX_KUBERNETES_NAMESPACE}-bucket"

Create namespace & GP3 performance disks

Hydrolix will be deployed in its own namespace. Lets create that now and make it the default namespace in kubectl

kubectl create namespace $HDX_KUBERNETES_NAMESPACE

# Set as current context in kubectl
kubectl config set-context --current --namespace=$HDX_KUBERNETES_NAMESPACE

Hydrolix deploys some StatefulSet infrastructure. This infrastructure benefits greatly from high performance EBS storage due to the volume of data we are processing. Therefore, you will need to provide high performance GP3 disks.

cat > gp3.yaml << EOF
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOF

# make this the standard storage type
kubectl apply -f gp3.yaml

👍

EKS Hydrolix Cluster

That is the base cluster preparation done. You still need to enable autoscaling so that node groups can grow and shrink as you scale services horizontally or vertically.

Configuring your cluster autoscaling

Deploy the metric server

The metrics server is a dependency for autoscaling. You can deploy via the url endpoint.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Create autoscaler node group policy

You need to provider the autoscaler with all the autoscale permissions in the IAM policy for nodes your namespace.

read -r -d '' AUTOSCALER_POLICY_DOC << EOF
{   
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/${HDX_KUBERNETES_NAMESPACE}": "owned"
                }
            }
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:DescribeLaunchTemplateVersions",
                "autoscaling:DescribeTags",
                "autoscaling:DescribeLaunchConfigurations"
            ],
            "Resource": "*"
        }
    ]
}
EOF

## create the IAM policy
aws iam create-policy --policy-name eks-${HDX_KUBERNETES_NAMESPACE}-autoscaler --policy-document AUTOSCALER_POLICY_DOC

Create autoscaler service account permissions

read -r -d '' AUTOSCALER_TRUST_DOC << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:kube-system:cluster-autoscaler"
        }
      }
    }
  ]
}
EOF

# create a new role base on this IAM policy
aws iam create-role --role-name "eks-${HDX_KUBERNETES_NAMESPACE}-autoscaler" \
    --assume-role-policy-document $AUTOSCALER_TRUST_DOC --description "eks-${HDX_KUBERNETES_NAMESPACE}-autoscaler"

# attach the role to the the cluster
aws iam attach-role-policy --role-name "eks-${KUBERNETES_CLUSTER}-autoscaler" \
--policy-arn="arn:aws:iam::${AWS_ACCOUNT_ID}:policy/eks-${KUBERNETES_CLUSTER}-autoscaler"

Deploy your cluster autoscaler autodiscovery

You need to download and modify the cluster-autoscaler-autodiscover.yaml

curl -o cluster-autoscaler-autodiscover.yaml https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

You need to replace the placeholder with your actual namespace. This can be done simply with the following command - or manually if you prefer.

sed -i'' -e 's/<YOUR CLUSTER NAME>/'"${HDX_KUBERNETES_NAMESPACE}"'/g' cluster-autoscaler-autodiscover.yaml

Then apply the deployment

kubectl apply -f cluster-autoscaler-autodiscover.yaml

The last step is to annotate the service account

kubectl annotate serviceaccount cluster-autoscaler -n kube-system \
eks.amazonaws.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/eks-${HDX_KUBERNETES_NAMESPACE}-autoscaler

👍

EKS preparation complete

Congratulations, you are now ready to deploy Hydrolix.