AWS (CF) Bucket Access

AWS Cloud Formation

If you are to import data from an existing bucket you will need to update the bucket allow-list. This will enable the batch-archiecture to see the bucket and retrieve data from those locations.

To enable access to a bucket the following command should be used:

$ hdxctl update <client_id> <cluster_id> --bucket-allowlist "<bucketname>"

For example:

$ hdxctl update hdxcli-12345678 hdx-abcdefgh --bucket-allowlist "mybucket"

If the source bucket is in a different AWS account from the one Hydrolix is deployed within, then you will also need to add a bucket policy to the source bucket allowing cross account access. Enable the Source Bucket

🚧

Note:

The update command is non-additive, ensure to include all your buckets each time you add one.

Configure Source Bucket

Normally, a source bucket will be located in the same AWS account as the Hydrolix Platform. If the source bucket is in a different account you will need to enable access permissions within AWS. AWS has some good documentation on this here

Here is an example S3 access policy to apply to the source bucket. The Principle represents the AWS account Id the Hydrolix Platform is contained within.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::12346579801:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket/*",
                "arn:aws:s3:::mybucket"
            ]
        }
}