If you are to import data from an existing bucket you will need to update the bucket allow-list. This will enable the batch-archiecture to see the bucket and retrieve data from those locations.
To enable access to a bucket the following command should be used:
$ hdxctl update <client_id> <cluster_id> --bucket-allowlist "<bucketname>"
$ hdxctl update hdxcli-12345678 hdx-abcdefgh --bucket-allowlist "mybucket"
If the source bucket is in a different AWS account from the one Hydrolix is deployed within, then you will also need to add a bucket policy to the source bucket allowing cross account access. Enable the Source Bucket
The update command is non-additive, ensure to include all your buckets each time you add one.
Updated 9 months ago