Cloud Resources

This page explains how to configure and manage cloud-based resources for e6data connectivity and storage.

The Cloud Resource component is a cross-account role that facilitates access to the user's S3 buckets and Glue catalog for workspace and cluster-level components.

Create Cloud Resource

1. Enter a name for your cloud resource.

2. Click on the provided link to be redirected to your AWS CloudFormation page.

3. Review the inputs on the AWS CloudFormation page and proceed by executing the script. Here are the following inputs available in AWS CloudFormation:

   1. BucketNames

   2. VPCEndpointId

   3. E6dataAccountId

   4. ExternalId

4. CloudFormation creates certain permissions when the script is executed. The following permissions are included:

   1. Lambda Permissions:

      • Creates AWS::Lambda::Permission (LambdaInvokePermission) to allow a Lambda function to execute.

   2. IAM Role for Lambda:

      • Creates an IAM role (LambdaExecutionRole) that allows the Lambda function to create IAM policies, attach them to the cross-account role, and update bucket policies.

   3. Manage Bucket Policies Lambda:

      • Deploys a Lambda function (ManageBucketPoliciesLambda) that creates bucket policies for the specified buckets (or for all buckets if * is provided). These policies allow clusters to access data via a VPC endpoint.

      • Additionally, the Lambda function creates a policy that allows the cross-account role to access the specified buckets.

   4. Custom Resource:

      • Creates a custom resource (ManageBucketPoliciesCustomResource) that triggers the Lambda function, accepting inputs such as BucketNames, VPCEndpointId, CrossAccountGlueRole, and Random.

   5. Cross-Account IAM Role:

      • Establishes a cross-account IAM role (CrossAccountGlueRole) with policies (GlueCrossAccountPolicy and e6data-cross-account-s3-policy) that grant Glue and S3 access. Roles in our account will assume this role.

5. After successful execution, the output panel of CloudFormation will display the Cross-Account IAM Role and any failed buckets. The customer needs to update the Cross-Account IAM Role in the e6data UI.

Manual Creation of Roles (optional)

Cross-Account Role Creation in the User Account: We create a cross-account role in the customer's account with the following access policies and a trust policy:

• S3 Bucket Policy: This policy grants the necessary permissions for the engine to access data stored in S3 buckets.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectTagging",
        "s3:GetObjectVersion",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::<BUCKET>/*",
        "arn:aws:s3:::<BUCKET>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<BUCKET>",
        "arn:aws:s3:::<BUCKET>"
      ]
    }
  ]
}

• Glue Policy: This policy allows e6data to access the Glue catalog for querying databases and tables.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabase*",
        "glue:GetTable*",
        "glue:GetPartitions"
      ],
      "Resource": "*"
    }
  ]
}

• Trust Policy: This policy permits e6data's account to assume the cross-account role. It includes an ExternalId for enhanced security.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::298655976287:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "FB5441BD-CDCE-4258-A161-680434024805"
        }
      }
    }
  ]
}

• Additionally, when the CloudFormation stack is deleted, this role, along with its associated access and trust policies, will be automatically removed.

• Bucket Access Policy: e6data creates a bucket access policy that allows it to access the S3 bucket via a VPC endpoint. This ensures that data is transmitted securely within AWS’s internal network. The policy is appended to the existing bucket policy, and only the newly created policy is removed when necessary, to avoid disrupting your current workflow.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::<BUCKET>/*",
      "Condition": {
        "StringEquals": {
          "aws:SourceVpce": "<E6DATA_VPC_ENDPOINT>"
        }
      }
    }
  ]
}

Cloud resource creation form

How to create a Cloud Resource