Appearance

Powered by Platform One

How-to: S3 ​

Overview ​

P1 utilizes IRSA to leverage AWS S3. We currently do not allow the usage of access/secret keys without the approval of Cyber. To utilize AWS S3, you must already have an S3 bucket provisioned. If it has not been provisioned, you will need to have Party Bus MDO provision an S3 bucket.

Kustomization Manifests ​

To utilize your IRSA S3 account, you need to add the serviceAccountName role to your kustomization deployment.yaml manifest. Example:

yaml
serviceAccountName: s3-role

To use the correct S3 bucket name, the S3_BUCKET_NAME variable prefix can be generated using the below:

yaml
envFrom:
  - configMapRef:
      name: cluster-prefix
env:
  - name: NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: S3_BUCKET_NAME
    value: pb-mdo-$(NAMESPACE)-$(CLUSTER_PREFIX)

If your application requires an endpoint and region, you can specify it below:

deployment.yaml

yaml
env:
  - name: S3_ENDPOINT_URL
    value: https://s3.us-gov-west-1.amazonaws.com

INFO

Some SDKs expect explicit values for the AWS_REGION variable. If so, set the following in your deployment.yaml.

deployment.yaml

yaml
env:
  - name: AWS_REGION
    value: us-gov-west-1

FIPS S3 Endpoints ​

To set up S3 to use the AWS S3 FIPS Endpoint, use the following environment variable in your deployment:

deployment.yaml

yaml
env:
  - name: S3_ENDPOINT_URL
    value: https://s3-fips.us-gov-west-1.amazonaws.com

Temporary S3 Credentials ​

Teams may also request temporary S3 credentials to their S3 buckets for the purpose of Extract, Transfer, and Loading data to S3. These are one-time, short-term 12-hour credentials. Please send all requests to the Party Bus MDO team.

Note

S3 is not a traditional file system (block storage) but instead is object storage. This means that folder paths do not need to be created when generating files, as the first time a file is created, it will create the path as part of the file name.

SDK Usage Examples ​

Java ​

We recommend using AWS SDK for Java, as it supports IRSA credentials by default.

1. Maven dependencies (pom.xml):

xml
<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
        <version>1.12.542</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-sts</artifactId>
        <version>1.12.542</version>
    </dependency>
</dependencies>

2. Java example code to list S3 buckets using IRSA:

java
import com.amazonaws.auth.WebIdentityTokenCredentialsProvider;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.Bucket;

public class IRSAExample {
    public static void main(String[] args) {
        String region = System.getenv("AWS_REGION");
        if (region == null || region.isEmpty()) {
            System.err.println("AWS_REGION environment variable is not set.");
            return;
        }
        try {
            AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
                .withCredentials(WebIdentityTokenCredentialsProvider.create())
                .withRegion(region)
                .build();
            System.out.println("S3 Buckets:");
            for (Bucket bucket : s3Client.listBuckets()) {
                System.out.println("- " + bucket.getName());
            }
        } catch (Exception e) {
            System.err.println("Error accessing AWS S3 using IRSA: " + e.getMessage());
            e.printStackTrace();
        }
    }
}

Spring Boot ​

1. Add AWS dependencies in Gradle (build.gradle):

groovy
dependencies {
  implementation platform('io.awspring.cloud:spring-cloud-aws-dependencies:3.1.1')
  implementation 'io.awspring.cloud:spring-cloud-aws-starter-s3'
  implementation platform('software.amazon.awssdk:bom:2.26.12')
  implementation 'software.amazon.awssdk:s3'
  implementation 'software.amazon.awssdk:sts'
}

2. Configure AWS properties (application.yml):

yaml
spring:
  cloud:
    aws:
      s3:
        enabled: true
        endpoint: "${S3_ENDPOINT_URL}"
        path-style-access-enabled: true
      region:
        static: "${AWS_REGION}"
  <appName>:
    bucketName: "${S3_BUCKET_NAME}"

3. Access Bucket Name via ConfigurationProperties:

java
import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties("spring.<appName>")
public record AppConfiguration(String bucketName) {}

4. IRSA Credentials Setup:

Spring Boot will automatically detect and use IRSA credentials via StsWebIdentityTokenFileCredentialsProvider. No additional configuration is required.

Javascript ​

The AWS SDK for JS uses the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN by default.

1. Install Dependencies:

bash
npm install @aws-sdk/client-s3 dotenv

2. Main file (app.js):

javascript
const { S3Client, ListObjectsV2Command } = require("@aws-sdk/client-s3");

const bucketName = process.env.S3_BUCKET_NAME;
const awsRegion = process.env.AWS_REGION;

const s3Client = new S3Client({ region: awsRegion });

async function listBucketObjects() {
  try {
    const command = new ListObjectsV2Command({ Bucket: bucketName });
    const data = await s3Client.send(command);
    if (!data.Contents || data.Contents.length === 0) {
      console.log(\`Bucket "\${bucketName}" is empty.\`);
      return;
    }
    console.log(\`Objects in "\${bucketName}":\`);
    data.Contents.forEach((item) => {
      console.log(\`- \${item.Key}\`);
    });
  } catch (error) {
    console.error("Error listing bucket objects:", error);
  }
}

listBucketObjects();

Python ​

We recommend using boto3 for AWS integration in Python.

1. Install boto3:

bash
pip install boto3

2. Python script:

python
import boto3
import os

def handle_s3():
    bucket_name = os.getenv("S3_BUCKET_NAME")
    if not bucket_name:
        return "S3_BUCKET_NAME environment variable is not configured."
    s3 = boto3.resource('s3')
    try:
        objects = list(s3.Bucket(bucket_name).objects.all())
        print(f"Objects in bucket '{bucket_name}':")
        for obj in objects:
            print(f"- {obj.key}")
        return f"{bucket_name} contains {len(objects)} objects."
    except Exception as e:
        return f"Error accessing bucket: {e}"

if __name__ == "__main__":
    print(handle_s3())

Go ​

We recommend using the official AWS SDK for Go (aws-sdk-go-v2).

1. Install AWS SDK for Go v2:

bash
go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/service/s3

2. Sample Go script:

go
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    bucketName := os.Getenv("S3_BUCKET_NAME")
    if bucketName == "" {
        log.Fatal("Environment variable S3_BUCKET_NAME is not set.")
    }

    cfg, err := config.LoadDefaultConfig(context.TODO())
    if err != nil {
        log.Fatalf("Failed to load AWS configuration: %v", err)
    }

    s3Client := s3.NewFromConfig(cfg)

    resp, err := s3Client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
        Bucket: &bucketName,
    })
    if err != nil {
        log.Fatalf("Failed to list objects: %v", err)
    }

    fmt.Printf("Objects in bucket '%s':
", bucketName)
    for _, item := range resp.Contents {
        fmt.Printf("- %s
", *item.Key)
    }
}

Points of Contact ​

If you have any needs or questions, use the following help channels to leave a message, and someone from the MDO team will respond.

Powered By Platform One