Writing to AWS S3 from Fly.io (in Go)
I'm currently in the process of moving my site's media (under media.jvt.me) from Netlify to AWS S3.
The underlying infrastructure for my site that writes this media is hosted on Fly.io, so needs a way to write to S3.
I really wanted to avoid long-lived AWS credentials, and fortunately Fly supports OpenID Connect which integrates with AWS' OpenID Connect Federation to allow you to use short-lived OpenID Connect tokens to authenticate as a given IAM role, and then perform any necessary AWS actions.
Although Fly has an example with curl, I wanted to blogument the "how to" with Go, as it wasn't super straightforward to do so.
I've written this up in a sample repo on GitLab.com.
Go code
The core pieces are:
// via https://gist.github.com/teknoraver/5ffacb8757330715bcbcc90e6d46ac74
var unixClient = http.Client{
Transport: &http.Transport{
DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
return net.Dial("unix", "/.fly/api")
},
},
}
func getOIDCToken(unixClient http.Client) (string, error) {
s3Body := bytes.NewReader([]byte(`{"aud":"sts.amazonaws.com"}`)) // or could be `{"aud":"https://sts.amazonaws.com"}` as long as it matches your Trust Policy
// get the OIDC token
resp, err := unixClient.Post("http://localhost/v1/tokens/oidc", "application/json", s3Body)
if err != nil {
slog.Error("Failed to call Fly OIDC tokens endpoint", "err", err)
return "", err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
slog.Error("Failed to read body", "err", err)
return "", err
}
return string(body), nil
}
func newAWSClient(oidcToken string) (*s3.Client, error) {
cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("eu-west-1"))
if err != nil {
return nil, err
}
stsClient := sts.NewFromConfig(cfg)
assumedRoleInput := &sts.AssumeRoleWithWebIdentityInput{
RoleArn: aws.String(os.Getenv("AWS_ASSUME_ROLE_ARN")),
RoleSessionName: aws.String("..."),
WebIdentityToken: aws.String(oidcToken),
}
resp, err := stsClient.AssumeRoleWithWebIdentity(context.TODO(), assumedRoleInput)
if err != nil {
return nil, err
}
sessionCreds := aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(
*resp.Credentials.AccessKeyId,
*resp.Credentials.SecretAccessKey,
*resp.Credentials.SessionToken,
))
s3Client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.Credentials = sessionCreds
})
return s3Client, nil
}
In this case we need to:
- Retrieve the OpenID Connect ID token via a Unix pipe (thanks to this Gist)
- Perform an
AssumeRoleWithWebIdentitywith that given token - Use the temporary credentials to seed the AWS client(s) we then use to interact i.e. with S3
You can then use this like so:
http.HandleFunc("POST /do-something-with-s3", func(w http.ResponseWriter, r *http.Request) {
userBody := r.URL.Query().Get("body")
if userBody == "" {
return
}
token, err := getOIDCToken(unixClient)
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
s3Client, err := newAWSClient(token)
if err != nil {
slog.Error("Failed to create S3 client", "err", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
inp := s3.PutObjectInput{
Bucket: aws.String(os.Getenv("AWS_S3_BUCKET")),
Key: aws.String("fly-s3/" + randSeq(10)),
Body: bytes.NewReader([]byte(userBody)),
}
_, err = s3Client.PutObject(r.Context(), &inp)
if err != nil {
slog.Error("Failed to PUT", "err", err)
w.WriteHeader(http.StatusInternalServerError)
return
}
slog.Info("Created path", "key", inp.Key)
w.WriteHeader(http.StatusAccepted)
})
AWS IAM/S3 polices
Once I'd created the OpenID Connect provider, I then I created a new IAM role example-fly-s3-access (remembering to make sure it had Trusted entity type: Web identity), with the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::5..........5:oidc-provider/oidc.fly.io/jamie-tanna"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.fly.io/jamie-tanna:aud": "sts.amazonaws.com"
},
"StringLike": {
"oidc.fly.io/jamie-tanna:sub": "jamie-tanna:fly-s3:*"
}
}
}
]
}
I found that trying to be a little more restrictive in the Trust Policy resulted in i.e.
ERROR Failed to create S3 client err="operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 403, RequestID: cd047481-ae53-40a8-b353-ab1d74ee582d, api error AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity"
It's worthwhile keeping to the standard claims like aud and sub, but would be nice if AWS flagged that themselves.
Then, for the S3 bucket policy, on one of my old test buckets me.jvt.nottsdev.workshop:
{
"Version": "2012-10-17",
"Id": "...",
"Statement": [
{
"Sid": "ListBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::5..........5:role/example-fly-s3-access"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::me.jvt.nottsdev.workshop"
},
{
"Sid": "ListBucketObjects",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::5..........5:role/example-fly-s3-access"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::me.jvt.nottsdev.workshop/*"
},
{
"Sid": "ModifyObjectsInBucket",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::5..........5:role/example-fly-s3-access"
},
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::me.jvt.nottsdev.workshop/*"
}
]
}
