Uploading files to S3 directly from the browser is a great way to increase performance by removing the need to process and then re-upload files from your own server.

Note: In this post we'll be examining the use of pre-signed POST requests. See pre-signed URLs as an alternative (less powerful) method.

Overview

A traditional approach to file uploading would typically involve the client making a POST request to an endpoint on your API server. Your API server would then be responsible for processing the handling of the transfer of the file itself, and any subsequent processing you might want to perform (image resizing/thumbnail generation etc).

There's certainly nothing wrong with this approach, however these days many people opt to use third party storage solutions, like S3, to avoid having to implement a scalable storage solution of their own. Great, right? This now means your API server has to transfer the file(s) to S3 and incur additional resource costs.

What if you could avoid these resource costs, think RAM, CPU, bandwidth in and out, etc by having your client upload directly to S3? Enter pre-signed POST requests!

Pre-signed POST requests allow the client to upload directly to S3, bypassing your server entirely. Your API server is only tasked with generating the pre-signed POST request and providing it to the client, no file handling at all! As you can imagine, this frees up significant resources which can then be utilized for handling additional API requests.

You might be wondering how we can perform any additional processes, like thumbnail generation for example, if we never actually get our hands on the file. This is where lambda functions come in handy!

Although outside the scope of this particular post, I will look to address post upload processing with lambda functions in a future post!

Process summary

The client makes a request to an endpoint which responds with a URL and pre-signed post data.
The client then uses the provided URL and pre-signed POST data to form a request containing the file to be uploaded directly to S3.

Now that we've gone over the benefits of utilizing pre-signed POST requests over the traditional approach, let's dive in further and take a look at how to implement them!

S3 bucket CORS

Assuming you already have an S3 bucket you'd like to upload your files to, the next step is to modify the bucket's CORS configuration to allow POST requests.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

Once that's done it's time to create our pre-signed POST request!

Creating the pre-signed POST request

An endpoint on your server that's accessible to the client would contain code similar to the following:

import aws from 'aws-sdk';
import { nanoid } from 'nanoid';

const s3 = new aws.S3();
const S3_BUCKET = 'launchpad';

const fileName = `uploads/${nanoid()}`; // create a unique file name
const fileType = contentType; // the content-type of the file
const s3Params = {
    Bucket: S3_BUCKET,
    Fields: {
        key: fileName
    },
    Conditions: [
        ['content-length-range', 0, 100000000],
        ['starts-with', '$Content-Type', 'image/'],
        ['eq', '$x-amz-meta-user-id', userId],
    ],
    ContentType: fileType
};

const data = s3.createPresignedPost(s3Params);

Specific conditions may be specified to add validation to the request. In this case, we verify that the file size is less than 100000000 bytes and that the content-type of the file begins with 'image/'.

It's also worth noting that additional custom meta information can be specified. Here we specify the user id of the person uploading the file which would be useful in post upload processing when we need to associate the file with the uploader.

The result of createPresignedPost would look something like the following:

{
  "url": "https://s3.amazonaws.com/launchpad",
  "fields": {
    "key": "uploads/ENirEL-xzcazcDjtD-EI0",
    "bucket": "launchpad",
    "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
    "X-Amz-Credential": "AKDSIAIO3EBJDSMZDNVBYYSD6MCA/21254530/us-east-1/s3/aws4_request",
    "X-Amz-Date": "20200530T020436Z",
    "Policy": "eyJleHBpcmFKJgKJHlowNDozNloiLCJjb25kaXRpb25zIjpbWyJjb250ZW50LWxlbmdfsdgrfgDBdLFsic3RhcnRzLXdpdGgiLCIkQ29udGVusdfsdfsdfaW1hZ2UvIl0sWyJlcfsdf4LWFtei1hY2wiLCJwdWJsaWMtcmVhZCJdLFsiZXEiLCIkeC1hbXotbWV0YS11c2VyLWlkIiwiMSJdLFsiZXEiLCIkeC1hbXotbWV0YS1pbWFnZWFibGUtdHlwZSIsImJlZCJdLFsiZXEiLCIkeC1hbXotbWV0YS1pbWFnZWFibGUtaWQiLCIxIl0seyJrZXkiOiJ1cGxvYWRzL28vRU5pckVMLXh6Y2F6Y0RqdEQtRUkwIn0seyJidWNrZXQiOiJzd3Rlc3RidWNrZXQzIn0seyJYLUFtei1BbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJYLUFtei1DcmVkZW50aWFsIjoiQUtJQUlPM0VCSk1OVkJZWTZNQ0EvMjAyMDA1MzAvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsiWC1BbXotRGF0ZSI6IjIwMjAwNTMwVDAyMDQzNloifV19",
    "X-Amz-Signature": "fb0f381d8c9176f86g786gc30f5a25e236546723894dgf2cddb347"
  }
}

Uploading the file to S3

Once we've written the code to create a pre-signed POST request, it's time to create the request itself and upload the file to S3!

import ky from 'ky';

// the selected file to upload
const file = <FILE>;

// get the pre-signed POST request
const presignedPost = await ky.post('/presignedpost', { contentType: file.type }).json();

const formData = new FormData();

formData.append("x-amz-meta-user-id", userId);
formData.append("Content-Type", file.type);

Object.entries(presignedPost.fields).forEach(([k, v]) => {
    formData.append(k, v);
});

formData.append("file", file);

await ky.post(presignedPost.url, {
    body: formData,
});

We use FormData to build out the POST request, appending all the necessary fields that were specified in our code to create the pre-signed POST request. We then append all the fields provided by the pre-signed POST request response and lastly, we append the file itself.

Please note: the file must be the last field appended to the FormData.

Now that the form data has been prepared with all the necessary fields, all that's left to do is make the request! In this example we use the popular HTTP request library ky but any will do!

If all went to plan S3 should reply with a 204 and if not, it should reply with one of the following error codes instead.

Security Concerns

Pre-signed POST requests are secure by nature. The pre-signed POST request itself contains no sensitive information so there is no harm in allowing the client access to it. That being said, it's important to note that anyone with access to a valid pre-signed POST request can use it to upload directly to your S3 bucket!

Be sure to only issue your pre-signed POST requests to trusted parties! It's important to protect your endpoint responsible for the request generation like you would any other secure endpoint on your API.

That's it!

I hope this post was helpful in getting a basic understanding of the process involved with uploading a file directly to S3 from the browser.

Utilizing this approach along with a lambda function to replace any file processing that would normally have been performed on your server makes for a really powerful combination.

What's next?

We'll look to create a lambda function to process the file once it's been uploaded to S3 and retrieve the meta data we passed along with it. Stay tuned!

STAY UP TO DATE