Uploading files is an integral part of most of the modern web applications.As most of the modern applications are backed by APIs, so we need to understandand get familiar with file uploads through APIs. Now the question arises whereshould we store the file? The easy answer is the local file system or withinour working directory. Yes, that is true but for a production-level applicationstoring files in the filesystem is not a good idea, and even if we do thatthere are overheads for managing the space, access rights, and above allsecurity. So there are couple of alternatives –

a) CDN – CDN stands for Content Delivery Network. It not only stores thedata but gives you efficient ways to access the data. But generally, they arecostly and there are few configuration mechanisms also involved.

b) Database – Store files in the Database. If you store it in a Database youwill have a transactional level control, whether the data is getting written ornot but the most obvious concern is the size and efficiency of the delivery ofthe image.c) Cloud Storage – Nowadays with the rising popularity of Cloud technologiesthe best place to store files is in cloud storage. You don’t need to worryabout space and security. You have full control over the access mechanism andalso the cost is relatively low and several cloud providers provide freestorage up to a certain threshold. The most popular of these Cloud Storages areAWS( Amazon Web Service) S3 (Simple Storage Service) or SimplyS3.

In this article, we will learn about how to upload a file through REST APIbuilt using Node.js and Typescript to upload files in S3

Pre-requisites –

  1. You must have an Account in AWS
  2. You must have Node.js installed in your machine most preferably the LTS version

So if you have AWS Account already created and Node.js installed on your machine lets get started.

Step-1 Creating bucket in S3 –

All your files that you are going to be uploaded to S3 bucket. So first task is to create the bucket. In AWS console search for S3 and then click on “Create Bucket” to create your first S3 bucket . Keep the following configuration while creating the S3 Bucket as shown in the screen shot below.

If the bucket is successfully created you should be able to see in the list of buckets.

The next step is to create an IAM user and credentials so that we can programmatically access S3 bucket. So let’s do that in the next step

Step-2 Creating IAM User and Credentials for Programmatically Access S3 bucket

IAM Stands for Identity and Access Management. Lets’ create an User and then attach policy for S3 Access and then we will generate the credential for that user. It is always advisable not to use root user for any of your programming or CLI access. Checkout the screen shot below which will take you through the user creation process.

Once you successfully complete the above steps you should see the created user in the list. Next step is to generate the credentials as shown in below screen shots. Click on the user name in the list and follow the screen shots

As you are developing now so select Local Code option as shown in below screenshot

You should be able see the access key and secret access key just like the above screenshot. Copy both of them and save it somewhere. We will need these two keys while we code. Next step is to create an application for building the REST API for uploading files to S3

Step-3 Creating project folder and install dependencies –

Create a project folder “s3-upload” and then go inside the folder and then create package.json file by issuing the following command

npm init -y

Once the package.json file is created install the dependencies

npm i express multer multer-s3 @aws-sdk/client-s3

Install dev dependencies

npm i -D @types/express @types/multer @types/multer-s3

Next step is to create a tsconfig.json file in the root of your project folder

Step-4 Create tsconfig.json file

Create a file tsconfig,json in the root of your project folder and put the following code inside it

{
  "compilerOptions": {
    "target": "es2016",
    "module": "commonjs",
    "rootDir": "./src",
    "outDir": "./dist",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
    },
    "include": ["src/**/*"],
    "exclude": ["**/*.spec.ts"]
}

Step-5 Folder Structure

A typical folder structure would look like the following screenshot

Step-6 Create .env file

Create a .env file and put it in the root of your project folder and put different config options inside the file.

#General Config 
NODE_ENV=development
PORT=4000

#AWS Config
ACCESS_KEY=<Your AWS Access Key>
ACCESS_KEY_SECRET=<your AWS Secret Access Key>
REGION=<Your Region>
BUCKET_NAME=<Your bucket name> 

Step-7 Creating the config file

Create a file config.ts under src/config and put the following code inside it

import "dotenv/config";

const config = {
  general: {
    nodeEnv: process.env.NODE_ENV,
    port: process.env.PORT,
  },
  aws: {
    accessKey: process.env.ACCESS_KEY,
    secretAccessKey: process.env.ACCESS_KEY_SECRET,
    region: process.env.REGION,
    bucket: process.env.BUCKET_NAME
  },
};

export default config;

Step-8 Create app.ts file

Create app.ts file under src folder and put the following code inside it

import express, { Application } from "express";
import config from "./config/config";

const app: Application = express();

const port = config.general.port;

app.listen(port, () => console.log(`Server started at port ${port}`));

Step-9 Create connection to AWS S3

Create a file aws-connect.ts under src/config folder and put the following code inside it. Remember we are using AWS JavaScript SDK Version 3 for this article

import { S3Client, S3ClientConfig } from "@aws-sdk/client-s3";
import config from "../config/config";
export function connectS3(): S3Client {
  try {
    const configuration: S3ClientConfig = {
      region: config.aws.region,
      credentials: {
        accessKeyId: config.aws.accessKey,
        secretAccessKey: config.aws.secretAccessKey,
      },
    };
    const s3Client: S3Client = new S3Client(configuration);
    return s3Client;
  } catch (err: any) {
    throw new Error("AWS Authentication Error");
  }
}




Next step is to construct the middleware for uploading files

Step-10 Building Generic Middleware for uploading files

Create a file upload.ts under src/middleware

import { Request, Response, NextFunction } from "express";
import multer, { StorageEngine } from "multer";
import multerS3 from "multer-s3";
import { connectS3 } from "../config/aws-connect";
import config from "../config/config";

const fileFilter = (req: Request, file: any, cb: any) => {
  if (
    file.mimetype.split("/")[0] === "image" ||
    file.mimetype.split("/")[1] == "pdf" ||
    file.mimetype.split("/")[1] == "plain"
  ) {
    cb(null, true);
  } else {
    cb(null, false);
    cb(new Error("File type not allowed"));
  }
};

export function uploadSingleFile(
  req: Request,
  res: Response,
  next: NextFunction
) {
  const storage: StorageEngine = multerS3({
    s3: connectS3(),
    bucket: config.aws.bucket,
    metadata: function (req, file, cb) {
      cb(null, { fieldName: file.fieldname });
    },
    key: function (req, file, cb) {
      cb(null, `${Date.now()}-${file.originalname}`);
    },
  });

  const fieldname: string = req.headers["fieldname"].toString();
  const fileUpload = multer({ storage, fileFilter }).single(fieldname);

  fileUpload(req, res, (err: any) => {
    if (!req.file) {
      return res.status(400).json({
        message: `File missing`,
      });
    }
    if (err) {
      if (err instanceof multer.MulterError) {
        return res.status(400).json({
          message: `Something went wrong with Multer file upload -${err.message}`,
        });
      }
      return res.status(400).json({
        message: `Something went wrong with file upload -${err.message}`,
      });
    }
    next();
  });
}

export function uploadMultipleFile(
  req: Request,
  res: Response,
  next: NextFunction
) {
  const storage: StorageEngine = multerS3({
    s3: connectS3(),
    bucket: config.aws.bucket,
    metadata: function (req, file, cb) {
      cb(null, { fieldName: file.fieldname });
    },
    key: function (req, file, cb) {
      cb(null, `${Date.now()}-${file.originalname}`);
    },
  });

  const fieldname: string = req.headers["fieldname"].toString();
  const fileUpload = multer({ storage }).array(fieldname);

  fileUpload(req, res, (err: any) => {
    if (err) {
      return res.status(500).json({
        message: "Something went wrong with file upload - ${err.message}",
      });
    }
    next();
  });
}

Here in the above file we have created two functions uploadSingleFile and uploadMultipleFile . They are respectively used for single and multiple file uploads. The fileFilter function checks the MimeType of the file and then allow or reject based of the type of file being uplaoded. You can add as many MimeType as you want based on the nature of your application.

For more details about multer file upload you can check the following article

Now let’s construct the controller for uploading the file(s)

Step-11 Creating controller

Create a file upload.controller.ts under src/controllers and put the following code inside it

import { Request, Response } from "express";

export async function uploadFiles(req: Request, res: Response) {
  try {
    return res.status(201).json({
      message: `File Uplaoded Successfully`,
    });
  } catch (err: any) {
    return res.status(err.statusCode).json({
      message: `Something went wrong while uploading files -${err.message}`,
    });
  }
}

Step-12 Finally creating routes

Create a file upload.ts under src/routes and the put the following code inside it

import express, { Router } from "express";
import { uploadFiles } from "../controllers/upload.controller";
import { uploadSingleFile, uploadMultipleFile } from "../middleware/upload";

const routes: Router = express.Router();

/** Route for Uploading Single File */
routes.post("/single", uploadSingleFile, uploadFiles);

/** Route for Uploading Multiple Files */
routes.post("/multi", uploadMultipleFile, uploadFiles);

export default routes;

And that’s it. Now we need to test our code using postman

Step-13 Calling the routes from our app.ts

Now we need to include our routes in app.ts file . So the modified file looks like as below

import express, { Application } from "express";
import config from "./config/config";
import uploadRoute from "./routes/upload";

const app: Application = express();

app.use(express.json());
app.use(express.urlencoded({ extended: true }));

app.use("/api/v1/upload", uploadRoute);

const port = config.general.port;

app.listen(port, () => console.log(`Server started at port ${port}`));

Step-14 create script is package.json

As a final step add the following line under scripts and you are good to test the API

"dev": "tsc&& node dist/app.js"

Test the API in POSTMAN

To test the API we will use “form-data” . In the “Headers” use key as fieldname and in the “Body” use the value of fieldname that you have used in the “Header” section. as show in below screenshot

Here in postman the Header key is fieldname and value is “image”. Use this value “image” as the key in your Body section as shown in below screenshot

Now press submit and you should see success message as shown below

Note that the value of fieldname and field for file upload is same because we wanted to make our file upload middleware a generic one so that you don’t need to use separate middleware’s for different file upload .

Happy Coding!!!

All You Need to Know About Closures in Javascript Previous post All you need to know about Closures in JavaScript