S3 DeleteObject – How to Delete Multiple S3 Objects with NodeJS

Do you have a lot of random files in an S3 bucket that need to be deleted? We had about 12,000 random PDF files that needed to be deleted. They were mixed in with legitimate files and folders, so we needed a way to target specific files and bulk delete them easily.

Sometimes, you may need to delete multiple objects from an S3 bucket efficiently. Here’s how we did it using S3 DeleteObjects and NodeJS.

Assumptions

1. You have some experience with Node.js and can set up your development environment.
2. You know what “S3” and “IAM” are without looking them up.
3. You understand that mass deleting files can be crazy stressful, yet satisfying.
4. You understand that you are responsible for what you do.

Do you fit these criteria? Good, let’s keep going.

Your AWS Environment Requirements

Before you can start using the AWS SDK for JavaScript to delete S3 objects, you need to make sure you have some information regarding your AWS environment:

1. AWS Account: You’ll need to have access to the AWS account.
2. IAM User: You’ll need an IAM (Identity and Access Management) user with the necessary permissions for accessing and deleting objects in the S3 bucket. You need their Access Keys.
3. S3 Bucket: You’ll need to know which S3 bucket you want to delete objects from. Make sure the IAM user has the required permissions for this bucket.

Installing and Configuring the AWS SDK for JavaScript

To use the AWS SDK for JavaScript, you need to install it and configure your AWS credentials. If you haven’t already installed Node.js and npm (Node Package Manager), make sure to do that first.

Installation

You can install the AWS SDK for JavaScript using npm:

npm install @aws-sdk/client-s3

Also, because we’re dealing with credentials, you’ll want to install dotenv-safe.

npm install @dotenv-safe

Alrighty. Let’s do this.

Configuration

There are three main files in this lesson. They could have been combined into a single file, but separating them simplifies things a bit.

After installing the SDK, you must configure it with your AWS credentials. The AWS SDK for JavaScript will use these credentials to interact with your AWS services.

You can configure the SDK credentials using the dotenv-safe library. Create a file named “.env” and update the lines below with your credentials and bucket information:

# credentials
S3_REGION = "us-east-1"
ACCESS_KEY_ID = "AKIX8S45D49DJDH7JUNG"
SECRET_ACCESS_KEY = "f6+wDCaGIadoLdxlsE3d8DsnRv02c7jgo2"

# bucket info.  Leave "S3_FOLDER" value blank if you want to start at the top level
S3_BUCKET = "yourBucketName"
S3_FOLDER = ""

In this script:

• Line 1-4: Your IAM credentials with proper permissions
• Line 7: S3 bucket name
• Line 8: Folder (S3 calls it a “Prefix”) below the bucket listed on line 7. Add a value here only if you want to limit the script to search and delete from this folder (and below).

Listing Out the S3 Objects to be Deleted

Now that your environment is set up and the AWS SDK is configured, let’s explore how to first list out the S3 objects you want to list. Once we limit the list to what we want, we can go on to the next step and delete them.

Here’s how I went about listing the files I wanted to delete:

//this file lists the objects in an S3 bucket.  Max rows is 1000.

import 'dotenv/config';
import { S3Client, ListObjectsV2Command } from "@aws-sdk/client-s3";

let client = new S3Client({
    region: process.env.S3_REGION,
    credentials: {
        accessKeyId: process.env.ACCESS_KEY_ID,
        secretAccessKey: process.env.SECRET_ACCESS_KEY
    }
});

const command = new ListObjectsV2Command({
    Bucket: process.env.S3_BUCKET,
    Prefix: process.env.S3_FOLDER,
    MaxKeys: 1000,
});

(async () => {

    const response = await client.send(command);

    // if you want to limit this list, add code here to change the "response" variable to only include files that fit

        //EXAMPLE 1 - reduce list to items between two dates
            // Define the date range for filtering
            //const startDate = new Date('2022-1-1');
            //const endDate = new Date('2023-1-1');
            // Filter the "Contents" array
            //const filteredContents = response.Contents.filter(item => {
            //    const lastModifiedDate = new Date(item.LastModified);
            //    return lastModifiedDate >= startDate && lastModifiedDate <= endDate;
            //});
            // Update the "Contents" property in the response object
            //response.Contents = filteredContents;
        //EXAMPLE 1 END

        //EXAMPLE 2 - reduce file list to items with "150" in the file name
            const filteredContents = response.Contents.filter(item => item.Key.includes("150"));
            // Update the "Contents" property in the response object
            response.Contents = filteredContents;
        //EXAMPLE 2 END

    //at this point we have the results in a single variable("response") with LastModified, Size, and StorageClass attributes included
    //console.log(response); //uncomment to see the full list

    // We just want the filename so we limit this list to get just the "Key" values
    const keys = response.Contents.map(item => item.Key).map(item => ({ Key: item }));
    console.log(keys);

})();

In this script:

• Line 3-4: Import the libraries
• Line 6-12: Create the connection to AWS and authenticate using the credentials in your .env file
• Line 14-18: Create the command that you’ll send to AWS to retrieve all S3 objects
• Line 22: Send the request to AWS and put the values returned in the “response” variable
• Line 26-37: Example code if you want to reduce the file list based on dates
• Line 39-43: Example code if you want to reduce the file list based on a text value in the file name. Comment out if not needed
• Line 49: This line extracts the file name (“Key“) value from the JavaScript object returned and puts it in a format to copy for deletion.
• Line 50: Logs the list of Keys in the console.

Run this script alone to get the full list of objects to be deleted.

Deleting Multiple Objects With S3 DeleteObject

Now that you have the file list you want to delete, let’s explore how to delete multiple S3 objects efficiently. To delete multiple objects, you can use the deleteObjects method provided by the AWS SDK for JavaScript.

Here’s a sample script to delete multiple objects from an S3 bucket:

import 'dotenv/config';
import { DeleteObjectsCommand, S3Client } from "@aws-sdk/client-s3";

let client = new S3Client({
    region: process.env.S3_REGION,
    credentials: {
        accessKeyId: process.env.ACCESS_KEY_ID,
        secretAccessKey: process.env.SECRET_ACCESS_KEY
    }
});

const command = new DeleteObjectsCommand({
    Bucket: process.env.S3_BUCKET,
    Delete: {
        Objects: [
            {Key: "my-random-image.jpg"}, // if in a directory: "directory_name/file.ext"
            {Key: "wp-content/uploads/2022/08/19185558/some-other-random-image.jpg"},
            {Key: "wp-content/uploads/2023/05/19182148/another-random-image.jpg"}
        ],
    }
});

(async () => {

    const response = await client.send(command);
    console.log(response);

})();

I’ve purposely kept this script’s object list manual since I didn’t want anyone to inadvertently delete more than they wanted. It’s easy enough to automate yourself, or you can ask me how to do it.

You’ll want to replace your list with the list I have on lines 16-18. Just specify the directory and filename.

In this script:

• Line 1-2: Import the libraries
• Line 4-10: Create the connection to AWS and authenticate using the credentials in your .env file
• Line 12-21: Create the command that you’ll send to AWS to define which S3 objects to delete
• Line 16-18: Replace these lines with the full list of the files you want deleted. You would have this list from the response that “list-all-s3-files.js” returned. Make sure it’s formatted correctly.
• Line 23-28: Send the delete request to AWS and display the response in the console.

Run this script alone and all items you listed should get deleted.

Summary

This is a quick and easy way to efficiently delete multiple S3 objects using the AWS SDK for JavaScript in your Node.js application. This can be particularly useful for automating cleanup tasks and managing your S3 resources more effectively.

Also, if you want information on how to set up your own web server in AWS, we’ve done that plenty of times too!

Good luck!