Download S3 Objects With Python and Boto 3

In this post we show examples of how to download files and images from an aws S3 bucket using Python and Boto 3 library.

Boto is an AWS SDK for Python. It provides easy to use functions that can interact with AWS services such as EC2 and S3 buckets.

Dowload S3 Objects With Python and Boto 3

In the following example, we download one file from a specified S3 bucket.

First we have to create an S3 client using boto3.client(s3).

import boto3

BUCKET_NAME = 'my_s3_bucket'
BUCKET_FILE_NAME = 'my_file.json'
LOCAL_FILE_NAME = 'downloaded.json'

def download_s3_file():
    s3 = boto3.client('s3')
    s3.download_file(BUCKET_NAME, BUCKET_FILE_NAME, LOCAL_FILE_NAME)

The download_file method takes three parameters:

The first parameter is the bucket name in S3. The second is the file (name and extension) we want to download and the third parameter is the name of the file we want to save as.

Download All S3 Objects in a Specified Bucket

In the following example, we download all objects in a specified S3 bucket.

The code snippet assumes the files are directly in the root of the bucket and not in a sub-folder.

import boto3

def download_all_files():
    #initiate s3 resource
    s3 = boto3.resource('s3')
    # select bucket
    my_bucket = s3.Bucket('bucket_name')
    # download file into current directory
    for s3_object in my_bucket.objects.all():
        filename = s3_object.key
        my_bucket.download_file(s3_object.key, filename)

Download All Objects in A Sub-Folder S3 Bucket

The following code shows how to download files that are in a sub-folder in an S3 bucket.

Suppose the files are in the following bucket and location:

BUCKET_NAME = 'images'

PATH = pets/cats/

import boto3
import os

def download_all_objects_in_folder():
    s3_resource = boto3.resource('s3')
    my_bucket = s3_resource.Bucket('images')
    objects = my_bucket.objects.filter(Prefix='pets/cats/')
    for obj in objects:
        path, filename = os.path.split(obj.key)
        my_bucket.download_file(obj.key, filename)

References

Boto 3 Documentation