How to Upload Things to S3 Bucket

Introduction

Amazon Web Service, aka AWS, is a leading deject infrastructure provider for storing your servers, applications, databases, networking, domain controllers, and active directories in a widespread deject compages. AWS provides a Simple Due southtorage Due southervice (S3) for storing your objects or information with (119's) of information durability. AWS S3 is compliant with PCI-DSS, HIPAA/HITECH, FedRAMP, European union Data Protection Directive, and FISMA that helps satisfy regulatory requirements.

When you log in to the AWS portal, navigate to the S3 bucket, choose your required bucket, and download or upload the files. Doing it manually on the portal is quite a time-consuming task. Instead, you tin use the AWS Command Line Interface (CLI) that works best for bulk file operations with piece of cake-to-use scripts. You lot can schedule the execution of these scripts for an unattended object download/upload.

Configure AWS CLI

Download the AWS CLI and install AWS Control Line Interface V2 on Windows, macOS, or Linux operating systems.

You can follow the installation wizard for a quick setup.

Create an IAM user

To access the AWS S3 bucket using the command line interface, we demand to set upward an IAM user. In the AWS portal, navigate to Identity and Admission Management (IAM) and click Add User .

In the Add together User folio, enter the username and access type as Programmatic access.

Next, nosotros provide permissions to the IAM user using existing policies. For this article, we accept called [AmazonS3FullAccess] from the AWS managed policies.

Review your IAM user configuration and click Create user .

Once the AWS IAM user is created, information technology gives the Access Key ID and Hole-and-corner access cardinal to connect using the AWS CLI.

Notation : You should copy and salve these credentials. AWS does non allow you lot to recollect them at a afterwards phase.

Configure AWS Contour On Your Figurer

To work with AWS CLI on Amazon web service resources, launch the PowerShell and run the following control.

          >aws configure        

It requires the post-obit user inputs:

• IAM user Access Key ID
• AWS Secret Access key
• Default AWS region-name
• Default output format

Create S3 Bucket Using AWS CLI

To store the files or objects, we need an S3 bucket.  We can create information technology using both the AWS portal and AWS CLI.

The following CLI control creates a saucepan named [mys3bucket-testupload1] in the us-eastward-ane region. The query returns the bucket proper noun in the output, equally shown below.

          >aws s3api create-bucket --saucepan mys3bucket-testupload1 --region usa-east-one        

You lot tin verify the newly-created s3 bucket using the AWS panel. As shown below, the [mys3bucket-testupload1] is uploaded in the US Eastward (North. Virginia).

To listing the existing S3 bucket using AWS CLI, run the command – aws s3 ls

Uploading Objects in the S3 Bucket Using AWS CLI

Nosotros can upload a single file or multiple files together in the AWS S3 bucket using the AWS CLI control. Suppose we have a single file to upload. The file is stored locally in the C:\S3Files with the name script1.txt.

To upload the unmarried file, utilise the post-obit CLI script.

          >aws s3 cp C:\S3Files\Script1.txt s3://mys3bucket-testupload1/        

It uploads the file and returns the source-destination file paths in the output:

Note: The time to upload on the S3 bucket depends on the file size and the network bandwidth. For the demo purpose, I used a small file of a few KBs.

You can refresh the s3 bucket [mys3bucket-testupload1] and view the file stored in it.

Similarly, we can use the same CLI script with a slight modification. It uploads all files from the source to the destination S3 bucket. Here, nosotros employ the parameter –recursive for uploading multiple files together:

          >aws s3 cp c:\s3files s3://mys3bucket-testupload1/ --recursive        

Equally shown below, it uploads all files stored within the local directory c:\S3Files to the S3 bucket. You go the progress of each upload in the panel.

We tin can run across all uploaded files using recursive parameters in the S3 saucepan in the following figure:

If you do not want to go to the AWS portal to verify the uploaded list, run the CLI script, return all files, and upload timestamps.

          >aws s3 ls s3://mys3bucket-testupload1        

Suppose we want to upload only files with a specific extension into the separate folder of AWS S3. You can practice the object filtering using the CLI script every bit well. For this purpose, the script uses include and exclude keywords.

For case, the query below checks files in the source directory (c:\s3bucket), filters files with .sql extension, and uploads them into SQL/ binder of the S3 bucket. Hither, we specified the extension using the include keyword:

          >aws s3 cp C:\S3Files s3://mys3bucket-testupload1/  --recursive   --exclude * --include *.sql        

In the script output, you lot can verify that files with the .sql extensions merely were uploaded.

Similarly, the below script uploads files with the .csv extension into the S3 saucepan.

          >aws s3 cp C:\S3Files s3://mys3bucket-testupload1/  --recursive   --exclude * --include *.csv        

Upload New or Modified Files from Source Binder to S3 Bucket

Suppose you use an S3 bucket to move your database transaction log backups.

For this purpose, we utilise the sync keyword. Information technology recursively copies new, modified files from the source directory to the destination s3 bucket.

          >aws s3 sync C:\S3Files s3://mys3bucket-testupload1/  --recursive   --exclude * --include *.sql        

Equally shown below, it uploaded a file that was absent in the s3 bucket. Similarly, if you modify any existing file in the source binder, the CLI script will choice it and upload it to the S3 bucket.

Summary

The AWS CLI script can make your work easier for storing files in the S3 bucket. You can utilize it to upload or synchronize files betwixt local folders and the S3 saucepan. Information technology is a quick way to deploy and piece of work with objects in the AWS cloud.

Tags: AWS, aws cli, aws s3, cloud platform Last modified: September xvi, 2021

scottevelostrues.blogspot.com

Source: https://codingsight.com/upload-files-to-aws-s3-with-the-aws-cli/

Share this post