In Progress

Bash Scripting - Copy Files from Amazon S3 and Server

I need two bash scripts written:

1. We store many large video files on Amazon S3. I want a script written to copy a batch of files (could be 100+ files) from S3 to a folder on our linux server (which happens to be an Amazon EC2 server). I have used S3CMD before successfully but open to alternate methods. The file list would be on a separate txt file (could be csv if needed). Variables that should be able to be changed easily are source S3 bucket, and destination folder. Not sure how the concurrent number of files copying at once would work or what would be most efficient. Freelancer would need to advise on this issue. Meaning do we limit the number of concurrent files at once or ? some of the files could be 15 GB each so the copy method must include multi-part transfer (s3cmd offers this). Also need a report generated at the end of the copy that would report on successful transfers and/or files that were not located.

2. Second script is similar to #1 above but would copy from a server folder on the server that would be running the script. These would be mostly image files so not as large as the files on S3. Same variables as above (source and destination) and a report in the end that lists successes and files that cannot be found.

Not sure of the price of the budget required on this.

Skills: PHP, Script Install, Shell Script, Software Architecture

See more: what the meaning of freelancer, the meaning of freelancer, the freelancer files, scripting freelancer, how could be freelancer, freelancer what happens, freelancer software price, freelancer server software, freelancer list to amazon, freelancer gb, freelancer files, freelancer bash scripting, freelancer bash, freelancer 2 amazon, efficient freelancer, ec2 bucket, bash copy, amazon server freelancer, amazon s3 architecture, alternate freelancer, what is amazon ec2, bash file, bash 0, amazon video linux, amazon ec2 price

About the Employer:
( 25 reviews ) Brentwood, United States

Project ID: #6171090

Awarded to:


Hi Randall, i like to write some Perl or bash script to automate the data transfer for you. .........

$166 USD in 5 days
(15 Reviews)

17 freelancers are bidding on average $132 for this job


Dear Customer, this is Yaseen, and i am glad to work for you project. We specialise in Server Management, Enterprise Email Systems, Email Marketing and Asterisk VOIP Support. Please see my past projects https://www More

$210 USD in 3 days
(319 Reviews)

Hello, I have over 9 years experience as an Linux admin and strong experience with AWS services. I have created many custom backup scripts to copy files from/to S3 bucket and between servers. If you are interested we More

$200 USD in 3 days
(354 Reviews)

Hi I work towards providing reliable, relevant and robust IT solutions at most competitive prices to my customers. I ensure 100% customer satisfaction so lets start Thanks

$190 USD in 7 days
(100 Reviews)

Hi, I (Myself a senior PHP, ASP.NET programmer using C# and VB) have a team having experience of more than 10 years in developing web and mobile applications. I have a team of expert programmers who write robust and bu More

$263 USD in 10 days
(14 Reviews)

Hello, I have worked with AWS EC2 in Bash using their command line interface. I can check their S3 command line reference and do your project. I can start working today. Thanks.

$70 USD in 1 day
(55 Reviews)

I have worked with Amazon EC2 and S3, i have also worked with s3cmd in python. This problem sounds challenging to me and I would like to work on this I have 5 years of professional experience.

$200 USD in 3 days
(11 Reviews)

Hi I know better much more methods with file transfering and can write this fcripts for you SSH root access is requied *****************************************************

$100 USD in 3 days
(17 Reviews)

Hello sir, I have an experience of 3 year in Linux filed.I'm an expert in Linux server.I will setup script for file transfer My resume - [url removed, login to view] My website - [url removed, login to view] Y More

$70 USD in 3 days
(5 Reviews)

Hello! I have made the similar script earlier, besides, I have extensive experience in shell scripts development. - No need csv. - All variables (as always) will be in head of the script. Furthermore, you will be ab More

$50 USD in 2 days
(2 Reviews)

I'm having good exposure of shell script and cloud computing. We can have discussion if you are interested.

$155 USD in 3 days
(1 Review)

Hello, I am the right person you need for this task ! I am a Sys Admin and bash scripting for tasks like this one is really my daily work. You are right to be unsure of the Project Budget you proposed : $250 is too mu More

$55 USD in 3 days
(0 Reviews)

Dear Sir, We are the globalized software services organization specialized in providing Mobile, ECommerce and Social media frameworks using cutting edge and emerging technology. Leveraging best-in-class people, proc More

$144 USD in 3 days
(1 Review)

Hello, I would like to do that project for you, please check my porftolio for my bash scripting samples. Hope to talk with you soon, best regards!

$45 USD in 1 day
(0 Reviews)

You can choose to pay what you think is best. The following is quick script that copies files given in a file name to a destination directory. ./[url removed, login to view] /tmp/input /tmp/tdir/ #!/bin/bash Usage() { echo $0 More

$155 USD in 3 days
(0 Reviews)

I do Linux admin with Bash all the time. This task would not be difficult for me to do. You probably want to use FTP or HTTP PUT method, unless you need encryption, which could use SFTP or HTTPS. You also want to ve More

$30 USD in 2 days
(0 Reviews)

Hey, I have good idea in creating scripts. If you like me to proceed, please give me complete details of servers paths etc. Do send me complete documentation with clear details. Otherwise, project cannot be done More

$137 USD in 5 days
(0 Reviews)