Find Jobs
Hire Freelancers

Insertion of 10+ billion records into an AWS DB (RDS)

$30-250 USD

In Progress
Posted over 9 years ago

$30-250 USD

Paid on delivery
15 Billion records have to be inserted into a DB from Text Files. (Each record about 100byte) There are about 250 zip-files (2-3GB compressed and 8-15GB uncompressed). The task of inserting the data in the DB should be done on AWS (Amazon Web Service). The files will be available on S3. The processing should happen on EC2 with an instance that is capable to deal with the task. The project is to write some software that process the files automatically, i.e. from S3 to uncrompression, into the DB. The task should also be done in parallel e.g. on 10 or more EC2 instances in parallel, so that it is done in a few (4 or 5) days. As DB, RDS should be used .. A MS-SQL DB model with trigger for a View is already available. However, I would be open for suggestions on other ways to import the data to the DB (e.g. LINQ or [login to view URL] etc.). The goal is also to have a rough cost estimation on how much it will cost to do this task with respect to AWS's EC2 and RDS storage costs. More details will be provided to qualified candidates (someone who has had experience with extremely large databases and GB-sized files before)
Project ID: 6900390

About the project

11 proposals
Remote project
Active 9 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

About the client

Flag of UNITED STATES
Las Vegas, United States
5.0
75
Payment method verified
Member since Nov 8, 2007

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.