In Progress

ETL and Cron Job for Custom Data Feed

I have a custom data feed (CDF) that gets posted to a secure FTP site every hour by my technology partner. Each file has approximately 2 million rows and 30 headers.

1)Read and understand the CDF file and its columns. CDF documentation will be provided.

2)Create a MySQL database schema

3)Create a table RM_CDF_EVENT_LOG. This table should have all the columns defined in the CDF. We need to add couple of extra columns called FILENAME, MODIFIED_ON (DEFAULT SYSDATE) to the table

4)Partition the table by DATETIME column in CDF and FILENAME

5)Create a script that would do the following

a) Download the file from a FTP server

b) Run GPG on the file to decrypt it (can stubbed for now)

c) Run unzip/untar on the file to extract the file

d) Create a new partition in the tbale for the DATETIME and FILENAME

e) Break apart column data (separated by spaces) into separate rows

f) Insert rows from the file into the table

g) Set this script to be executed as a cron job to be executed every 30 minutes

6)Script can be written in any language (Java, PHP, C, etc)

7)If possible we need to use Apache Hadoop to break apart the rows that are delimited by spaces inside a column. (This is not a requirement but is considered a bonus if you can do it)

Skills: Apache, C# Programming, Java, MySQL, PHP

See more: etl file feed, cron job etl, cron etl, etl cron job, data feed etl, cron database schema, cron mysql insert, unzip ftp, technology job, ftp unzip, every job, etl job, language job, cron data feed, server job, job requirement, hadoop job, hour job, hadoop, gpg, ETL, d custom, cron script, php data table, java unzip file

About the Employer:
( 8 reviews ) San Francisco, United States

Project ID: #595273

Awarded to:

gord

Please check PMB.

$750 USD in 5 days
(4 Reviews)
5.8

17 freelancers are bidding on average $475 for this job

omsoftware

Hello We Understand the project and we can create a Data Feed Reader and Update the Database as cron Job written in PHP or ASP.NET Please check PMB fore more details Thanks Raj

$450 USD in 5 days
(82 Reviews)
8.2
A2Design

We're interested in this project. Take a look on our portfolio at: http://www.a2design.biz/portfolio_

$750 USD in 16 days
(30 Reviews)
7.7
sunztech

Please see PMB.

$750 USD in 10 days
(31 Reviews)
7.3
AndrwProjects

Hi, interested to take the job.

$399 USD in 7 days
(175 Reviews)
6.3
interpb

Hello, I am interested in this project Thanks

$280 USD in 8 days
(53 Reviews)
6.1
MAST3R

You have PM

$450 USD in 10 days
(10 Reviews)
5.5
Ivan83

Hi, Please check PM. Thanks.

$450 USD in 10 days
(26 Reviews)
5.3
savtargm

Please check PM. Thanks.

$900 USD in 20 days
(1 Review)
4.1
amsak

Pls see PM

$250 USD in 14 days
(7 Reviews)
3.6
waqasshami

Hi ropak I am ETL Developer. Please see PM for more details. Thanks

$400 USD in 7 days
(3 Reviews)
2.1
suryavikas

Hi, I have vast experience in ETL, and have done lot of similar projects in my company. I have the knowledge and the skill set to finish the project on time and without bugs. Idea is to use an Open source ETL More

$500 USD in 10 days
(1 Review)
1.6
ephraiminjamuri

I am a c#, SQL and scripting programming. I wrote this kind of applications. only thing different is the number of rows.

$500 USD in 14 days
(0 Reviews)
0.0
frosters

Can do your task using perl and some unix tools (cron, shell). It looks not so difficult :)

$250 USD in 3 days
(0 Reviews)
0.0
DISoln

Please check your inbox for my Proposal

$500 USD in 14 days
(0 Reviews)
2.4
sailu145cw

Hi! I have gone through your requirement and i am glad that i can accomplish this task, i would be more interested to speak to you on IM. Pls give us an opportunity to work with you.

$250 USD in 2 days
(0 Reviews)
0.0
toronto0013

Hi We have done a similar job. See pm Thank you Project manger More

$250 USD in 15 days
(0 Reviews)
0.0