develop end to end data pipeline in hadoop

1. Develop/Propose various ETL pipelines for the combination of hadoop eco-system with/NOT spark using AWS cluster;

2. Implement the multi-purpose ELT pipeline (like kappa or lambda architecture) that ingress data from various sources (NOT limited to mySQL, Oracle, NoSQL, Flat files like PDF, Excel, CSV, .docx, Live/streaming) to hadoop/hdfs by using wide range of bigdata tools;

3. Ensure the data quality like format, fields, precision, No. of rows etc.., during data migration from data sources to HDFS;

4. Pipeline must able to handle huge data in GB's;

5. Troubleshoot by creating breakpoint in the ETL pipeline at various levels (at each hadoop eco-system tool) like resource level & code level towards memory management, performace tuning, optimization;

Skills: Amazon Web Services, Data Processing, Data Warehousing, Hadoop, Spark

See more: data entry front end excel, develop linux multimedia front end, develop data extract ssis, data pipeline course, data engineering pipeline, what is data pipeline in hadoop, data analysis pipeline, data pipeline tutorial, data pipeline definition, data pipeline examples, how to create data pipeline in hadoop, use vba access develop data centre, develop data access layer vbnet, collect user data joomla front end, excel data input front end, data entry front end, develop data access layer asp net , develop data entry screen mobile phone data entry sending sms, spreadsheet data entry front end, collect data joomla front end

About the Employer:
( 0 reviews ) Atlanta, United States

Project ID: #19016439

8 freelancers are bidding on average $176 for this job


Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. More

$250 USD in 4 days
(179 Reviews)

Have 10 years of IT experience with more than 4.5 years of experience in hadoop technologies like hive,pig,spark,sqoop,map reduce and [login to view URL] have very good experiemcence in Java,scala,Python and shell scripting. More

$222 USD in 1 day
(11 Reviews)

Hello there, We are a team of expert Big Data developers with more than 10 years of rich inductry experience & have succesfully delivered multiple projects in the past like a)recipe recommendation b)movie recom More

$333 USD in 3 days
(2 Reviews)

Hi, I have 8 years of experience and working on hadoop, spark, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud with ha More

$55 USD in 1 day
(3 Reviews)

We have great young and dedicated team, trustworthy and proper completion of project is our moto. Please let me know if you are intesred

$30 USD in 1 day
(0 Reviews)

Hello there, Gone through your details and after reading over your application this looks like a perfect fit for my skill sets!! // Please ping me here so we can discuss details and after analysis, I will provid More

$200 USD in 3 days
(0 Reviews)

I have built several data pipelines using AWS and can definitely design and develop the solution. I will need more details however on this.

$166 USD in 2 days
(0 Reviews)

I work with similar use cases on day to day basis. I have around 3 years of hands on experience with big data technologies. Along with work I can help you with understanding. Reach out to me if you are interested.

$155 USD in 3 days
(0 Reviews)