Completed

Setup regular job for cleaning and joining s3 files

I have setup some crawlers which store data into S3. Each crawler stores the daily scrape load as a separate file. I need to write a regular job which cleans the daily data scraped for each of the different crawlers, joins the data together, and adds a date column as partition. The cleaned and joined data should then be added to another s3 table which aggregates all the clean data on daily basis. Ideally all this should happen within the AWS environment. This final aggregated dataset will then be used to query and extract a daily report and showcase historic trends.

Skills: Amazon Web Services, Elasticsearch, NoSQL Couch & Mongo, Python, Web Scraping

See more: setup cron job linux plesk, appointment setting job cleaning, joining nzb files, s3 automatically delete old files, s3 bucket lifecycle policy, aws s3 lifecycle example, s3 lifecycle prefix wildcard, elastic transcoder thumbnail interval, s3 bucket object lifecycle, s3 lifecycle not working, when do s3 lifecycle rules run, setup hotspot job, setup cron job dolphin linux hosting, setup cron job sent mail, setup cron job via ssh, setup cron job send mail, setup backup job sql express, job opportunities transcribing audio files, joining video files ffmpeg, job opportunity editing audio files

About the Employer:
( 1 review ) München, Germany

Project ID: #19647993

Awarded to:

JesperLabs

Hello Client, I see that you require help having a perfect data set ready for querying. I can help you write your aggregator which aggregates from various s3 objects to one single object which shall be your golden sou More

€25 EUR / hour
(0 Reviews)
0.0

19 freelancers are bidding on average €26/hour for this job

peng72083

Hello, how are you? I'm a senior web developer and I have rich experience in web scrapping, AWS infrastructure etc. I've read your description and I'm very confident of this job. Let us discuss in detail via chat. More

€30 EUR / hour
(46 Reviews)
7.4
adeelpirzada

Hi, i have done scrapping almost on Half of Worldwide web including eCommerce giants (Amazon, eBay, craigslist) News Feed, Social media websites, API's. I develop my own scrapers and Bots based on requirements w More

€18 EUR / hour
(30 Reviews)
6.3
schoudhary1553

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl More

€18 EUR / hour
(34 Reviews)
6.2
khangcnttit92

Hi nice to meet you Firstly, Thanks for visitting my profile. With more than 8 year in aws. Then I can make it as best. Can we talk? Thanks you in me. Could I use lambda or batch job to make this crawl?

€20 EUR / hour
(21 Reviews)
5.9
kirillsoft

Hi, As a Senior Python engineer, I have experience of web scraping and using amazon web services. I am available to discuss more through private chat. Sincerely. Kirill

€30 EUR / hour
(7 Reviews)
6.2
Ipullar

Hi, there. AS a Senior Python engineer, I have experience of writing cron jobs using python. And also familiar with amazon lambda. I am available to start immediately. Yuri Ren

€30 EUR / hour
(12 Reviews)
5.0
yuriyes43

Dear, sir I've read your specification carefully and I'm very interested in your job. I have rich experience of web scraping. I ensure high quality and deadline. I would like to have a detailed conversation with you r More

€35 EUR / hour
(7 Reviews)
4.8
KashirinA

⭐⭐⭐⭐ i am Full stack developer. ⭐⭐⭐⭐⭐⭐ I read through the job details extremely carefully and I am absolutely sure that I can do the project very well. * 6+ years experience in web and mobile app development. If More

€30 EUR / hour
(3 Reviews)
4.5
xxemre

I can do this by using aws lambda or similar serverless technology. It will trigger on pushes to S3 store and do your job. AWS lambda code will be in javascript or C#.

€44 EUR / hour
(1 Review)
4.2
alkajain2906

Hi, I have 5+years Experience in Website/Software Design & Website Development. I'm an Expert in: -Frontend & Backend Development -Python -Data Science -Machine Learning -JVM technologies -Jquery, PHP -Ion More

€30 EUR / hour
(6 Reviews)
4.0
christinestraub

Hello Nice to meet you. I'm a full-stack developer with 5 years of experience in web development. [login to view URL] [login to view URL] [login to view URL] [login to view URL] and also http://ww More

€27 EUR / hour
(4 Reviews)
3.8
fastestJohn

Hi, sir. am Gavrichkov. I have read carefully your requirements. I have the best skills and much experiences in building robotic process automation tools for scraping in Python. I have already completed so many project More

€20 EUR / hour
(5 Reviews)
3.6
zpedia723

Hello, how are you? My name is Fengshu L. I have read your description very carefully and it was interesting. I have developed WebSites with WordPress for over 5 years and have good design senses. Especially More

€18 EUR / hour
(1 Review)
3.1
Mindlarz

We are a web developing and designing company with various other facilities such as web content and digital marketing related to all types of specializations. Additional considerations and implementations are assisted More

€22 EUR / hour
(0 Reviews)
0.0
falconsoft6

"Hi, Hope you are doing well! Thanks for sharing your project requirement with us. It will be our great pleasure to work on your project. I have checked your requirement, yes we can do it, because we already work on si More

€30 EUR / hour
(0 Reviews)
0.0
erashrafkamal

*****WEB SCRAPPING EXPERT,,,,,I CAN START NOW****** Dear Employer, Hope all's well..!! Thanks for job posting. I have read the description carefully and give you 100% assistance as React.js + Node.js backend for re More

€25 EUR / hour
(0 Reviews)
0.0
ishika11

Hello, To come up with all your requirements I have prior experience in the following: - In-depth knowledge of at least one high-level programming language. - Proficiency in Core PHP and its frameworks (Laravel an More

€20 EUR / hour
(0 Reviews)
0.0
xavient3

"Hi, Hope you are doing well! Thanks for sharing your project requirement with us. It will be our great pleasure to work on your project. I have checked your requirement, yes we can do it, because we already work on si More

€30 EUR / hour
(0 Reviews)
0.0