Master-Scraper with modules for each website

$30-250 USD

Closed

Posted

over 9 years ago

$30-250 USD

Paid on delivery

I would like to scrape several recruitment agency websites with job listings in regular intervals. Tool needs to scrape the websites every X hours (e.g. 48) and save the output into one single XLSX file. Ie all scraped data goes into this one file, file gets constantly updated but we avoid duplicate records in it. Multi-threading would be preferred. You can use any programming language but I need the full source code plus a program which runs on my Windows PC (normal exe, no need to run as a service). Frontend GUI: Show a grid with all modules which are currently implemented in the tool (every website must have its own module), allow user to select/deselect which modules shall run. BUtton "Start" to launch scraping manually. Checkbox "Automatic mode" if checked then launch scraping immediately and then every X hours from last scraping. Will provide details of how to scrape the various websites. This is a project which can ultimately contain up to 100-200 websites. At the beginning I need the master scraper and 3 modules. If it works we will add additional modules afterwards and I will pay you on a per module basis. If you bid for this project let me know if you can do multi threading, and which programming language you will use. I am looking for someone to work on this long term. Will prefer Ukrainian, Russian or Eastern European freelancers. No bids from India, Pakistan or Vietnam please.

Web Scraping

Project ID: 6838046

About the project

16 proposals

Remote project

Active 9 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

16 freelancers are bidding on average $250 USD for this job

@mantislin

Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi

$267 USD in 6 days

4.9

(94 reviews)

6.5

@mhmhz

Hi i can provide a multi-hreading desktop application. can you provide the initial 3 sites and which data need to be extracted? can work on a demo if you like thanks

$555 USD in 4 days

5.0

(32 reviews)

6.3

@proauthor

Dear employer, I've very good experience in web scraping. I can scrape the jobs sites as per your requirement. I can provide you a java jar file which can be run from any OS. Hope this works for you. Please send in more details to proceed with the project. I'm ready to start working on the project. Looking forward to working with you. Awaiting a positive reply from you. Thanking you Shaik Allabakash.

$99 USD in 5 days

5.0

(33 reviews)

5.1

@JoaoRoque

This is exactly my area of specialty. Given your requirement for a controller GUI I propose using two languages. Each of them the best to accomplish their task. I would use C# to develop the master scraper where you can select which scraper(s) to run and some other options like the automatic mode. And I would use python with the scrapy framework for coding the actual scraper, as it is by far the best scraping framework of any language. It would make further development easy and cheaper than with any other solution. It is also very fast and runs multi threaded. I'm very interested not only in doing this project but also to continue working for you on the further upgrades you mentioned would be necessary. I hope you choose me for this job!

$250 USD in 7 days

5.0

(4 reviews)

3.7

@cyberguard

Hi, we'd be very glad to build such application. However before making long introductions, I do have a few questions to clarify the project: Could you provide a little more detail in regards to the desired output (example of output), plus would you need to customize the output (the name and location of the file, or perhaps the format)? In terms of the language used, we'd be using Perl language compiled into an executable with multithreading enabled. That being said, may I ask why you insist on an executable, to me it seems that this kind of project would definitely be easier and cheaper to do as a simple website on a VPS hosted server; as you need for scraping internet access anyway, you are not limited in any sense, and because you have the tool implemented centrally you can use it from unlimited amount of computers... Secondly it would be much more reliable to have a server for the automated scraping, because if something happens to a computer running executable, automatic scraping will stop of course. Lastly could you provide the website (the initial 3) to be scraped? Anyhow thanks for consideration of the bid, and in case of any questions you might have, don't hesitate to ask! Thanks Simon

$200 USD in 14 days

5.0

(1 review)

3.1

@gomcodoctor

I am expert in scraping and data processing, I can scrap any website using php, benefit of PHP scraping is that we can put it on server and whole process go automatic without need to open our pc, message me for full details

$60 USD in 1 day

5.0

(2 reviews)

2.9

@papawady146

I have already scraping tools to scrape any website. I have many years of experience in Data Entry and Web Research. I put a lot of effort into perfecting every task handed to me. I can help with any admin, virtual, excel, data entry tasks.

$55 USD in 3 days

0.0

(0 reviews)

0.0

@metasoft5

Dear Client, Greeting of the day ahead !!! Thanks for providing us opportunity to place bid over the project and communicate with you. I am a serious bidder here and i have already worked on a similar project before and can deliver as u have mentioned .I have checked your requirements.i have right skills to work on this assignment Here we would like to bring in your notice that We are a team of professionals including experienced analysts, designers, project managers, developers and QA people having great expertise in web applications development mainly on core PHP, PHP with open sources (Joomla, Wordpress, Codeigniter, Cake PHP), .NET, Asp.NET, Vb.NET, HTML 5 etc. and mobile applications on ios and Android platform. We find our expertise, skills and capabilities are perfectly blends with your project requirements because we have already worked on many projects. For Quality Purpose we use : Testing of project in different stages” like module testing, integration testing and system testing. So there will be less probability of any error or issue".can u provide your email or sky-pe etc for further discussion about the project. plz add me to discus I am ready to discuss with you Looking forward to hear from you. with best Regards

$155 USD in 3 days

0.0

(0 reviews)

0.0

@exansoft

Hi Alex Do you remember I successfully completed several Delphi projects for you some time ago? And I would be glad to work with you again. I'm interested in your Master-Scraper project. I have serious experience in both multithreading programming and creating scrapers. And I can implement the scraper using any of following languages - Delphi or Python. Could you please provide URLs of that 3rd sites and provide scraping instructions ? Thank you. Regards.

$400 USD in 12 days

0.0

(0 reviews)

0.0

@makarigor

Hi! I’m a highly skilled Software Developer from Russia, specialized in various web-automation tools, web-scrapers, crawlers, data extractors, automated posters, human behavior emulators and so on. I’ve successfully developed a lot of apps in this sphere, and some of them were not trivial. It looks like this is what you're looking for. I’m really interested in your project as it is exactly within scope of my expertise. I’ve carefully read all your requirements. Some technical aspects should be agreed, but generally it’s clear. And yes, multi-threading will be added. I always offer a high-quality, reliable and scalable solutions, developed in a timely manner. Please, contact me to discuss the details. I’m ready to start working on your project right now. Mike

$250 USD in 10 days