Scrap thousands of pages from classified ads website
- Status Closed
- Budget N/A
- Total Bids 18
I need someone to build a bot to crawl thousands of pages of a classified ads website. This website maybe has an anti-crawl mechanism so our bot would have to use a few different IP adresses, have some kind of time management (to not crawl all pages at a time) and change user-agents as well.
I don't think this would be very difficult for someone who already made this kind of bot. Literally several hundred thousands of pages must be crawled. Each page containing maybe 3-4 fields to crawl.
The bot would run once every month. I don't need real time data.
All data should be fed to a MySQL database.Get free quotes for a project like this
Looking to make some money?
- Set your budget and the timeframe
- Outline your proposal
- Get paid for your work
Hire Freelancers who also bid on this project
Looking for work?
Work on projects like this and make money from home!Sign Up Now
- The New York Times
- Wall Street Journal
- Times Online