Closed

Scrap thousands of pages from classified ads website

This project received 18 bids from talented freelancers with an average bid price of $832 USD.

Get free quotes for a project like this
Employer working
Skills Required
Project Budget
N/A
Total Bids
18
Project Description

I need someone to build a bot to crawl thousands of pages of a classified ads website. This website maybe has an anti-crawl mechanism so our bot would have to use a few different IP adresses, have some kind of time management (to not crawl all pages at a time) and change user-agents as well.

I don't think this would be very difficult for someone who already made this kind of bot. Literally several hundred thousands of pages must be crawled. Each page containing maybe 3-4 fields to crawl.

The bot would run once every month. I don't need real time data.

All data should be fed to a MySQL database.

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online