Spider/crawl/scrape 100 million URLs
This project received 56 bids from talented freelancers with an average bid price of $2158 USD.Get free quotes for a project like this
Project Budget$500-$2900 USD
I have a list of 100 million URLs.
I need them to be spidered/crawled (the website limits the rate at which the URLs can be crawled and also blocks IP addresses that hit it multiple times, so multiple IPs should be used) and scrape about 5 variables of data from each URL (the structure of the information on all the URLs will be the same).
You should then deliver the scraped information in a CSV file or MySQL database.
Looking to make some money?
- Set your budget and the timeframe
- Outline your proposal
- Get paid for your work
Hire Freelancers who also bid on this project
Looking for work?
Work on projects like this and make money from home!Sign Up Now
- The New York Times
- Wall Street Journal
- Times Online