Closed

Spider/crawl/scrape 100 million URLs

I have a list of 100 million URLs.

I need them to be spidered/crawled (the website limits the rate at which the URLs can be crawled and also blocks IP addresses that hit it multiple times, so multiple IPs should be used) and scrape about 5 variables of data from each URL (the structure of the information on all the URLs will be the same).

You should then deliver the scraped information in a CSV file or MySQL database.

the urls are all from one domain

Skills: Data Mining, Data Processing, Web Scraping

See more: data scrape database, crawl data, crawl a website, mysql scrape website, scrape data list urls, scrape data mysql, url crawl, crawled data, 100 million, scrape addresses website, scrape data website mysql, website scraping mysql, scrape website url, url spider, rate scraping, database scrape, database spider, spider csv, mysql web scrape, scrape url list, spider website csv, mysql web crawl data, csv million, spider data website, crawl data mysql

About the Employer:
( 89 reviews ) San Francisco, United States

Project ID: #4776744

52 freelancers are bidding on average $2133 for this job

pandeypriya2003

Please check inbox. Thanks

$2577 USD in 10 days
(400 Reviews)
7.1
greggfletcher

Hello, i have expertise in web scraping and data mining. Please refer to your INBOX for details. Thank You. Kind Regards.

$2577 USD in 15 days
(66 Reviews)
6.8
ehsankayani

Hi, Kindly see details in PMB Thank you

$1578 USD in 30 days
(40 Reviews)
6.0
cheapexcell

Ready to start

$2898 USD in 30 days
(54 Reviews)
6.0
dreamchandra

Greetings! Ready to start from the right away. Refer pm for more details. Thank you!!!

$3052 USD in 25 days
(12 Reviews)
5.9
enamulraju

Ready to do this for you

$5263 USD in 25 days
(83 Reviews)
5.8
jrahman10

Ready To Start .... Professional In This Area .....

$2888 USD in 3 days
(75 Reviews)
5.7
phpXpertbd

Seasoned web scraper. I worked on many similar projects, I have big experience in data mining projects. I can finish this task in short time, with the best quality.

$2444 USD in 3 days
(18 Reviews)
5.7
fancywwh

PM for more details, hope can help you on these big data, thanks.

$3052 USD in 30 days
(33 Reviews)
5.6
HongGiang

Hello sir! Thank you for your invitation. I'm very interested in this project and ready to discuss more. Please see my letter in PM. Thanks, Giang

$2421 USD in 30 days
(54 Reviews)
5.7
uumairkhalid

Hello. Expert is here to help you in this project. Lets do it with me. Thanks

$2631 USD in 3 days
(43 Reviews)
5.6
svteam

Hello, I'm an expert in web scraping. ready to do it. Please check PM for details. Thanks.

$1500 USD in 30 days
(44 Reviews)
5.6
Narfo

Hello, please check my message!

$2400 USD in 30 days
(8 Reviews)
5.5
Game256

Hello, please check PMB

$2157 USD in 30 days
(5 Reviews)
5.2
evidyarthi

Hi, Thank you for posting the project

$500 USD in 30 days
(2 Reviews)
5.2
dablu11

Thanks for invitation. Please check pm for details.

$3333 USD in 45 days
(30 Reviews)
5.2
DeepSyaal

Lets Start. Having a team of Professionals.

$2210 USD in 30 days
(75 Reviews)
5.1
tjawad17

Expert Web scraping tam here. Please find PM for details. Thanks

$2900 USD in 20 days
(22 Reviews)
5.1
aoefmpes

ready to do, pl check your inbox

$1888 USD in 300 days
(29 Reviews)
5.0
abupabuya

hi sir.i have a list , pls check my sample at message inbox

$526 USD in 1 day
(31 Reviews)
5.0