Closed

Need extremely fast web scraping solution

I require you to help me implement a solution that will allow me to scrape and process a huge amount of data per minute.

The end product should support scraping of approximately 1000 random webpages per minute.

We will assume that these pages are from random websites on the internet and take approximately 3-5seconds to load and a further 2 seconds to process (extract patterns and insert into database). You however, will only be required for the Sever / Language recommendation part and some basic programming to show me how it all fits together.

Ideally I would like to work with PHP/Multi-Threading/PHP-SIMPLE-DOM but I have a strong feeling this is to resource intensive for what I require, hopefully someone can prove me wrong. What's the fastest way we can get this done?

You know exactly what is needed, now you need to sell yourself to me! Answer these questions:

How much RAM would we need?

How much CPU would we need?

How many server instances?

Approximate monthly server costs?

What language would you do it in?

Is multi-threading supported in this language and if so, how does it work?

No point bidding and not telling me what you're plan is, so please, no copy&paste replies.

Just be honest with your ideas and answer my questions in full and you'll be more likely to be chosen!

Skills: PHP, Software Architecture

See more: what you need to know for programming, threading programming, sell yourself, programming patterns, php programming patterns, need help with php programming, how to know programming language of a software, fast web programming language, fast web, fastest programming language, dom programming, architecture recommendation, need programming help fast, multi threading, multi part php, php multi part, basic programming database, require someone extract, need product sell someone, php dom, copy data internet database, know product help sell, 1000 questions copy paste, software architecture plan

About the Employer:
( 39 reviews ) NY, United Kingdom

Project ID: #4302371

6 freelancers are bidding on average $203 for this job

SigmaVisual

I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

$199 USD in 4 days
(218 Reviews)
7.7
TheInnoVibes

Please check private message box.

$100 USD in 2 days
(32 Reviews)
5.7
navelsoft

please check inbox

$220 USD in 30 days
(2 Reviews)
3.7
ldanadrian

Hello, I'm very experienced with crawlers/spiders, in the past 10 years i've made at least 10-20 spiders/year. Check my private message for my opinion.

$250 USD in 7 days
(1 Review)
3.0
pythonshell

consider it done . !!! check pm.

$250 USD in 4 days
(9 Reviews)
2.9
mialox

Hello! Ready to work on this project/ Check PM.

$200 USD in 3 days
(0 Reviews)
0.0