Web Crawler / Spider / Robot

CLOSED
Bids
24
Avg Bid (USD)
$507
Project Budget (USD)
$250 - $750

Project Description:
Hello,

Please find a description of my projet in English.
Be aware that I'm french, so if you speak french, it will be a plus but don't worry I'm fluent in English too !

So, I made a project, few month ago to get a crawler.
It was coded in .NET. Even if the work was well done, the spider is too slow, the connexion to the database is slow too, and it's not optimized to crawl hudge websites.

Because of all theses problems, I want to do a new projet, and the last one. I'll choose carefuly the freelancer to be sure to not spend time and money for a no project.

About the crawler, the main goal is, as you know, to crawl the web !
So starting by a given URL, it will crawl the website and find child URL, external URLs ...
It will have to use some options (not follow style sheets, javascript ... for example).
The crawler can be done in java or any language of your choice, the only thing I need is to control it by PHP.

I don't know java... so it has to be easy to use (and setup if I have to setup it on my server).

You can use any OPEN SOURCE project (as nutch ...), you will save time and me money :)

So the request included programation, source and php interface to control the spider which should be working without human.

More details will be give at the time of the choice.

What I want ?
- A unique reply, don't need to send me the same presentation that you send to anyone, you will be exclude of the short list
- Tell me which language you plan to use, if you have an idea of a open source project to use ?
- tell me if you already coded a spider for some (sample, example enclosed)
- after, other details are up to you.

The presentation is important, I don't have time to ask the same question every time !

Kind regards

Additional Project Description:
03/03/2013 at 8:50 AKST
Please do no bid if you know ONLY PHP.

The program has to be multi threads so Java, Python ... but not PHP for the core

Skills required:
Java, PHP, Python, SQL
Hire bournend
Project posted by:
bournend France
Verified
Public Clarification Board
Bids are hidden by the project creator. Log in as the employer to view bids or to bid on this project.
You will not be able to bid on this project if you are not qualified in one of the job categories. To see your qualifications click here.


$ 299
in 5 days
Hire jfreyre
$ 250
in 0 days
$ 520
in 8 days
Hire bob1982
$ 700
in 10 days
$ 399
in 10 days
$ 530
in 14 days
$ 300
in 3 days
Hire tlyx
$ 550
in 30 days
$ 250
in 2 days
$ 663
in 3 days