Seeking an experienced web crawler/spider developer to develop a website crawler/spider or use an existing one that they may have.
The web crawler/spider will have the following characteristics :
Preferably written in PHP but Perl will be accepted as well.
1. Primary purpose is to crawl a number of real
estate sites for property listings and return the specific listing
results to our MySQL dB to populate our free property listing database.
2. Each site being crawled will have differing layouts, and so
it must be possible to specify per site the layout and what to search for/return.
3. The crawler must run on a Linux server and be capable of being
scheduled to automatically run at certain times.
4. Admin functions must be allowed to enable the addition/removal of
sites and to update layout instructions per site.
This is the first project to complete.
The successful winner will also have the first chance to write a new crawler for our new hotel listing website which will bring hotel price results from different hotel/travel websites and enter them in to MySQL dB (will be paid separately).
Please reply with specific experience and examples of crawlers/spiders you have developed that meet similar requirements.