Crawlers and Spiders

I would like to have someone review the crawlers, and optimize them in a better way. Answering questions like: Do we need them to constantly run in there entirety or is there a way to consolidate various crawlers into one? The completion of this project, would include a seamless intregration of the crawlers that will not hinder the sites performance This project is to optimize the current 19 web scrapers / crawlers so that they will run efficiently. The scripts are created, but it seems that they will need to be recreated in a different type (i.e. Perl, CGI Bin or otherwise). The best solution will allow the site to run faster, more efficently and be scalable. We believe that it is because the crawlers were made in html, is the reason that the site is running slow. The crawlers are currently running off another server domain. As long as they are not affecting the speed of the site (with a different scraping approach), I would rather them be all on one server. Attached is a word doc that has all of the 19 crawlers explained. ## Deliverables 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables): a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment. b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request. 3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement). ## Platform LAMP, PERL, Cron Jobs, Spiders, and crawlers