This Project will create [url removed, login to view] a clone of a major Search Engine Portal. Bidders must have had experience with Lycos or Google functional clones. Preferably you've just completed a search engine project that you feel good about. I will check your feedbacks ratings.
1. Website must have all functionalities of a major search engine.
Spider Crawling Speed, frequency, Search and report speed, state of the art search engine operational capabilities, User Friendliness and professional quality website extremely important.
(Only Freelance providers with prior major search engine clone experience can bid)
2. A scipt with google sofistication is a must. This can be done with your own script or vendor script. The result should be a list of over 5 million URLs crawlled consistently and search results displayed with professional quality page view.
3. capabilities to include but not limited to a multi-thread web crawler that will access lists of www URLs crawling the entire span of the Internet to the end of the internet. Spidering each homepage and gathering information (meta-tags, content, etc.) The spider will harvest the first page and any secondary pages (future crawls may go deeper into the site).
4. It will also use "stop words", meaning it will not gather certain words (a, the, an, etc.). All content will be sent to a database (or any suggestions). The size of the database will be large so speed is a concern. Suggestions welcome. Simply put, a professional clone of a major search engine portal.