We need about 40 crawlers/scrapers for different web sites to be written and maintained for at least 12 months.
Crawlers/Scrapers should be written in .NET (With Source Code presented) and have the following setup fields and control functions/buttons
• Web Site crawled
• Current URL
• Mode (Detailed/List)
• Number of URLs crawled
• Number of Products Crawled
• Crawling Time
• Proxy List File Path (in csv format)
• Load new Proxy List
• User Agent setting Configuration
• Random Delay between URLs setting (From/To sec)
All Bids should be done as per 1 Crawler/Scraper.
All crawlers/scrapers should internally use SQLSever2008 or MySQL database and have 2 crawling modes:
1. Detailed with the following information extracted from Product Details Page and output in SQLSever2008 or MySQL Database and XML file Format
2. List With the following information extracted from Product List and output in SQLSever2008 or MySQL Database and XML file Format: