Completed

We need about 40 crawlers/scrapers for different web sites to be written and maintained

This project was successfully completed by MSGuys for $50 USD in 3 days.

Get free quotes for a project like this
Employer working
Completed by:
Project Budget
$30 - $250 USD
Completed In
3 days
Total Bids
13
Project Description

We need about 40 crawlers/scrapers for different web sites to be written and maintained for at least 12 months.
Crawlers/Scrapers should be written in .NET (With Source Code presented) and have the following setup fields and control functions/buttons
• Web Site crawled
• Current URL
• Mode (Detailed/List)
• Number of URLs crawled
• Number of Products Crawled
• Start/Stop/Exit/Restart
• Crawling Time
• Proxy List File Path (in csv format)
• Load new Proxy List
• User Agent setting Configuration
• Random Delay between URLs setting (From/To sec)
All Bids should be done as per 1 Crawler/Scraper.
All crawlers/scrapers should internally use SQLSever2008 or MySQL database and have 2 crawling modes:
1. Detailed with the following information extracted from Product Details Page and output in SQLSever2008 or MySQL Database and XML file Format
2. List With the following information extracted from Product List and output in SQLSever2008 or MySQL Database and XML file Format:

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online