Large-scale web-crawler shopping website

  • Status Closed
  • Budget $3000 - $5000 USD
  • Total Bids 30

Project Description

We invite bids/ proposals from high-caliber firms/ individuals to build a top-notch shopping crawler website which will extract data from a large number of websites across different geographical locations. The distributed web crawler should be robust, able to crawl dynamic websites, perform very deep crawl and have features such as politeness, scalable in design, optimal freshness etc.

The extracted data will be parsed, indexed and stored in a scalable database for a user to query via website. We are not decided on technical design and architecture so would welcome suggestions/ proposals.

If you have previous experience in building a large scale crawler along with aesthetically pleasing aggregator website with high-volume of products then that’s a huge [url removed, login to view] need to be designed in such way as to maximize affiliate revenue. The design and code should be top class.

Typical skills required will be - strong coding skills (PHP, PERL, Python etc) , database design and architecture (MYSQL etc), Distributed computing, SEO etc

We will work with you and provide specific details in our subsequent contacts.

Get free quotes for a project like this

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online