You have chosen to sponsor your bid up to a maximum amount of .
We invite bids/ proposals from high-caliber firms/ individuals to build a top-notch shopping crawler website which will extract data from a large number of websites across different geographical locations. The distributed web crawler should be robust, able to crawl dynamic websites, perform very deep crawl and have features such as politeness, scalable in design, optimal freshness etc.
The extracted data will be parsed, indexed and stored in a scalable database for a user to query via website. We are not decided on technical design and architecture so would welcome suggestions/ proposals.
If you have previous experience in building a large scale crawler along with aesthetically pleasing aggregator website with high-volume of products then that’s a huge plus.Website need to be designed in such way as to maximize affiliate revenue. The design and code should be top class.
Typical skills required will be - strong coding skills (PHP, PERL, Python etc) , database design and architecture (MYSQL etc), Distributed computing, SEO etc
We will work with you and provide specific details in our subsequent contacts.