This is the full requirement of the Web Crawler:
1) Products to be picked from every online store that exists on the web. Web Stores would be automatically searched.
2) Products would have to filtered as per Category/Sub-category (The Category/Sub-category taxonomy will be provided)
3) The products would also be filtered by brands so that the Products can be tagged with their respective Brands
4) The data collected would have to be structured so that the main attributes of the Products can be filtered using those attributes/parameters.E.g Price, Accurate Key words, Stores, Url of store, Advanced Search options.
5) The Crawler should be able to distinguish between Online Webstores and websites
I'm attaching the current database structure screen-shots which will help you understand that the current database is unstructured for any kind of search due to the lack of proper attributes.
The crawler would be setup on a Cloud server with cron job or something similar concept so that it works continuously to search online stores and add product data into a MySQL database.
Please let me know what you can do & what would be the time-frame to do this. Please only experienced people who understand and are confident they can handle this should apply. If you cannot show previous experience with crawling scripts please do not bid.