Build an E-commerce site crawler

  • Status Closed
  • Budget N/A
  • Total Bids 9

Project Description

I need a custom crawler than can crawl 10,000s of pages from an e-commerce site (single domain) to collect the necessary product/pricing information that can be presented back in an XML file and Excel file format. I need to then integrate this file to my e-commerce site to add the necessary information to my product pages. I don't have my site ready yet but I will have to get the integration done based on the data I can get from the crawler. The crawler will also have to login to the website first before crawling the pages as the key information on the pages are only available when logged-in.

There are lots of open source software in the market than can be used to build this crawler so ideally I don't want to pay anything on an on-going basis. But I am happy to discuss the additional cost based on the platform that you suggest using, What I want is someone experienced to create the custom code for the crawler, ensure the data is collected and presented the way I need. I have to use the crawler on a weekly basis to crawl the pages and collect the data.

I can provide any more details that you need if you are interested in this project. Previous experience with creating web crawlers is a must. And I need this to be delivered within a week from the start date



Get free quotes for a project like this

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online