Build an E-commerce site crawler


I need a custom crawler than can crawl 10,000s of pages from an e-commerce site (single domain) to collect the necessary product/pricing information that can be presented back in an XML file and Excel file format. I need to then integrate this file to my e-commerce site to add the necessary information to my product pages. I don't have my site ready yet but I will have to get the integration done based on the data I can get from the crawler. The crawler will also have to login to the website first before crawling the pages as the key information on the pages are only available when logged-in.

There are lots of open source software in the market than can be used to build this crawler so ideally I don't want to pay anything on an on-going basis. But I am happy to discuss the additional cost based on the platform that you suggest using, What I want is someone experienced to create the custom code for the crawler, ensure the data is collected and presented the way I need. I have to use the crawler on a weekly basis to crawl the pages and collect the data.

I can provide any more details that you need if you are interested in this project. Previous experience with creating web crawlers is a must. And I need this to be delivered within a week from the start date



Skills: Anything Goes, PHP, Software Architecture

See more: web crawler architecture, ready to use e commerce website, i need someone to market my product, e commerce website php code, creating e commerce website, architecture of web crawler, an e commerce site, what is web crawling, what is a crawler, what are web crawlers, website crawler, web crawlers, key information, e-commerce product data, e-commerce data, e commerce site, data crawler, custom web project discuss, crawling of data, crawler, commerce site, build my site, build my e-commerce website, build e-commerce, build e-commerce site

About the Employer:
( 0 reviews ) London, United Kingdom

Project ID: #2792206

8 freelancers are bidding on average $249 for this job


Ready to start. Thank you!

$50 USD in 2 days
(85 Reviews)

Hi, Please check inbox for details and samples of our works done. Thanks

$180 USD in 6 days
(23 Reviews)

Automation expert here. I can easily write such a bot/crawler.Please check your PMB.

$250 USD in 3 days
(11 Reviews)

Hi, I ve already finished many scraper projects like this.

$60 USD in 5 days
(21 Reviews)

==========I am ready to start. Kindly check my work details.=========

$200 USD in 10 days
(3 Reviews)

ecommerce/php team

$400 USD in 4 days
(9 Reviews)

Already we have experience for search engine web application. Crawler, Index and SE for web html content, image... So please consider my bid.

$300 USD in 5 days
(1 Review)

Dear Sir, Greetings! We are Reform Consulting with 5 year of experience under our belt. We are really interested to work on your project. Please see PM. Thank you. Regards, RCPL.

$250 USD in 15 days
(3 Reviews)

HI, I can do this task for you, I have extensive experience in writing crawlers using Java and Python, I have written crawlers that can crawl websites and export data to csv. see you pmb for details

$500 USD in 7 days
(1 Review)