Basic Webpage Spider Crawler Script

  • Status Closed
  • Budget $30 - $250 CAD
  • Total Bids 20

Project Description

I am looking for a reliable developer who has a passion for development. This will be an ongoing project with multiple iterations / stages towards building a larger project.

We would like to build a simple webpage crawler script. The purpose of the crawler is to crawl specific websites for very basic information and then spit the desired data out as a simple xml file.

It should work as follows:


1. Crawler's Subject matter in this example will be shoes

2. Admin will create a set of categories categories

2. Admin user can Enter the desired site to crawl

3. The Crawler will crawl the chosen site

4. The Crawler will pull the data and organize the information into the appropriate categories (based on its data)

5. The Data will be stored in either an XML, CSV or what ever solution is be best for pulling the data QUICKLY and on the fly.

6. Information that will be pulled will be: "Category(s), Product Name, Product #, Size, Color(s), Price, Website, Product image URL, Product URL"

This info will be stored and frequently checked against for updates (via the crawler).

This will be use for 100s of sites.


Get free quotes for a project like this
Awarded to:
Skills Required

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online