Webcrawler to Mysql

This project was awarded to chirgeo for $350 USD.

Get free quotes for a project like this
Project Budget
$30 - $250 USD
Total Bids
Project Description

I'm looking for an expert who has experience with Nutch or Scrapy to help me set up a webcrawler to scan websites and webfiles and then update a database with the info.

Client-based user interface:

1. create/edit/remove rules

a. real-time webpage scan

b. real-time webpage + crawl scan (crawl means it follows links on the website to other pages, and then scans these pages, for X levels)

c. real-time file scan

d. real-time multiple/batch file scan

rules can be executed one time, continuously, or over a timeframe (every x minutes)

rules can be turned on/off

2. Display live status of the engine in an

3. write to database accroding to structure predefined in rules

Awarded to:
Skills Required

Looking to make some money?

  • Set your budget and the timeframe
  • Outline your proposal
  • Get paid for your work

Hire Freelancers who also bid on this project

    • Forbes
    • The New York Times
    • Time
    • Wall Street Journal
    • Times Online