Webcrawler to Mysql

In Progress

I'm looking for an expert who has experience with Nutch or Scrapy to help me set up a webcrawler to scan websites and webfiles and then update a database with the info.

Client-based user interface:

1. create/edit/remove rules

a. real-time webpage scan

b. real-time webpage + crawl scan (crawl means it follows links on the website to other pages, and then scans these pages, for X levels)

c. real-time file scan

d. real-time multiple/batch file scan

rules can be executed one time, continuously, or over a timeframe (every x minutes)

rules can be turned on/off

2. Display live status of the engine in an

3. write to database accroding to structure predefined in rules

Skills: Java, MySQL, PHP, Python, Web Scraping

See more: python to java, Webcrawler, nutch, d scans, c++ c java python mysql,, mysql create database engine, python webpage, batch python, python create file, scrapy create, python file create, python batch, python webcrawler, java crawl, rules engine interface, mysql batch, java time based update, mysql java client, write structure java file, python remove file, file scan, create mysql database java, python write database, python interface, python create webpage

About the Employer:
( 9 reviews ) Hong Kong, China

Project ID: #4262909

Awarded to:


Hi! I'm an expert in Scrapy. Did lots of projects based on this framework. Check please PM, have some questions.

$350 USD in 6 days
(5 Reviews)

10 freelancers are bidding on average $325 for this job


I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

$225 USD in 5 days
(247 Reviews)

Let us discuss further about the project

$320 USD in 12 days
(109 Reviews)

##### YOUR SEARCH ENDS HERE! ##### GET IT RIGHT THE FIRST TIME!. Check the message and contact us. SI TEAM.

$250 USD in 10 days
(16 Reviews)

Automation expert here. I can easily write such a bot/scraper.Please check your PMB.

$400 USD in 5 days
(13 Reviews)

Interested to work on your project. Thanks.

$400 USD in 25 days
(30 Reviews)

Ready to start. Thanks

$350 USD in 15 days
(6 Reviews)

Sir, first and foremost allow me to thank you for inviting me to this interesting project. I truly believe you chose the right developer for the job. Please see my message for more details, thank you.

$200 USD in 7 days
(8 Reviews)

Hi. Experienced Python/C++ programmer. I have made several scrapers, also with the ability to scan n-levels deep.

$250 USD in 7 days
(5 Reviews)

Hello I'm experienced python developer. My specialization is data mining and scraping. I will be happy to help you.

$250 USD in 5 days
(3 Reviews)

I am ready

$500 USD in 90 days
(0 Reviews)