PHP Crawler/Scraper

We would like someone to build a PHP crawler/scraper using cURL.

The application should have a form with 2 input fields.

Input 1: a URL

Input 2: text string for search

Input 1 is the starting URL to start crawling a web directory. The application will crawl the directory and follow outgoing links to websites listed in the web directory.

It should be able to search the HTML code of the website for the text string we specify in Input 2 and then search for the specified string through a maximum of 5 pages.

If the text string is not found in any of the first 5 pages of the site, the application should stop crawling that site. That domain should be stored in the database as a domain to not attempt to crawl again in the future.

If it finds the text string in the code, the scraper should crawl the entire site and collect the following data:

Scraper should retrieve the following content:

The Domain



Meta Description Tag

Email Address - Email Address should be associated with domain it was found on and not page URL it was acquired from.

This data is to be placed into a MYSQL database. One table should contain Domain, URL, Titles and Meta Description Tag. Second table should contain Domain and email information.

We would also like a throttle function to control the number of URL's the program will be crawling at a given time.

We would also like to integrate the Free SEO Moz API into this project to capture the URL analytics.

Skills: MySQL, PHP, Software Architecture

See more: web crawler architecture, text string search, search text string, php string to html, php form search mysql database, php directory software, php code search mysql database, html 5 form tag, architecture of web crawler, curl software, website crawler, scraper, scraper software, php software architecture mysql, email scraper, email crawler, database crawler, data scraper, data crawler, crawling of data, crawler, crawl a we, scraper mysql php, web data crawler, function crawl

About the Employer:
( 0 reviews ) Chicago, United States

Project ID: #1200302

7 freelancers are bidding on average $274 for this job


I can do this PHP crawler to scrap the details from different websites using cURL and integrate Free SEO Moz API also. Thanks, Suresh

$250 USD in 5 days
(561 Reviews)

I can help you on this project. Please check pm. Thanks

$250 USD in 5 days
(19 Reviews)

I can do it for you. Please PMB for me.

$250 USD in 4 days
(8 Reviews)

plz check PM for details

$400 USD in 2 days
(22 Reviews)

We are professional developer for any Website Design with PHP, Joomla, Wordpress, Flash, MySQL, CSS etc. We have separate specialist for these work with guaranty of satisfaction. Recently finished project http:/ More

$200 USD in 1 day
(3 Reviews)

Hi there, I have a few years expereince in coding in php and would like to take this project on becuase it seems interesting. Yours sincerely Dan

$250 USD in 3 days
(0 Reviews)

Hi, We have go through the Project requirement and ready to do this project for you. Please check your PMB

$320 USD in 6 days
(0 Reviews)