Page Scraping Script

• Project ID:

  788338

• Project Type:

  Fixed

• Budget:

  $30-$250 USD
  

Project Description:

Page Scraping.

Hi,

I am looking for someone who is proficient in writing a PHP script to page scraping multiple URLs, and storing resulting data into mysql database / tables.

There are 3 levels of URLs to scrape. Each level of URL, has multiple pages.

I will supply the first level URLs to scrape.
The data scraped will result the 2nd and 3rd level of pages to scrape.

The first level of URL is approximately 290 pages.
Based on the result of a singular first level page, there are 1-20 second level URLs
Each second level URL will result in 1-10 third level of URLS to scrape.


The data must be stored in tables so that data can be referenced to each other.
The levels are naturally hierachical eg

RaceDay (Level 1 URL Data) → Venues (Level 2 URL Data) → Races (Level 3 URL Data)



Example
Level 1 url : http://unitab.com/data/racing/2010/6/28/RaceDay.xml

From the data scraped at level 1, using the MeetingCode variable we create the 2nd and 4d level URL to scrape
2nd level : http://unitab.com/data/racing/2010/6/28/NR.xml
To create the 3rd level use the “HiRaceNo” variable. Eg HiRaceNo="7" would result in :
3rd level : http://unitab.com/data/racing/2010/6/28/NR1.xml
3rd level : http://unitab.com/data/racing/2010/6/28/NR2.xml
3rd level : http://unitab.com/data/racing/2010/6/28/NR3.xml
3rd level : http://unitab.com/data/racing/2010/6/28/NR4.xml
3rd level : http://unitab.com/data/racing/2010/6/28/NR5.xml
3rd level : http://unitab.com/data/racing/2010/6/28/NR6.xml
3rd level : http://unitab.com/data/racing/2010/6/28/NR7.xml

http://unitab.com/data/racing/2010/6/28/VR.xml
http://unitab.com/data/racing/2010/6/28/CR.xml
http://unitab.com/data/racing/2010/6/28/BT.xml
http://unitab.com/data/racing/2010/6/28/NT.xml
http://unitab.com/data/racing/2010/6/28/VT.xml
http://unitab.com/data/racing/2010/6/28/AT.xml
http://unitab.com/data/racing/2010/6/28/PT.xml
http://unitab.com/data/racing/2010/6/28/TT.xml
http://unitab.com/data/racing/2010/6/28/IT.xml
http://unitab.com/data/racing/2010/6/28/BG.xml
http://unitab.com/data/racing/2010/6/28/SG.xml
http://unitab.com/data/racing/2010/6/28/MG.xml
http://unitab.com/data/racing/2010/6/28/NG.xml
http://unitab.com/data/racing/2010/6/28/VG.xml
http://unitab.com/data/racing/2010/6/28/AG.xml
http://unitab.com/data/racing/2010/6/28/PG.xml
http://unitab.com/data/racing/2010/6/28/EG.xml
http://unitab.com/data/racing/2010/6/28/ZG.xml
http://unitab.com/data/racing/2010/6/28/TG.xml
http://unitab.com/data/racing/2010/6/28/AS.xml
http://unitab.com/data/racing/2010/6/28/ES.xml

There are further finer details to the script, but this will be discussed further with the right programmer!

Skills required:

MySQL, PHP, Web Scraping