Screen Scrape to XML to Save Local file
Project ID:
547331Project Type:
FixedBudget:
$30-$250 USD
Project Description:
I am looking to screeb scrape a specific site http://offender.fdle.state.fl.us/ to collect data on registered sex offenders. The criteria for the search url to return the records I am interested in is as follows: http://offender.fdle.state.fl.us/offender/offenderSearchNav.do?county=marion&link=doSearch&commaSeparatedOffenderStatus=1,6,7,8,9&stateStatus=1&offenderType=3However, hitting that URL directly seems to redirect you back to the homepage unless you already have an active session on the site. I suspect this is the first tricky spot as a session or something needs to be set with the parsers.
Once you do get the results, you will notice in a hidden field that all the IDs exist for the results. I anticipated using those ids to build the urls for the next part of the scrape where the offenders record would be built. It is a hidden field. <input type="hidden" name="commaSeparatedPersonIdsALL"
From these ids, the url to the respective record can be formed: http://offender.fdle.state.fl.us/offender/flyer.do?personId=16687 using the ID for the personID.
From this form I would like the following data scraped, including a url to the image and combined into an XML feed which will later be imported into our database (the DB import is not part of this project).
From right of photo....
-------------------------
Designation: Sexual Offender
Name: Samuel E Ackerson
Status: Released - Required to Register
Department of Corrections #: D93831
Search the Dept of Corrections Website
Date of Birth: 05/28/1975
Race : White
Sex: Male
Hair: Blond
Eyes: Blue
Height: 5'10"
Weight: 153 lbs
Below Photo....
--------------------
Samuel E Ackerson
Date Of Photo: 11/03/2009
Aliases
Scars, Marks & Tattoos
From Address Information I would like the first Address and Address Source Information> I would also want longitutde and latitude extracted from the map link for the address being imported. This will be stored in db on import for Geo coding on map.
From Crime Information - Qualifying Offenses I would like all the information brought into the feed as a table using the same headers as the page but without color or formatting.
Again, this data should all be produced into an XML file that I will later use to import into the DB. The XML file should be stored on each run when completed and named with time/date stamp. The process should be setup to be able to be run via windows task manager so maybe php curl from command line or something similar... not my area of expertise.
Additional Project Description:
11/23/2009 at 0:33 EST:Also note that I will need personID in the XML output for each record.
Skills required:
PHPPublic Clarification Board
1 messages
-
marketmike
Please note the opening sentence " am looking to screeb scrape a specific " should have been am looking to screen scrape a specific
Also, an example of the hidden field with ids is below...
<input type="hidden" name="commaSeparatedPersonIdsALL" value="16687,41644,30020,31961,66222,62217,10716,11841,36104,30272,17925,2554,40761,56296,57853,24907,7915,60755,22795,46897,3354,56903,4023,63671,31381,6865,7118,20152,48253,36359,1879,24696,49276,60560,61866,7149,41445,43920,29696,34644,62132,50487,44017,2751,25632,38396,55597,8403,2846,41867,52692,23159,29889,15619,25364,38214,43196,36678,66572,8701,32372,39280,43413,1714,50078,9090,35765,4594,2271,3409,26521,28740,44401,63743,1743,41789,36446,60813,68684,28815,6872,47738,8828,20681,21883,59715,14223,42455,27369,29897,53954,11434,26182,42884,41905,6854,32394,33623,9096,33739,16331,32050,7108,36539,35747,10301,51572,21370,37554,573,13391,65458,4559,47594,57261,65604,2643,65165,43224,6387,31989,4108,8971,63766,6868,37644,44339,22787,67188,57302,65035,6856,41843,38353,42789,14099,8823,51283,38491,6873,57115,26069,60995,39426,63547,65858,15937,36950,57050,25288,2464,44941,64997,68491,46285,26889,34103,55294,30451,37825,2090,8303,26099,55869,25289,15403,65711,9227,2796,30043,44011,14544,45925,31623,62878,65077,59046,29236,1636,42089,23667,213,48838,53387,67251,1419,41046,34292,51608,42803,48346,36597,1739,8711,1764,41315,36578,44060,25406,63399,2753,15405,23046,5841,11217,17052,29812,8302,12462,9228,700,33400,27179,33219,14242,726,14749,16831,25194,29753,63754,29809,18682,34059,47937,29752,32979,31935,67564,34205,32943,46003,1767,7730,32127,27605,68979,61932,1441,15831,39070,39069,40721,57328,31741,61401,30287,10616,6430,6906,41499,68517,40388,16052,33472,48832,50744,7991,29833,28458,29567,43804,27492,31473,23963,52491,27472,57185,60906,18069,60986,39054,30816,20362,63824,46773,40232,6916,32038,55201,64230,22596,17533,36019,68255,1949,326,43843,36747,48811,39452,3401,3989,43205,42090,5338,36981,32868,8329,20142,39057,68353,38277,41820,26273,7401,36423,38254,34026,27968,3299,36571,2694,43851,39944,9048,43337,14067,48620,17041,11845,51401,4933,40906,9716,30880,41422,29792,29882,20798,8722,357,4374,56124,29052,21516,60096,2950,40009,39855,57160,61367,26445,38572,15155,2959,37765,49326,29726,61107,39133,64877,26315,24908,59584,38142,9003,5400,9704,381,39082,23258,15805,56176,9952,1092,42931,36773,24563,65415,24868,30776,22173,12912,31116,25856,32043,36887,63685,43478,29332,2979,30893,5796,63846,33550,68944,2541,31908,6890,31097,28608,1756,15853,68047,23346,10418,21658,6859,65378,6852,24741,30274,32443,26007,1738,52194,1771,22639,16387,55167,4459,29613,8647,44819,68490,39214,14676,42015,1753,21018,11425,61503,1779,33923,58329,2509,51559,38896,6861,35405,36996,32985,2937,6881,4569,9471,25880,710,27373,25296,55030,54368,2968,6904,6903,43602,8546,31013,1742,32399,38790,10373,59874,22904,35428,8650,399,43883,35640,38005,8361,62410,13014,53683,27676,67486,30564,48874,58437,67015,22280,12266,9992,50225,8866,28716,54413,38068,48167,52329,33126,42213,40920,10384,66122,40516,1757,23776,7638,41674,1747,28971,6886,29069,8832,57517,12636,43108,429,8708,41382,53193,21810,48543,49242,25387,40437,5745,13482,982,16881,22161,25830,30907,41277,31901,54893,46500,29428,36684,41761,52824,18435,41970,16639,52444,65193,4197,14675,62769,6901,21514,47873,33471,27627,39783,10812,46428,9846,31975,33770,28402,48751,6857,24808,25490,7455,46774,38066,6057,59969,502,21697,46671,2715,61220,51774,11243,25733,27020,8337,18189,3812,29826,37006,27083,36144,14222,34342,62472,9161,36757,34715,54728,4572,6874,38092,48807,40917,26771,25832,6955,44608,8704,66285,66993,47646">over 2 years ago
Awarded Bids
All Bids ()
VALUEONWEB
|
VALUEONWEB
From India
Gold Member
Offline
|
Foundation LimeExchange Member
Foundation EUFreelance.com Member
SigmaVisual
|
SigmaVisual
From Pakistan
Gold Member
Offline
|
General Freelancer Orientation (80%, 90th percentile)
Foundation EUFreelance.com Member
websree
|
websree
From India
Gold Member
Online
|
General Freelancer Orientation (80%, 90th percentile)
Teknowledge
|
Teknowledge
From India
Gold Member
Online
|
General Freelancer Orientation (95%, 100th percentile)
sunztech
|
sunztech
From Pakistan
Gold Member
Online
|
General Freelancer Orientation (75%, 81st percentile)
NishantBamb
|
NishantBamb
From India
Gold Member
Offline
|
General Freelancer Orientation (95%, 100th percentile)
dboyzhang
|
|
dboyzhang
From China
Gold Member
Offline
|
Foundation Freelancer.co.uk Member
General Freelancer Orientation (100%, 100th percentile)
Foundation EUFreelance.com Member
MAnkita
|
|
MAnkita
From India
Gold Member
Online
|
General Freelancer Orientation (80%, 90th percentile)
Foundation EUFreelance.com Member
mantislin
|
mantislin
From China
Gold Member
Online
|
General Freelancer Orientation (85%, 95th percentile)
Foundation EUFreelance.com Member
clayarcs
|
clayarcs
From United States
Gold Member
Online
|
Foundation Freelancer.co.uk Member
General Freelancer Orientation (90%, 98th percentile)
Foundation LimeExchange Member
rsdsoft
|
|
rsdsoft
From Russian Federation
Offline
|
Foundation EUFreelance.com Member
gangabass
|
gangabass
From Russian Federation
Gold Member
Offline
|
Foundation EUFreelance.com Member
camMcKinnon
|
camMcKinnon
From Canada
Offline
|
Foundation LimeExchange Member
Foundation EUFreelance.com Member
vibhub
|
|
vibhub
From India
Offline
|
Foundation EUFreelance.com Member
Foundation TycoonTalk.Freelancer.com Member
aashnaa1
|
|
aashnaa1
From India
Offline
|
Foundation EUFreelance.com Member
techpista
|
techpista
From India
Offline
|
Foundation LimeExchange Member
Foundation EUFreelance.com Member
