In Progress

Web Crawler (Spider) Data into MySQL database

DESCRIPTION:

A Data Procurement team will be responsible for creating spiders to crawl various websites which provide structured data. In the attachment we provide a short listing of sites from where we want to obtain information. (Note that this list may be updated before the start of the job.) The data will be imported into a MySQL database; the database schema is being finalized. This team will also be responsible for creating scripts which will execute the spiders in a batch process on a periodic basis. Therefore, scripts should be written to handle the update or deletion of existing records or creation of new records.

OTHER TEAMS:

The Web development team will use PHP, JavaScript (AJAX), CSS, DHTML, and XHTML and any other technologies as needed. Rudimentary HTML pages with CSS and JavaScript have been created to mock-up the business process flow. Additionally, a majority of the Use Cases and User Interface documents have been completed for each of the features enumerated in the attached listing.

The Java development team will work on a Java midlet which will exchange information with the MySQL databases; this will be accomplished by parsing information via either XML (KXML) or pass variables via the POST command. (Servlets would be ideal but the need to expedite the development process to meet the deadline rules this approach out.) The midlet will be communicating using GPRS (SMS will be an option in future releases). As such, a representative from the Web Development team will liaise with the Java developers to provide any PHP code they might require; also, the liaison needs to coordinate the access to the MySQL database to ensure the schemas remain consistent.

*************** NOTE ***************

PRICING:

Please provide a customized quote. The price range is the estimate to complete your team task only. If you believe that your team can provide additional services, provide a quote for each individual task as well as for the aggregate work you believe you can complete.

Refer to other projects.

NO GENERIC RESPONSES:

Generic quotes and responses will be ignored and deleted. Comment on your experience and where you might be able to assist. Successful candidates will pose relevant and appropriate questions and provide detailed quotes. Specify in your quote the number of individuals who you expect will be working on the project and their roles.

DEADLINE:

End of November

*************************************

ADDITIONAL OPPORTUNITIES:

When the prototype is completed, there may be additional opportunities for extending the functionality in future releases.

ADDITIONAL REQUIREMENTS:

1 - Once a project starts, you must provide daily updates and must be available on a ad-hoc basis for conference calls via Skype or a toll-number to the United States.

2 - Each payment will be contingent upon meeting milestones throughout the project (i.e., delivering working code).

3 - The final payment will be made before the end of the entire project dependent on meeting all requirements and conclusion of testing.

4 - Any delays in the project timeline must be agreed by both parties.

5 - You must include 4 months' of assistance and upgrades on a weekly basis.

6 - Finalization of the contract will be dependent on progress during the first week, to ensure a mutual fit.

PAYMENT:

Payment will be made every 2 weeks.

Skills: Data Processing, Link Building, Perl, PHP, Script Install

See more: web crawler database, crawler database, crawler mysql, database web crawler, mysql crawler, spider crawler database, web crawler using mysql, webcrawler mysql, web crawler mysql, webcrawl database, mysql spider, php web crawler spider, mysql web crawl data, crawl data mysql, database crawler mysql, data crawler database getafreelancer, php web crawler database, php web crawler, mysql spider crawler, webcrawler php mysql, web crawler spider, spider procurement, xml aggregate, working web crawler, working first data

About the Employer:
( 5 reviews ) Wilmington, United States

Project ID: #32612

Awarded to:

neon

we have done similar work before and we can do this as per your requirements. could you provide us with the details on the website which needs to be scraped or extracted.

$120 USD in 20 days
(6 Reviews)
3.6

7 freelancers are bidding on average $234 for this job

cstl

Chandusoft is a customer-specific service oriented company has got an Professional and creative team with 6 years experience in Web design and development. We have expertise and experience in OS commerce, mambo, zen ca More

$270 USD in 15 days
(22 Reviews)
6.9
fnasim8192

Please see PMB.

$300 USD in 10 days
(1 Review)
2.8
rajesht

Hi, Please see PMB for details. -Rajesh

$300 USD in 30 days
(1 Review)
2.6
vasanthegde

Hi, For this kind of projects we use a simple method of web crawling using Mercury tools. Due to the existing tool we can easily create specific scripts for any website and export the data into text/csv or mysql dat More

$100 USD in 10 days
(1 Review)
1.1
enggdatasoft

Dear sir, We are a data mining and extraction expert and we are applyting dor the data expert team. We have understood your project requirement and they can be met. But your budget is low.This project requir More

$300 USD in 300 days
(0 Reviews)
0.0
EmpEroR3

See PMB, pls

$250 USD in 1 day
(0 Reviews)
0.0