Connect existing PHP scraper script to MySQL database on Magento website and create an ADMIN page to run the script and perform updates -- 3

Closed Posted Apr 14, 2014 Paid on delivery
Closed Paid on delivery

Hi all,

What you need to do:

1) Connect an existing PHP script (product scraper, JSON based) to our MySQL database. At present, the script is meant to scrape all product data in full.

2) Update the script to be run as either:

a) in full and scrape all products and their info, or

b) only as an update (i.e. do not scrape all product info but only scrape price changes and stock level changes or only add products to our database that do not exist yet)

3) Enable multi threading so that the script runs a lot faster. Use of proxies is preferred. There are over 100,000 products that we are trying to scrape so we need to bring the running time of the scraper down to under 2 days.

4) Enable the optional use of proxies (we can use hidemyass for the proxy list or another list based on your suggestions)

5) Allow us to archive products that are no longer in the data feed being scraped (i.e. the item has been removed from the supplier website). So we need to archive it in our database (do not delete it as we still need a record of it), so that clients do not buy the item anymore. One idea is to remove the product from the front end and move the product into an ARCHIVE sub-category.

6) Update the script to also scrape URL links to:

a) product descriptions

b) product specifications, and

c) other product tabs shown in the main product view page (there can be 2-5 tabs per item)

7) Create an ADMIN page in our Magento store where:

a) we can run the script to scrape all product details in full (like what it is designed to do now)

b) we can run the script only to perform product updates (i.e. scrape only products that are not in the database, update prices and stock levels for existing products in our database + anything else you might suggest). The idea here is to run the script a lot faster.

c) we can switch between hardcoded photos and descriptions for products or URL links for selected products (ie. by brand, by category). So we either load the descriptions and photos from our own server or we point to them on the B&H site.

8) Install any additional PHP modules on our server if need be to make the above work.

9) The script needs to scrape data from www.bhphotovideo.com.

Follow up scraping projects will be posted for:

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

Do this project well and you can guarantee yourself a continuation of work.

OPTIONAL:

Feel free to write your own scraper if you do not want to use our existing one.

What we have:

1) Magento online store, running CE 1.8.1

2) MySQL database

3) CentOS dedicated server will full root, SSH and cPanel access

4) PHP script that scrapes products from a 3rd party website

Who you are:

1> Good feedback on Freelancer (do not bid if you have 0 feedback)

2> Good project completion on Freelancer (must be above 65%)

3> Know web scraping, PHP, MySQL, Apache, CentOS and Magento.

START DATE

The project needs to be started ASAP

DEADLINE

3 weeks

Please note: If the project is not completed in the above timeframe, we reserve the right to remove you from the project and choose another developer. You agree that no money will be paid to you unless the project is fully completed and we are happy with the result. Do not bid if you are not able to focus on this project 100%.

MILESTONE RELEASES

Milestones will be released within 24 hours of successful UAT completion.

Please ask any questions before bidding. If you bid a certain amount, do not try to revise it later on.

Good luck!

Apache Magento MySQL PHP Web Scraping

Project ID: #5803653

About the project

20 proposals Remote project Active May 21, 2014

20 freelancers are bidding on average $455 for this job

mantislin

Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi

$250 AUD in 5 days
(197 Reviews)
7.0
evidcomm

Hello Hiring manager, Greetings from Evidcomm!!! Happy to have your attention on our proposal.. We have gone through your requirements and would be pleased to work on.I have read your all project detail and as More

$250 AUD in 6 days
(58 Reviews)
6.5
VnBestSolutions

Dear Sir, We claim to get it done perfectly for you EXACTLY in the way you want it - Kindly give we a chance and we will prove myself - Ready to prove our words, let's get it done right away and I mean RIGHT AWAY !! More

$568 AUD in 10 days
(70 Reviews)
5.9
gopisoft

hi we have an excellent experience with web scraping and designing skills and we are available to start work immediately and have completed over 700 projects at freelancer.com thanks gopisoft pvt ltd

$250 AUD in 16 days
(54 Reviews)
5.5
prashushinde9

Hi, I (Myself a senior PHP, ASP.NET programmer using C# and VB) have a team having experience of more than 10 years in developing web and mobile applications. I have a team of expert programmers who write robust and bu More

$263 AUD in 10 days
(13 Reviews)
5.2
marketingmindz

A proposal has not yet been provided

$147 AUD in 3 days
(12 Reviews)
5.1
indofxlancer

A proposal has not yet been provided

$155 AUD in 5 days
(3 Reviews)
4.5
Time2win

Hello, We have excellent team of programmers and designers to work on your project efficiently and complete job in time. We have read your deepest requirement at our best and will surely give better results. thanks

$263 AUD in 10 days
(14 Reviews)
4.4
MagentoAus

Dear Friend, Get your work done right and even the way you are expecting. Please let's start. Thanks

$163 AUD in 3 days
(9 Reviews)
4.1