Web Scraping into Database

We want to build a tool that will scrape several websites on a regular basis. Initially, we want to do this to collect all of the information we can. Subsequent scrapes will be to look for changes/updates (i.e. new images, new prices, new products added, products deleted). Some websites will be suppliers, others will be competitors. The information collected needs to be stored in a database.

We already had some work done to scrape one supplier's site. I can give you access to the scraping code and the database.

Once the data is collected, we need to come up with a way to easily use it. The first major challenge will be aligning the products from the different websites so that we can accurately compare information.

If would be fantastic if much of this "matching" was done programmatically. Once we are certain the product information collected from each of the websites "matches," we can compare the products. We can then determine what our competition is selling the item for, and compare it to our price.

Another example is product description and images. With 7000 products, it is very time consuming to enter a meaningful product description and up-to-date image for every product. I'd like the ability to use the information we scrape to populate our store's database. For example, use this description for this product.

Once the data collection is done, the manipulation/use of the data will evolve. This may be best done on an hourly basis. If possible, I'd like to know the cost to set up each scrape on our server and get the information into a database.

NOTE: If it is not clear, our end goal is to data mine. We want to know what our competition is doing. If you have experience manipulating and using the collected information, please let us know.

The websites are

[url removed, login to view] (supplier/competitor)

[url removed, login to view] (supplier/competitor)

[url removed, login to view] (competitor)

[url removed, login to view] (competitor)

[url removed, login to view] (competitor)

[url removed, login to view] (competitor)

[url removed, login to view] (competitor)

Please look at the attached excel file for the list of fields to be scraped

we do not want to run these scrapes from our home computers. We will either run them from one of our (Linux based) servers, or we will use a Linux based cloud solution. Is your application windows based, or do you have a version that will run on Linux? While I am personally a Windows guy, our server resources are Linux based.

Skills: MySQL, PHP, Software Architecture, Web Scraping

See more: attached list websites scrape, web scraping price, web d cost, software scraping, software competition, software challenge, scraping prices from websites, scraping data from web database, scraping a server, product supplier database, is php web scraping, famous smoke, excel challenge, challenge websites, challenge web, challenge tool, challenge software, best web scraping software, best web scraping, best tool for web scraping, best architecture websites, what is data scraping, Web Scraping Software , web scraping products, web scraping from websites

About the Employer:
( 1 review ) Chevy Chase, United States

Project ID: #4740461

20 freelancers are bidding on average $267 for this job


Seasoned web scraper. I worked on many similar projects, I have big experience in data mining projects. I can finish this task in short time, with the best quality.

$350 USD in 3 days
(29 Reviews)

Hi, This is the bid for only one scraper, example : I will write a PHP/MySQL solution that will run on any Linux server. If you are interested .. I'm almost always on chat :) Regards More

$222 USD in 5 days
(193 Reviews)

Hello, I can help you. I have done thousands of crawlers and has very reliable engine for automation. Please check pmb for more information. Thanks, Steve.

$631 USD in 10 days
(18 Reviews)

Hi there. This bid is to scrape one of the websites you mentioned. I can build this using either a Java application or PHP script which you can run from your linux server or computer. The application will be fast, and More

$222 USD in 5 days
(39 Reviews)

Can be done but need some discussion

$273 USD in 3 days
(44 Reviews)

Thanks for giving your precious time to review my bid and check PM! Thank you

$180 USD in 3 days
(60 Reviews)

Web scrapper is ready to start. Thanks

$169 USD in 3 days
(26 Reviews)

hi sir im an expert in [url removed, login to view] check my message

$250 USD in 6 days
(28 Reviews)

Hi, I have an experience in this webscraping by java and I can do this project for you easily as I already have my own webscraping system.

$333 USD in 10 days
(2 Reviews)

Hello, I can help you, I have done so many scraping projects, check my profile, thanks

$309 USD in 5 days
(17 Reviews)

Dear Sir, We claim to get it done perfectly for you EXACTLY in the way you want it - Kindly give we a chance and we will prove myself - Ready to prove our words, let's get it done right away and I mean RIGHT AWAY !! More

$300 USD in 5 days
(12 Reviews)

Hi, please check PM.

$333 USD in 7 days
(3 Reviews)

>>>>>Genuine and Reliable<<<<<< Let us do this for [url removed, login to view] are up for a demo [url removed, login to view] follow all the instructions. Thanks and Regards :-Karan

$180 USD in 3 days
(12 Reviews)

Please see PMB for detail and our expertise.

$277 USD in 5 days
(10 Reviews)

I did a lot of projects like this ,check pm.

$157 USD in 6 days
(13 Reviews)

Hi, Ready to start now. Thanks

$210 USD in 10 days
(6 Reviews)

You need a PHP script to scrape seven of your competitor's websites.

$321 USD in 7 days
(7 Reviews)

Hello, Please see my PM. Thank you

$280 USD in 3 days
(5 Reviews)

Hello sir, I am kindly interested in this type of project works I will perfectly finish the work within the stipulated time kindly give me an offer to work for you I expect your cordial relationship in the due course More

$164 USD in 2 days
(7 Reviews)

Hello I Can complete this page for you its is very simple. please check my pm for how I would complete this. I am very well versed in linux as I am using it now as my main computer and my main server. I will give you More

$125 USD in 7 days
(1 Review)