Web Scraping To DB With Python or Perl

This is a web scraping project that can be done in either python or perl.

I am developing a social shopping comparison site, and have authorisation from the

sites we are scraping to crawl and extract data daily.

Scraping needs to be setup for 10 ecommerce sites, for example Firebox .com . About 10,000

urls in total as each site has about 1000 products.

The data that needs to be extracted includes:

Prod Page Url

Product Name

Product Price

Stock status

Image url1

This data needs to be stored in a database that will be hosted on Amazon AWS. The image also needs to be

downloaded and renamed and stored on our local server, and the local image path also added to the DB.

There can be a very basic scraping UI that we can set the interval for new scrapes.

It should also be easy to setup new sites to scrape, but we can also pay you to add these if needed.

In case needed you should be familiar with routing the scraper through Tor.

After urls are scraped, additional social media data needs to be added to each url in the db, for example for Facebook,

with this JSON API:

Facebook*: [url removed, login to view]

In addition to designing this scrape program, I will require ongoing support and maintenance of the code, and have

many such similar projects in mind that I would like to work on with the successful applicant.

I want to work with someone who is responsive via Skype and can get back to me very quickly regarding issues.

PLEASE NOTE: Please apply stating your experience in web scraping, and also your advice on whether to use

python or perl, and also your experience of Tor.

If you cannot be available to discuss issues on skype text chat, please do not bid.

Skills: C++ Programming, Perl, Python, Software Architecture, Web Scraping

See more: perl python web scraping, web sites of web developing, web scraping price, web scraping for responsive, web scraping c#, web scraping api, web developing projects, web developing program, web designing code in php, use case includes, use case amazon, ui status, social media issues, social media designing, site web responsive, scraping com, scraping a server, python media server, projects in web designing, perl get

About the Employer:
( 6 reviews ) london, United Kingdom

Project ID: #2467936

24 freelancers are bidding on average $1035 for this job


I have 7+ years of web scraping experience so I'm sure you'll be satisfied with my work. See PM for details.

$1000 USD in 5 days
(215 Reviews)

check the PMB please, thanks.

$1500 USD in 10 days
(61 Reviews)

Hi sir, please check PM, thx Kimi.

$1440 USD in 6 days
(68 Reviews)

Hello. I am an experienced web developer and web automation specialist with python. I can help you with this project. Check pmb for further questions and details. Regards, -Arthur.

$950 USD in 15 days
(50 Reviews)

I worked on many similar projects, I have big experience in data mining projects. I can finish this task in short time, with the best quality.

$999 USD in 15 days
(17 Reviews)

Hi, I'm an expert in web scraping, especially by Python and Java. Please kindly check your inbox and let me handle your project.

$999 USD in 15 days
(53 Reviews)

Greetings! We have done a few price comparison websites for our clients. Regular communication on poject issues/updates is assured. I suggest building your project in PHP if that is acceptable to you. I would like to s More

$1400 USD in 30 days
(3 Reviews)

I have experience with scraping.

$750 USD in 5 days
(1 Review)

Hi, I suggest Perl solution for your project. More in PM

$1100 USD in 21 days
(27 Reviews)

Hello. I have two years of experience with Python, and the last couple of projects I've done have involved web scraping. Almost every project I've done on this website has dealt with JSON. Because of how easy JSON inte More

$1000 USD in 14 days
(4 Reviews)

Hello I'm python developer. My specialization is data mining and scraping. I will be happy to help you.

$800 USD in 12 days
(3 Reviews)

I can do this job in a simple script. Perl or python does not matter much but I advise perl.

$750 USD in 2 days
(2 Reviews)

Hi, I am an expertise in this area and want to work on this. Please contact for further details.

$750 USD in 5 days
(1 Review)

Hi. Im ready to start

$750 USD in 15 days
(1 Review)

Hello Sir, I am interested to do this project in 'Perl' Please check PMB.

$1000 USD in 15 days
(2 Reviews)

Read your details and interested to work, thanks.

$1000 USD in 20 days
(2 Reviews)

I'm quite intrigued by your project as we have done similar tasks in the past, and I'm interested in learning further details. Please review your inbox for further details.

$1300 USD in 10 days
(1 Review)

We have web scrapping experience and we provide the solution.

$750 USD in 5 days
(2 Reviews)

Custom software development (<b><i>Removed by Admin</i></b>)

$1500 USD in 1 day
(0 Reviews)

I can provide the solution in python. Please check PMB for details.

$750 USD in 15 days
(0 Reviews)