Python data grabbing from web pages

Closed

Description

Need to grab data from web pages and insert into local database.

The source website is www.autoscout24.it.

The script shall fetch all pages from the Vehicles Search Engine ([url removed, login to view],U&sort=price&results=80&page=1&event=pag) then, for each vehicle details page it has to scrape all data found and save it into a single CSV file, comma separated, with 1st line containing Headers, so it can be easily imported into an RDBMS.

The CSV must contain the vehicle AD Unique ID, in order to avoid duplication on our database.

Text can be delimited by single (') or double apexes ("), and needs to be correctly escaped accordingly.

Requirements:

- All other ADs must be removed (no Google AdSense ADs or whatever), CSS, must be removed.

- All vehicles details data must be retrieved.

- For each vehicle AD, pictures must be saved, using the AD Unique ID, and saved into a .zip file.

- No duplicate records must exists.

- The fastest the better, obviously. The script is meant to be run on a daily basis, and possibly in the next future multiple times a day. It mustn't generate any memory leak.

- It must provide CLI parameters to select destination directory of the .CSV + .ZIP files, along with the possibility to be extended to directly insert data into a PostgreSQL RDBMS (PGSql variables, libraries, insert/modify/delete functions must be included).

- The script MUST be completely commented with plenty of details in each and every part, in plain standard English language.

Notes: as long as the project is absolutely meant to be 100% functional (I won't pay for anything who doesn't work exactly as described), it serves for an educational purpose, that's the reason for choosing a so big site and the need of complete and detailed comments.

Skills: HTML, HTML5, PostgreSQL, Python, Web Scraping

See more: scrape multiple site search results python, grab data csv python, search for a destination or search the web, memory engine, long web pages, google postgresql, postgresql any, web pages, scrape python, scrape ads, python data, modify a python, grabbing, grab pictures, fetch data, sort csv file python, 100 pages adsense, scrape web ads, insert multiple records, python csv html, web script python, python html script, python scrape web, pag, project need python script

Project ID: #5168607

18 freelancers are bidding on average $263 for this job

SigmaVisual

Dear Client, I can help in your project. We have already experience of working on similar projects. Please see below to get idea of our experience: Amazon/Ebay Bots: http://sigma-dns.sigmavirtual.com/PDemo1/Am More

$206 USD in 5 days
(57 Reviews)
6.7
NTechcorporate

Dear Cleint, Hope you are doing well !!!! Thank you for posting this job, We have gone through your requirement specification and confident to deliver you best "Python" solution as we have expert in-house team of More

$283 USD in 7 days
(4 Reviews)
5.0
tanveerjavaid

Wow scrapping.. I am interested in this project as I am good in scrapping so please response if you are interested.. thanks

$555 USD in 12 days
(8 Reviews)
4.8
mhmhz

Hi I know you asked for Python script. But i am here to offer doing the job as Desktop application in C# as alternative option. If you are interested, i could start preparing a demo for you Thanks

$263 USD in 3 days
(21 Reviews)
4.9
techvolcano

Hi, Ready to work Can you explain what are the fields required to parse? We can use multithreading for this Thanks

$180 USD in 3 days
(25 Reviews)
4.6
Peterpay

i can build this on 2 ways python and some scrapping libraries or nodejs parallel code super light code

$277 USD in 1 day
(10 Reviews)
4.5
marchent

Hi, I am interested to write a Python script for you for this project. Let me know if you are interested. You don't need to pay before. Just create milestone, and release this when you are satisfied with the work More

$250 USD in 10 days
(12 Reviews)
4.0
AstreyLabs

Hello, my name is Maxim. I'm a python developer with 5+ years of experience. My specialization is data mining, scraping and parsing. I've developed a hundreds of crawlers, bots and parsers for amazon, youtube, g More

$333 USD in 3 days
(4 Reviews)
3.8
robinsjp

Hi, I write data scrapers as part of my job. The page you linked to looks fairly straightforward to scrape - but I notice that your specifically asked for a python script. Is python an absolute requirement? I u More

$244 USD in 0 days
(3 Reviews)
3.7
sunnyrpandya

Hello Sir, We have good experience in web scraping. My past work was on "http://www.ae.com/web/international/index.jsp". I have scrape all the information and dump in to DB. For more detail please refer to www.quixo More

$266 USD in 3 days
(2 Reviews)
2.5
Bence4hire

Hi, Scraping expert here. I can create a script for you fitting all your requirements using Python and the powerful Scrapy framework. It can be extended to put results directly into a PostgreSQL database (amongst More

$150 USD in 3 days
(2 Reviews)
2.3
anuyadav1

i am well experienced with scraping data from websites and storing them in local mysql database. i can do this project well.

$190 USD in 3 days
(5 Reviews)
2.3
dibsweb

i want to give you web 2.0 design and seo portable coding after developing seo and mentence free 6 month my best quality design and developing 2013 www.hdri4you.com please check our work example rest More

$360 USD in 13 days
(1 Review)
1.3
brij2103

Hi, I have gone through your requirement.I can develop in PHP/Python My Working style:- -Requirement Analysys -System Design and Database Design -Development by following coding standards like getter,s More

$155 USD in 3 days
(0 Reviews)
0.0
elhossari

A proposal has not yet been provided

$333 USD in 3 days
(0 Reviews)
0.0
useevil

This can be accomplished with by using Python/Scrapy/SQLite. It would be scrapy can be used to scrape the website for data, parse it and put it directly into your RDBMS. Or it can be inserted directly into SQLite dat More

$555 USD in 3 days
(0 Reviews)
0.0
saeedzhd

Bis jetzt wurde noch kein Vorschlag eingegeben

$148 USD in 3 days
(0 Reviews)
0.0
ttakamoto

A proposal has not yet been provided

$222 USD in 3 days
(0 Reviews)
0.0
kamalteju

I have 7+ yrs of exp as programmer in java Can i make this java with some gui ,if you want a demo i'll try to prepare it . Hope for the positive response

$122 USD in 3 days
(0 Reviews)
0.0
alexphisher

As a testament to both my experience and enjoyment of python web scraping I have just finished a project involved scraping a website for sales leads which were then saved in a well formatted SQL database all using pyth More

$333 USD in 5 days
(0 Reviews)
0.0