Find Jobs
Hire Freelancers

Web scraping with Python -- 2

$250-750 USD

Closed
Posted over 7 years ago

$250-750 USD

Paid on delivery
Web scraping with Python Hi all, Introduction: I am an intermediate python programmer and recently got hit with many HTTP errors(most of them are 503) on Amazon.co.uk. I think [login to view URL] recently up their method of banning scraping. I need a professional programmer to write new codes that can continuously scrape the site without all the proxies being banned. Preferred Candidates: Programmer EXPERIENCED in data mining with [login to view URL] Requirements: -Language Required: Python 2.7.11- Mechanize, working in IDLE mode. -Continuous scraping of Amazon links @ rate of at least 5 pages/s with (~300) rotating proxies. -Rotating proxies, useragents, threading or concurrent or whatever spider method to achieve the required scrape rate -Overcome captchas -Output chunks into a text file. - 1 year support that the code will work with any new proxy banning method from Amazon. -IMPORTANT: In order to be consider successfully completed this project and the money to be released, the code MUST be able the continuously scrape for 1 week without all ~300 proxies being banned. Links example: "[login to view URL]%s" %ISBN "[login to view URL]%s/ref=olpOffersSuppressed?ie=UTF8&overridePriceSuppression=1" %ISBN "[login to view URL]%s/ref=olp_f_used?ie=UTF8&f_new=true&f_used=true&f_usedAcceptable=true&f_usedGood=true&f_usedLikeNew=true&f_usedVeryGood=true" %ISBN ISBNs= 0321103289 0551025794 0471721573 0534377297 1740595815 019541926X 0582089433 0812042689 0299052400 .... Useful hints: - I use my same code to scrape [login to view URL] and [login to view URL], [login to view URL] still works fine, only [login to view URL] had the 503 problem. - Up-to-date common useragents and captcha solvers were used in my code. - [login to view URL] usually starts banning the proxy after 1 hour scraping, and then it will ban ANY combination of proxies, useragent, scrape speed that tries to scrape the similar links. - The banning of a proxy is usually not permanent, it usually gets released after 24 hrs. Thank you!
Project ID: 11272352

About the project

16 proposals
Remote project
Active 7 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
16 freelancers are bidding on average $407 USD for this job
User Avatar
Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database or excel or csv or xml file. I worked on many similar projects, I have big experience in data mining projects. I have written hundreds of web scrapers which scrape millions of pages each day. I'm ready to fulfill your requirement. I can finish this task in short time, with the best quality. I can assure 100% accuracy. Please give me the opportunity to do the work. With Kind Regards, Debdulal Roy Proshanta
$400 USD in 10 days
4.9 (46 reviews)
6.6
6.6
User Avatar
I am an expert in scraping and can deliver the project to your specifications and requirements asap .
$333 USD in 10 days
4.9 (73 reviews)
6.8
6.8
User Avatar
Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi
$411 USD in 6 days
4.9 (118 reviews)
6.7
6.7
User Avatar
Hello Sir, We've done a number of web scraping projects for our clients. We have scraped many directory websites including yellowpages, yelp and e-commerce websites including amazon, walmart etc and many more. We can deliver the data very quickly. We use proxies with IP rotation to avoid being detected as bots. We use python with wget, scrapy, urllib and other tools to fetch webpages and parsers like HtmlXPathSelector, regular expressions etc to extract information from the html. We have the right skill set to do this job effectively and within time and would like to discuss more about this opportunity. Looking forward to hear from you. Thanks, Shiv Agrawal SuiGen Solutions
$526 USD in 10 days
4.7 (51 reviews)
6.1
6.1
User Avatar
Dear Sir/Ma'am, I am a Web research, Data Entry & Webs Scrapping expert. I checked and understood your requirements. I can handle this job very well to your appreciation. I can find and extract the information from different websites into an Excel sheet. I am ready to hear the details of the project more in detail now. I have always created a long-term collaboration with my clients through hard work and quality output for a reasonable price. If you have questions or doubts about anything, please feel free to ask me. Sincerely, Mir
$526 USD in 10 days
4.9 (30 reviews)
5.4
5.4
User Avatar
Hello, I can manage this web scraping task of required data as per your requirement. Let's discuss further. - I can manage lead generation, web researching and lead list building - I can manage data submission in Back end of websites as well as other online directores - I can manage online data collection & data mining - I can manage calling projects & have also worked on few of cold calling tasks for countries, USA, Canada, Australia and New Zealand - I can manage online/offline data entry tasks - I can manage WORD/EXCEL/ACCESS tasks efficiently thanks.... Rajni
$300 USD in 4 days
4.7 (22 reviews)
4.5
4.5
User Avatar
Hi, i am scraping expert please give me any kind of scraping task i ensure i give you good job i am waiting your response,thanks
$250 USD in 10 days
4.8 (35 reviews)
4.5
4.5
User Avatar
rich experience in web scraping with Python, had crawled many sites such as Twitter, iTunes app store, sina weibo, etc.
$333 USD in 7 days
4.9 (4 reviews)
2.6
2.6
User Avatar
Happy to help
$250 USD in 3 days
5.0 (2 reviews)
2.2
2.2
User Avatar
Hi, this is Anshuman. I love scraping, crawling and getting data from various dom structures. I have 6 yrs of experience in scraping, crawling, processing and mining data. My previous projects include- 1. Automated Crawling of Google for SEO Keywords 2. E-Commerce Crawling. Crawling websites like Amazon, Ebay,Alibaba etc. 3. E-mail list scraping and phone number scraping for targeted users 4. Scraping Data from within Android Apps 5. Dynamic Data crawling through JS Manipulation 6. Automated Form Filling and Scraping 7. Proxy Emulation and Authentication in order to prevent server blocking 8. Mobile Site emulation and crawling mobile site specifically 9. Scraping data from Desktop Apps, PDFs etc. 10. Artificial Intelligence to emulate human behaviour while crawling and scraping sites Programming Languages I can use - Python, PHP, NodeJS, Jquery and Rails. Frameworks I have been using - Python Scrapy, Apache Nutch, Selenium, DOM Manipulation using Chrome Extensions, URLLib2, Python Requests, PHP Syphony etc. Data Can be exported to- Excel Files, MySQL, MongoDB, CouchDB, Cassendra, Redis, Docx. File, Amazon s3, HDFS, Oracle, MSSQL etc. I have read your project description and I think I would be the right person to do your project. I will ensure great communication throughout the project with timely updates about the progress of the project. I would like to have a chat with you about the project and discuss more about how I will be approaching this task. Ping me
$250 USD in 10 days
5.0 (1 review)
0.8
0.8
User Avatar
Dear Client, Thanks for providing us opportunity to place bid over the project and communicate with you. I am a serious bidder here and i have already worked on a similar project before and can deliver as u have mentioned .I have checked your requirements.i have right skills to work on this assignment Here we would like to bring in your notice that We are a team of professionals including experienced analysts, designers, project managers, developers and QA people having great expertise in web applications development mainly on core PHP, PHP with open sources (Joomla, Wordpress, Codeigniter, Cake PHP), .NET, Asp.NET, Vb.NET, HTML 5 etc. and mobile applications on ios and Android platform. We find our expertise, skills and capabilities are perfectly blends with your project requirements because we have already worked on many projects. I am ready to discuss with you Looking forward to hear from you. with best Regards
$555 USD in 10 days
0.0 (0 reviews)
0.0
0.0
User Avatar
I'm a full stack developer with Python experience currently working at one of the largest tech companies in the world. This project looks interesting, and I am currently working under my true hourly rate in an effort to build my Freelance reputation and experience. Please contact me if you need any questions answered
$555 USD in 10 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hello, Sites like amazon provide API to get search results, otherwise the application will banned and it may be more complexe to develop in future if the the owner of the site tighten the security. In the past I have worked in similar project, giving the application the UPC code, and the app return the information of the product. if you are really interested in this app, I can continue in it's development, last time I checked amazon updated it's API so some changes must be applied.
$530 USD in 10 days
0.0 (0 reviews)
0.0
0.0
User Avatar
================== Amazon MWS API Experts ================== NOTE: Most of the requirement of your project scope is already completed by us and we have demo for you as well. We are Amazon MWS API experts and completed so many projects using its API I have ready to use API for -- 1)Amazon Orders of seller 2)Amazon Product API 3)Amazon Price API 4)Amazon Repricer 5)Amazon SES API 6)Amazon SQS API 7)Amazon Product Advertising API I have done so many complex projects based on Amazon MWS API and i am sure your project would be very easy for me. I have demos ready along with me, ping me so that i can share the links of demos with you. Thanks
$773 USD in 20 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of CANADA
Calgary, Canada
5.0
1
Payment method verified
Member since Feb 14, 2013

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.