Find Jobs
Hire Freelancers

Selenium and Scrapy web scraper doesn't work (score = 0.1)

$30-250 USD

In Progress
Posted almost 5 years ago

$30-250 USD

Paid on delivery
Hello, I own a web scraper in Python 3, which uses Selenium. It connects to good quality private proxy and scrapes the website using Chrome browser, exporting content to HTML files. The problem is the scraper is getting 0.1 score (1.0 is maximum), which is too low and searching on site is being blocked due to unusual traffic. When searching manually without using the scraper, my score is 0.9 (when logged to Gmail account). I assume that the problem is Google's verification system v3 (no images, just score) is detecting the bot. Scraper remembers the login and password to Gmail account, as it should increase the score but Google blocked the try of logging in, due to critical error (unauthorized use of password). Scraper uses chrome driver. 0.5 score would be enough to scrape. I own a source code. If you are interested in fixing this problem, please post an offer. Important: Please describe what is your idea on fixing the problem. If you advice to create totally new scraper, please also describe how it would work. I respect your know-how, but please be precise as possible. Thank you.
Project ID: 19978336

About the project

12 proposals
Remote project
Active 5 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
12 freelancers are bidding on average $151 USD for this job
User Avatar
hi, I have read you description in detail. I have experience with SCRAPY AS WELL AS SELENIUM scrapers in PYTHON. I can look at the code and resolve the issue. ALSO, i believe scrapers made using WEBREQUESTS are faster and more reliable as compare to SELENIUM based scrapers. But lets discuss the details in private chat. Looking forward to your reply.
$155 USD in 3 days
5.0 (349 reviews)
7.5
7.5
User Avatar
Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spiders I worked with 50+ differenct individuals/startups/companies and helped them archive their goals. Here's what you will get by working with me: * Working only on projects that I can deliver 100% * Ridiculous quality of work * Smart suggestions on increasing results/revenue and decreasing costs * Responsibility for tasks/project I work on * Reliability * Short response time * Good sense of humor ;) Hope I could awake you interest in me. Thank you for your attention. Shuaixing.(call me Shuaixing L.)
$200 USD in 7 days
5.0 (34 reviews)
6.8
6.8
User Avatar
Dear Hiring Manager.⭐ I am a python programmer and I have faced many python problems.. Hope you give this project to me.. Relevant Skills and Experience I have worked with python + web development with Django framework and worked with several libraries in python including Selenium, Beautiful Soup, Flask, Django, Scrapy and many more. I have that much experience to complete your project. If you give me an opportunity I assure you that I will give you my best. Thanks . :)
$100 USD in 1 day
5.0 (84 reviews)
6.4
6.4
User Avatar
want to see the source code. can try with proxy rotation and user agent. These are my skills related to web scraping and crawling Have done scraping in CasperJS Phantomjs, python. Have done testing and automation with selenium also. Know to deal with database like mongo, mysql, Elasticsearch. Also know to handle proxy and captcha while scrapping.
$166 USD in 2 days
4.9 (64 reviews)
6.3
6.3
User Avatar
Hi there, I have scraped amazon, aliexpress, yellow pages ,yelp ,zomato etc. I have 500GB internet with 20mbps speed, i have 6 systems I can do it. I have done many related projects like this. If you are provide this work, it will help my career also. Give me a chance to do this. Waiting for your precious reply. Thank you. Please Check my mastery work at:- https://www.freelancer.in/u/Stephenrajs *Why you are choose me 1. 24/7 hours support 2. Quick response 3. Deliver on time 4. Smooth communication. I assure that I can satisfy you completely and want to have a long term relationship with you. Best Regards Stephenrajs
$90 USD in 3 days
4.9 (130 reviews)
6.0
6.0
User Avatar
Hi, Python Ninja here! I have understood your requirement with the Selenium and Scrapy web scraper work, and I am very confident with that. I can make it work for you. I am Usman. I have 6 years of experience in the Web & Mobile App Development Department and Designing. I have reviewed the description and understand it very well. I want to start the project right now, just need to discuss it little to more clear the image of the project. Please come to inbox so that we can lock the scope of the project. I am an expert in my domain areas like PHP, Laravel, CodeIgniter, My SQL, Angular, React, Node, Mongo DB, Ionic, React Native, Android, Java, IOS, Swift & Golang. I have extensive knowledge of Html5, Css3, Javascript, Ajax, jQuery, Bootstrap, and Payment Integration modules as well. I assure you that you will experience the WOW factor in my work. I am looking forward to your response. Many Thanks Usman Adil
$250 USD in 5 days
4.5 (24 reviews)
5.2
5.2
User Avatar
Hey there! Selenium scraper here, I can do any sort of data mining or web scraping that you need to be done in a reasonable amount of time. I have years of Python, HTML, CSS and JavaScript experience under my belt and am ready to apply it to whatever your project requires! Here are some examples of things I've done for clients before: Writing custom website scrapers to get you the data you need! Scraping census data on a large scale and finding correlations in the data Mining hundreds of thousands of tweets and performing sentiment analysis Transforming data into Python data structures like pandas data frames. Creating PDF and document scrapers to help you sift through thousands of documents in minutes and get exactly the information you're after. Creating & managing a Django, Flask, or other pythons web-framework website. E-commerce data harvesting/Scraping bots/scripts development. Web Scraping of any publicly available data. Web Scraping of the business listing. Data collection from any directory/listing. Any type of Web Automation. Kindly reply to me, I am waiting for your kind response Thanks
$120 USD in 3 days
5.0 (10 reviews)
4.6
4.6
User Avatar
The blocking issue can be because of Google ReCaptcha. i faced this problem in when i was try to scrap google search results on Mass amount. i didnt totally bypass it but i was able to avoid it for much longer time using different country domain of google while searching google. it helped me to raise my score from 15 to 70 approximate (max is 100)
$50 USD in 5 days
5.0 (1 review)
0.5
0.5
User Avatar
Hello there, Hope you are doing good...!!! As per your requirement, I am very much suitable for the job kindly invite me for a chat for further discussion regarding the project. I have more than 6 years of experience in web & mobile app development you will get good quality work from me. Thank you
$155 USD in 3 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of JAPAN
Tokyo, Japan
5.0
1
Payment method verified
Member since Apr 18, 2019

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.