In Progress

Automated Web Login, scraping, data storage, and on-demand filtered output to google sheets

I'm a private individual looking to automate the collection of my personal wearable sensor data. This data is logged automatically by the wearable device to the company's website, and then they have a dashboard, behind a login, where the data is available as JSON format.

This project will consist of:

1. Web scraping the data after logging in

2. Storing the data in a database long term

3. retrieving the data by date range, reformatted, on demand, via all three output methods below:

1. To google sheets

2. To CSV Text

3. In JSON

Code/script/app hosting/execution:

I would like advice on how to automate this on a daily scheduled basis with existing hosted providers, such as Integromat, Zapier, Apify, Google app script, Google sheets or similar, that provide free access for private hobbyists.

I do not have a server to host any code on, so if would like to demonstrate a solution that is not integrated in the previously mentioned type of automation platform (apify, zapier, etc) (i.e. requires running scripts locally), please make sure to show how to run it scheduled in a free (or less than USD 5$/month) commercial online service that has point-and-click web user interface for scheduling. (if proposing use of a VPS, must not use Windows)

You must demonstrate that the web login and scraping actions will function (not be blocked by the host as robot or scraper) while it is running on the platform that you suggest.

Code Ownership:

Provide all source code with an MIT style software license ( [login to view URL] ) , or we can do the Intellectual Property Agreement

Please indicate in your proposal the following things:

* Where you would propose to host such a tool (multiple ideas ok)

* What language(s) you suggest that it be written in (multiple ideas ok)

* What kind of database

* How you would prefer to interface with Google Sheets (custom macro, Google Apps Script, using a third party service that already provides API authentication with Google Oauth2 like Apify, etc.)

* Have you written a regularly executing web scraping LOGIN script that ran on a commercial hosting provider that didn’t get blocked by the hosts as a robot?


These importing and processing tasks should happen automatically, once daily.

1. Web scraping:

1. Login to: https: //[login to view URL] using given credentials

2. Request the given date range of data

3. Receive the JSON data result

1. Data storage:

1. De-duplicate any data that is already present in the database

2. Store in online database

1. Retrieve data on demand:

1. Data output formats:

1. Direct populate a google sheet/cells

1. Sheet that function is called from, or via some other web API site such as Apify, zapier, integromat etc that can access google sheets.

1. CSV text


1. User Interface

1. Adjusting settings at this phase of the project should be limited to simple OPTIONAL variable inputs loaded at runtime

1. Possible examples:

1. passing arguments to a function in a function call: functionname(20190321,12,5)

2. Reading from a preferences file

3. Reading from a set of google sheet cells

1. Input variables. Variables should be optional, and if not provided, default to the value provided here:

1. Starting and ending dates for retrieved data

1. Format: (same as imported date/time format)

1. YYYY-MM-dd HH:mm:ss

2. Example: 2019-03-17 00:17:32

1. Default to the current date in the US/Los Angeles time zone. (Not the current date GMT)

1. Error handling:

1. Return numbered english language error messages

1. Unparsable date/time range

2. Other errors as needed

3. Use your best judgement on the format of the machine readable error messages.

1. Log errors to file or database

Skills: Data Processing, HTTP, JSON, RESTful, Web Scraping

See more: parsehub, extract data from website to excel, octoparse, chrome web scraper multiple pages, google scraper software, go to https chrome google com webstore detail web scraper jnhgnonknehpejjnehehllkliplmbmhn hl en, web scraper tool free download, web scraper, automated web login data extraction, automated web data, scraping data web page excel, excel retrieve data web login, dynamic data web login, extract data web login, scraping data web match word, scraping data web pagejob, web scraping data collection website, automated web scraping, web scraping data facebook, web scraping data extraction

About the Employer:
( 1 review ) Walnut, United States

Project ID: #19030216

Awarded to:


Hello sir, I am a Python developer with huge experience in Web Scraping. For the presented project, first I took a look at the HTTP response from healbe. It looks easy to automate a login and retrieve data, I've don More

$250 USD in 3 days
(3 Reviews)

20 freelancers are bidding on average $470 for this job


Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. More

$250 USD in 5 days
(188 Reviews)

Hi there, I am interested in your project. I would approach your project by using Python. I have 4+ years’ experience at Web scraping. If you'd like to view my previous work, take a look at my profile [login to view URL] More

$1987 USD in 3 days
(48 Reviews)

Hello, I am experienced bot developer and for years I've been automating websites that requires login and similar. I've read the project description and I like it because you clearly know what you want. So, More

$1222 USD in 10 days
(19 Reviews)

Dear Customer! I have built many scrapers. This is my favorite type of job. I am absolutely confident I can complete your project easily and on time. Ready to begin and finish as soon as possible. Please contact to dis More

$250 USD in 3 days
(33 Reviews)

hi there , i have scrap with script which site have login let me know more detail like url credential i didn't find it . waiting for response regards saleem

$250 USD in 3 days
(3 Reviews)

I am a python developer. I have great experience in web scraping and I am an expert in it. I have all necessary skills by which I can scrape any website. I have even scraped sites like google, whatsapp web, etc. which More

$140 USD in 2 days
(10 Reviews)

Hello there, As far as I understood, you want to scrape the data from the website you mentioned with your login credentials, and then store them in a database for later use. And you want this website to be scraped d More

$250 USD in 3 days
(2 Reviews)

Hello, I can get your data scraped either on a dedicated server and then have it available on cloud web site free tier. How long do you intend the lifecycle to be? Have a look in my profile. i am a great python More

$100 USD in 3 days
(1 Review)

Hello I have experience with web scraping. I can create the system to scrape the site using login and upload data to required places. The solutions I have in mind is either curl + php, curl + python, or python/php + More

$100 USD in 3 days
(7 Reviews)

Hello, my name is Vladimir and i am Web Developer from Serbia. I already made some scrapers that is output in Excel file, for example Instagram Locations [login to view URL] getting usernames that posted picture More

$250 USD in 3 days
(1 Review)

I am confident I am the right candidate for this project as I have done many similar projects in the past. With years of experience in this field, I believe this project will be very easy for me.

$491 USD in 5 days
(2 Reviews)

Hi, I have done similar jobs and I will finish in less than two days. I'm online now, you can chat me up. Regards. I will be using python. prosgresql or any available database. I have written scriot that interact wit More

$333 USD in 4 days
(0 Reviews)

Hello, I can work with this, i am work with RPA and UiPath tool, i have 2 years to experience with RPA and 8 year with software testing. I share this video, maybe help you to know my job. In this video, you More

$155 USD in 10 days
(0 Reviews)

hi, i will create this automation using uipath the automation tool. you don't need any server to host it . and it is code free so that you don't face any issues or bugs. please contact us for more details. As a rpa More

$155 USD in 3 days
(0 Reviews)

Hi there! My name is Orestes and I am a Software Engineer with specialty in Java/Spring. I represent a team of Software Engineers (so our 'technology' stack is actually quite big: Java, C#, Javascript, Python among ot More

$846 USD in 20 days
(0 Reviews)

"Hi, Hope you are doing well! Thanks for sharing your project requirement with us. It will be our great pleasure to work on your project. I have checked your requirement, yes we can do it, because we already work on si More

$208 USD in 7 days
(0 Reviews)

Hello, Thanks for taking the time to read this proposal. I'm a software developer with 4 years of experience. In the past few years, I've built dozens of scraping applications. Because of your extensive job descrip More

$666 USD in 14 days
(0 Reviews)

Hello, Interested in the project. I am a Software Engineer specializing in end-to-end Web Development with expertise building applications that leverage Google's Cloud Platform (aka GCP) and Google's GSuite servi More

$1250 USD in 28 days
(0 Reviews)

Hello there, I am a computer engineer with nearly 8 years of software development experience, mainly in .net technologies. I have worked in quite good companies and have developed lots of automated tests with with us More

$240 USD in 3 days
(0 Reviews)