Closed

ETL/data engineering for Python web app

Looking to have a pipeline to retrieve data via API call from data source, transform the data, and load into BigQuery on GCP.

What we are doing:

Building a web app that combines features (ie. data fields) from many different sources for many different records. For example, there are over 600 records and each record has tens of data fields from various data sources. The data will be combined into a single, rank-ordered list based on user input. For example, there will be a static list of features (ie. data fields) from various sources (eg. US Census, FBI crime data, NOAA weather, etc) and the user will fill out a questionnaire. Based on the input from the questionnaire, the static data will be scored and combined into a single number.

What we need from you:

- Write the code to data from source via API call for each record

- Adjust the data (ie. clean, impute missing, and transform)

- Load the data into BigQuery

This pipeline will be run one time per year. However, the pipeline will need to pull a lot of data (5-15 features for more than 600 records).

This projects is to build a pipeline for a single source. Based on the quality that is delivered, additional pipelines for other sources can be contracted.

The pipeline needs to written in Python and must be deployed on GCP.

Skills: Python, API, Google Cloud Platform, ETL, Software Development

About the Client:
( 3 reviews ) Madison, United States

Project ID: #34044723

57 freelancers are bidding on average $38/hour for this job

(16 Reviews)
7.2
AwaisChaudhry

Hi there! I have read what you exactly need and I have great experience with Software Development, ETL, Google Cloud Platform, Python and API. However I would like to ask you a few questions. I do work smart and do not More

$50 USD / hour
(16 Reviews)
6.8
saubhagyamweb

* * * * * * * * * * * * * * * * * * * * * * Hello Joe , I’m a Professional and Expert Python Developer. I am interested in doing this job for you.I do have the necessary skill set & resources to complete this project. More

$25 USD / hour
(35 Reviews)
6.6
(62 Reviews)
7.0
(90 Reviews)
6.3
(35 Reviews)
6.2
(17 Reviews)
5.8
seekdeveloper

Hi, I have read your post and understood your requirement. Looking for the freelancer to work on your next project? Or just need some issues/bugs/fixes ASAP? I have 11+ years of software experience in fortune 500 com More

$30 USD / hour
(3 Reviews)
6.1
gogora

Hi That is quite easy task for now. For start please specify the APIs (How many there are, what documentation they have etc.). I could deliver a single source pipeline for the start and we can continue from there. Ple More

$30 USD / hour
(5 Reviews)
5.5
(2 Reviews)
5.3
oleksdevr

Hi . I am a Senior Python Data Processing developer. I've completed lots of similar projects using python. I will provide high-quality results in time. I'd like to know more about the project. I can start your project More

$35 USD / hour
(14 Reviews)
5.6
sapnathakur14

Hello, How are you? I read your job post, and I am interested to work with you on this project, as I have relevant skills set, you can check my profile for your surety. Rest we will finalize everything after complete More

$25 USD / hour
(6 Reviews)
5.3
techplusintl

Hi there, ★★★ Python Expert ★★★ 5+ Years of Experience ★★★ I've read requirements and ready to retrieve data via API call from data source, transform the data, and load into BigQuery on GCP.. My skills and services: ✔ More

$40 USD / hour
(18 Reviews)
5.9
letshappy

Hello I understood your requirements but I have a few doubts, text me so I can clear all my doubts right away

$60 USD / hour
(20 Reviews)
5.4
(15 Reviews)
5.4
(4 Reviews)
4.9
ognjenstev00

Hi, there. I am an experienced web developer and have previously worked on several GCP applications, so I will provide you quality work according to your instructions. Please contact me so we can discuss in more detail More

$38 USD / hour
(4 Reviews)
4.9
cassargammer

Hello there, Thanks for your job posting and I am interested in your project. I have an excellent background as a Data Engineering and Data Science Expert, have rich experiences with ETL, ELT in python. I would like t More

$40 USD / hour
(5 Reviews)
4.6
jaxpeng

Hi there, I’m a Professional and Expert Data engineer. I am interested in doing this job for you. I do have the necessary skill set & resources to complete this project. I have extensive experience in selenium and he More

$35 USD / hour
(4 Reviews)
4.2
(1 Review)
4.0