[PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 2

I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)


Complete writing cloud function which does the following:

1. Picks up data from pub/sub.

2. Processes the data and compares it to data in another table in BigQuery.

3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'

Please find project details in the attached excel file.

[login to view URL]

1. Data flow -> Pub/sub - Cloud function - Big Query

2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.

Skills: Python, Software Architecture, MySQL, Data Mining, Hadoop

See more: data entry processing officer, excel data entry processing controls, data entrydata processing, data entry processing skills, python script extract data web page, python script crape data, simply data email processing, python read input data file, scarborough data entry processing, python script extract data, python script read data text file, python script extract data website, expert data processing, python data stream processing, gcp pub sub ack, data being published to pub/sub must be sent as a bytestring., data being published to pub sub must be sent as a bytestring, google functions pub sub python

About the Employer:
( 56 reviews ) Siem Reap, Cambodia

Project ID: #30965942

4 freelancers are bidding on average $88 for this job


I have read your job description and assure you that I am a perfect fit for the job for [PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 2 I am well versed in various types of graphic d More

$100 USD in 1 day
(2 Reviews)

Hello, sir. How are you? Thank you for your job posting:<CACA> I am interested in your project. I have read your proposal and I am confident. I am ready to start just now. I have rich experience in python development. More

$75 USD in 7 days
(1 Review)

Hi, Greetings, I have gone through the given job description and I'm very sure that I can do this. last 1 year working in GCP projects. So i can do that. current working project technology stack is NIFI, Pub/sub, cl More

$75 USD in 1 day
(0 Reviews)

Hello there! Good day! I am interested in this project. I can get this done for you at the earliest. I am a Python & GCP expert with more than 7 years of experience. Please reach out to me to discuss further. Thank More

$100 USD in 7 days
(0 Reviews)