Data Processing Software Architecture


This project is a proof of concept of a modern data processing engine that will bring to ingest, explore, prepare, manage, and serve data for immediate BI and machine learning needs. Currently, it is restricted to personal usage only.


The purpose of the modern data processing engine is to bring modern data warehousing capabilities with reasonable pricing ease.

System Features

• At the moment, the system is powered by a spark engine and other components.

• Functional Requirements: The system connects to the following data sources.

 MS SQL Server

 AWS Redshift

 Azure synapse

 web APIs

 Flat Files

 Snowflake

• The system has query engine capacity using SQL which allows running federated queries on the above-mentioned data sources except for web API.

• The system has integrated data processing functionality that allows users to use Apache Spark to process data from the above-mentioned data sources.

• The system has directed acyclic graph capability using Air flow.

• The system provides a search facility that will allow full text searching of all data in the memory of the engine that the user is permitted to access. The system must support the following searches:

 find all words specified.

 find any word specified.

 find the exact phrase.

• The software is built on a framework or architect which lets it run on distributed systems.

• Secure impersonation inside its execution model for added security and offers data encryption and masking services.

• The system has connectors to share the output to third-party BI tools such as Power BI, or Looker.

Job Requirements

• Experienced architecting and developing software for scalable, distributed systems.

• Understanding the current architecture of the application and assisting in scaling the architecture cost effectively.

• Develop orchestration in the application.

• Enable users to schedule jobs on the basis of time and event.

• Containerization/packaging of the application so the application is not reliant on any local systems.

• Cost analysis of current configuration & architecture of the application.

• Comparative analysis of applications with various configuration with the same architecture.

• Writing efficient and modular code.

• Experience with cloud technologies and distributed system servers.

• Ability to facilitate demonstrations, proof of concepts.

• Deep understanding of spark application.

• Able to expertly convey ideas and concepts to others.

• Understanding of the public cloud market and pain points driving enterprise cloud adoption.

• In-depth understanding and the ability to demonstrate expertise in designing, deploying, and maintaining custom enterprise web applications.

• Prepare a high-level PowerPoint presentation and detailed word document of the application on completion of the project.

Nice to Have

• Strong knowledge of Python Machine Learning standard libraries.

• Strong understanding of all commonly used Machine Learning models and the main algorithms that compose the models.

• Good understanding of the built-in data types. (Lists, dictionaries, tuples sets).

Skills: Database Development, Data Processing, Software Architecture

See more: form collecting data processing software, data processing flow concept data entry, access data processing software, project data processing analysis writting reort software spss, act data processing software, data processing comparison details software buy, data processing software house hyderabad sindh, software architecture data security related projects, data processing template software, mobile/tablet data collection software needed like surveytogo with offline and online data collection purpose, mobiletablet data collection software needed like surveytogo with offline and online data collection purpose, online concept of data processing, electronic data processing software, jullious software company for outsourced data processing, am new member in this but i have 9 years experience in online processing and data processing also i have don data entry project, lidar data processing software, data processing software, data processing software list, what is data processing software, different between word processing and data processing software

About the Employer:
( 0 reviews ) Brampton, Canada

Project ID: #32246717

16 freelancers are bidding on average $2597 for this job


Hi, This is M Zahid I went through the details of your project and would like to offer our services to deliver the results that you expect and we are sure you will not be disappointed. I got interested in your proje More

$2600 CAD in 7 days
(29 Reviews)

Hello, Hope you’re fit and fine during this pandemic era! We have gone through the requirements of yours and love to say that we have already created many customized software solutions for my clients. In fact not onl More

$2600 CAD in 20 days
(13 Reviews)

Hi, We at Tecogno Solutions are a team of Passionate Data Science and Full Stack professionals having more than five years of combined experience in multiple areas including Backend, Frontend, Machine learning (ML), C More

$3000 CAD in 7 days
(1 Review)
(2 Reviews)

This project is a proof of concept of a modern data processing engine that will bring to ingest, explore, prepare, manage, and serve data for immediate BI and machine learning needs. Currently, it is restricted to pers More

$2600 CAD in 7 days
(3 Reviews)

Hello, Greetings! ___Hope you're reading my proposal, will wait for your positive response.___ ___I'm interested in your project.___ ___I can start the work right away & I assure you for the best quality work.___ I'v More

$2600 CAD in 7 days
(1 Review)

Hi, I have very a good understanding of data processing systems, in fact I developed an application which allowed to gather data from 1 million shopify stores, which was used to later to apply machine learning usage. More

$1950 CAD in 7 days
(5 Reviews)

Hi, I’m Senior Data Architect working for Globant company with having many Hadoop ecosystems like HDFS, Spark, Hive, ETL, Azure etc Please contact me for more details

$3000 CAD in 7 days
(5 Reviews)

Hey there ! Good day. I am Pankaj Ranga I would like to work with your project and I can surely provide the details you have specified on the project description. I can confidently say that I am knowledgeable enough to More

$2600 CAD in 7 days
(2 Reviews)

Hi There, I am flexible with my working hours and would appreciate it if you could discuss your project as soon as possible. I would greatly appreciate the opportunity to be working with you and to discuss my qualifi More

$2600 CAD in 7 days
(1 Review)

Hello Greetings for the day, Am a professional freelancer expertized in various MSBI ,Power platform, Python and Azure data tools , with hands-on experience around 5 years in different segments of business, I will pro More

$2600 CAD in 7 days
(3 Reviews)

Hi there, I am a Senior Software Engineer, having 9+ years of top experience in .NET Web, SaaS, Database, Migration, API, Library, Services, Reports, Dashboards. I can analyze, design, develop and manage small to larg More

$3000 CAD in 30 days
(2 Reviews)

Hello I will do and i am not zero profile my old profile has bin close and i have a Big Company my company Name Webblu Technologies [login to view URL] I am read your project description carefully, and i am expert and fastly wor More

$2600 CAD in 36 days
(0 Reviews)

Hello there, Hope you are doing well! I have 5+ years of strong experience as Python Developer. I have expertise with Django, Flask, Python, AI, ML, NLP and Data Science. Can we have a quick chat/call to discuss thi More

$2600 CAD in 7 days
(0 Reviews)

Hi, Thanks for the detailed requirements We have a hands-on experience in the ML Domain and we can develop this software for you. Here is one of our ML-based project: -[login to view URL] I can share my portfo More

$2600 CAD in 7 days
(0 Reviews)

Hello there , i am a full stack software developer having 5+ years experience , i have experience in developing data processing softwares , database softwares, APK softwares , Mobile apps and website , i have complet More

$2000 CAD in 7 days
(0 Reviews)