Etl jobs

I'm looking for an experienced professional who can assist me with some technical challenges I'm facing. AWS - I need help primarily with S3 Storage, Lambda Functions, and Redshift. Python - My main requirement is support with Data Engineering tasks. PySpark - I need assistance with ETL Processes, Data Cleaning, and Data Integration. Ideal candidates would have extensive experience with AWS, Python, and PySpark, particularly in a Data Engineering context.

Data Integration Design and Implementation: Design and implement data integration solutions using SAP SDI for real-time and batch data replication. Configure and manage SDI adapters for various data sources (e.g., databases, files, applications). Develop and optimize data flows for data extraction, transformation, and loading (ETL/ELT). Implement data virtualization and remote data access using SDI. Ensure data quality and consistency during data integration processes. SAP HANA Integration: Integrate data from diverse source systems into SAP HANA using SDI. Optimize data loading and performance for SAP HANA-based applications. Configure and manage remote sources and virtual tables in SAP HANA. Cloud Integration: Integrate on-premise and cloud-based data sources using SAP SDI. Des...

...data in real-time, and apply predictive modelling techniques. The ideal candidate should possess: • Experience in data ingestion, ETL pipelines, and normalization for large datasets • Proficiency in Python (Pandas, NumPy, PySpark), SQL, and cloud-based data processing (AWS) • Familiarity with handling rolling weekly updates and CIDR-level data processing • Strong knowledge of anomaly detection, confidence scoring, and statistical filtering (e.g., exponential smoothing, rolling averages) • Experience with predictive modelling Your primary responsibilities will include: • Developing a product capable of processing and analyzing geospatial data • Implementing ETL processes for large datasets • Utilizing AWS for cloud-based data proces...

I'm seeking an experienced Data Engineer trainer for my Ed-Tech firm to provide comprehensive training on ...training on advanced Data Engineering concepts. The course will be completely virtual and is expected to last up to 35 hours. Key Training Areas: - Data Engineering Tools: The training should cover the use of key industry-standard tools including Apache Spark, Hadoop, and AWS Data Pipeline. - Specific Data Engineering Topics: The course should delve into critical areas such as Data Warehousing, ETL Processes, and Data Modeling. The commercial aspects can be discussed after a detailed conversation about the course content and deliverables. The ideal candidate should have extensive experience in data engineering and a proven track record in training professionals at an ...

...data integration, extensive ETL processes and the preparation of Standard Operating Procedures (SOPs). Start proposal with IBPI. The ideal candidate will have solid experience with SQL and API connections, as well as a knack for crafting advanced, interactive data visualizations. Key Responsibilities: - Develop and maintain interactive Power BI dashboards. - Integrate various data sources including SQL databases, APIs and Excel spreadsheets. - Conduct comprehensive ETL processes to ensure data accuracy and relevance. - Prepare detailed SOPs for dashboard usage and maintenance. Ideal Skills and Experience: - Extensive experience with Power BI - Proficiency in SQL and API connections. - Strong background in dataflows - Excellent understanding of comprehensive ETL pr...

...solutions using Power BI. - Create interactive dashboards, reports, and visualizations with Power BI to meet business needs. - *Data Modeling:* - Develop and maintain complex data models in Power BI, ensuring high performance and scalability. - Perform data transformations using Power Query (M) and DAX (Data Analysis Expressions) for reporting and analysis. - *ETL Process:* - Design and implement ETL (Extract, Transform, Load) processes to bring in data from different sources (databases, flat files, APIs, etc.). - Ensure data quality and integrity in reports and visualizations. - *Collaboration:* - Work with cross-functional teams to identify key metrics and KPIs. - Provide insights and recommendations based on data analysis and visualizations. ...

...initiative aims to reduce unnecessary doctor visits and ensure timely access to trustworthy health guidance. Prediction of various diseases as per symptoms provided by users along with recommendation of doctors for the same disease is also implemented to ease the medical consultation for the user. 2. Developed a real-time weather analysis stream using Hive, PySpark, Kafka and AirFlow. Developed ETL pipelines for data ingestion, transformation and storage, enabling real-time streaming and automated workflows for accurate forecasting. 3. Developed a machine learning model to predict loan approval based on applicant details such as income, credit history, and loan amount. Utilized classification algorithms to achieve accurate predictions. Performed data pre processing, fea...

Project ID: OF1283 Job Title: Sisense Data Developer – Dashboard Development, SQL, ETL Work Schedule: Days: Monday to Friday Hours: 3 hours/day Timings: 8 PM to 12 AM IST Location: Remote via Zoom (Screen share and remote control required) Job Description: We are seeking a Sisense Data Developer with expertise in dashboard development, data modeling, SQL optimization, and ETL pipelines. The ideal candidate will work on designing, developing, and optimizing Sisense dashboards while ensuring data integrity, performance, and scalability. As a part-time developer, you will collaborate with one of our clients who is new to their role and requires daily support. You will connect with the client via Zoom (screen sharing required) to assist them in completing their assigne...

I am seeking a seasoned Python developer with significant experience in Google Cloud Platform (GCP), particularly with Cloud Storage and Ai...(GCP), particularly with Cloud Storage and Airflow. The primary focus of this project is the creation and management of data pipelines. Key Responsibilities: - Develop and manage robust data pipelines using Python and Airflow - Execute intricate ETL (Extract, Transform, Load) processes - Conduct thorough data cleaning - Implement effective data integration techniques Ideal Skills: - Proficient in Python programming - Extensive experience with GCP, specifically Cloud Storage - Familiar with Airflow for orchestrating complex workflows - Skilled in ETL processes, data cleaning and integration - Capable of developing scalable and efficie...

...goal is to visualize data in Power BI. Key components of the project: - DuckDB Work: You will need to integrate and work with DuckDB based on the outline and proposal I have prepared. - ETL Pipeline: A crucial part of the project is setting up an ETL (Extract, Transform, Load) pipeline. This pipeline should be designed in such a way that it can seamlessly feed data into Power BI for visualization. - Power BI: Familiarity with Power BI is necessary as the end goal is to visualize the data using this platform. Ideal Skills and Experience: - Proficiency in DuckDB and Spark - Extensive experience in creating ETL pipelines - Familiarity with Power BI - Ability to follow project outlines and proposals meticulously Please note that I have not provided specific details a...

...proficiency for querying and optimizing large datasets. • Experience integrating data from platforms like Google Analytics 4, HubSpot, BigQuery, CRM systems, and paid media (Google Ads, Facebook Ads). • Proficiency with Google Tag Manager (GTM) for event tracking and data collection. • Familiarity with BigQuery and other cloud-based data warehousing solutions. • Solid understanding of data modeling, ETL processes, and API integrations. • Ability to create automated reports and alerts for key metrics. • Experience managing Looker user permissions, caching, and performance tuning. Preferred Skills: • Experience with additional BI tools (Google Data Studio, Power BI, Tableau). • Familiarity with Python or R for advanced data analysis....

...pipelines for structured, semi-structured, and unstructured data. The ideal candidate will have extensive experience in MongoDB Atlas, ETL processes, database architecture, distributed data processing, and cloud platforms (AWS, Azure, GCP). This role is a full-time freelance position, requiring 8-hour workdays as part of a Scrum team, with occasional meetings in Pacific Time but most work done in IST. Key Responsibilities: Data Pipeline & Architecture: Design and architect scalable data pipelines for various data types. Develop, manage, and optimize databases, including RDMS (MySQL, PostgreSQL), NoSQL (MongoDB), and data lakes (S3). Implement efficient ETL processes using PySpark and Hadoop to transform and prepare data for analytics and AI use cases. Optimize ...

UPDATE 2: I will add a flow diagram, data in/report out and examples so the estimates can have more meaning. But it will be tomorrow. UPDATE: The data I have is backup data from an on...PostgreSQL database on AWS for data warehousing purposes. The project involves: - Building the AWS PostgreSQL database, management tools, and running database. - Integrating the database with CSV/Excel files as data sources. - Assisting with recommendations for suitable AWS services for data integration and ETL tasks. Ideal skills and experience for the job include: - Proficiency in AWS and PostgreSQL. - Prior experience with data warehousing. - Strong skills in data integration and ETL. - Expertise in working with CSV/Excel files. Please, share your previous relevant experience when bi...

.../ Data Analyst Location: Remote Employment Type: Overview: We are seeking a Business Analyst / Data Analyst who specializes in data manipulation, ETL processes, data quality control (QC), and dataset preparation for analytic reporting. The ideal candidate will have experience working with business intelligence tools and ensuring the right data is structured to enable accurate and effective reporting. Some SQL knowledge is required, and familiarity with AI-driven analytics is a plus. Key Responsibilities: Work closely with business stakeholders to understand reporting requirements and ensure the right data is available for analysis. Perform ETL (Extract, Transform, Load) operations to cleanse, merge, and organize data for reporting and analytics. Conduct data quality contr...

We are looking for our first full-stack engineer who enjoys a fast-paced work environment and...what’s needed to quickly deliver them to production. • You should have 4+ years of Fullstack Engineering Experience: You have a strong background in front & back-end engineering, with a proven track record of delivering high-quality software solutions. • We are looking for a systems thinker who can easily recognize patterns to develop frameworks and solutions to solve complex needs. Familiar with ETL, Queues. Understands data ingestion concepts. • Proficiency in Python and Typescript: You have at least 4 years of experience working with one or more of these languages, enabling you to contribute effectively to our tech stack. Bonus points if you have • Experi...

I need a professional to create an architecture diagram for an Enterprise Data Warehouse (EDW) solut...should encompass the following components: - Data Sources: Including Relational Databases and APIs. - Data Warehouse: A clear representation of the central hub for our data. - ETL Processes: Illustrating the Extract, Transform, Load processes. The specific relational databases that will need to be integrated into the diagram include PostgreSQL, Oracle, SQL Server, and Db2. Ideal skills for this job include: - Strong understanding of data warehousing concepts. - Proficiency in creating clear and comprehensive architecture diagrams. - Experience with PostgreSQL, Oracle, SQL Server, and Db2. - Ability to represent ETL processes accurately. - Familiarity with relational da...

I'm looking for an expert in Apache NiFi to construct an ETL pipeline for my SQL Server data. Key Requirements: - Data Extraction from SQL Server: The first step involves extracting data from my SQL Server database. - Data Transformation: This includes various tasks such as data cleaning, data aggregation, and data format conversion. - Data Loading: The final step is to load the transformed data back into SQL Server. Ideal Skills and Experience: - Proficient in Apache NiFi: Your expertise in this tool is crucial for the successful completion of the project. - SQL Server Knowledge: A strong understanding of SQL Server is necessary, as both extraction and loading will be done from/to this database. - Data Transformation Skills: Experience in data cleaning, aggregation, and for...

I am seeking a SSIS data engineer with expertise in SQL Server Integration Services (SSIS) to enhance ETL processes, Query Optimization and Performance Tuning. This project primarily focuses on educating an individual to handle various data sources in real-time in a crash course format (1 Day or Continue 4-5 Hours Depends, Open for Discussion). The ideal candidate will have a strong industry background with some teaching experience in data migration, transformation, and ETL system optimization. Key Responsibilities: - Analyze and improve the current ETL processes. - Implement real-time ETL processes. - Work with diverse data sources including SQL Server, E-Commerce platforms, and social media platforms. Platform: SSIS Server Studio Ideal Skills and Experience...

We are seeking highly skilled databricks data engineer who have experience in building data pipelines and data modelling in delta should have a deep understanding of Databricks advanced features, including Unity Catalog, Delta Lake optimizations, storage efficiency techniques, and security best practices. Key Responsibilities: Design and develop scalable, high-performance ETL/ELT pipelines using Databricks and Apache Spark. Implement Delta Lake best practices, including schema evolution, vacuum, and optimization for efficient data storage and query performance. Manage security, access control, and data governance using Unity Catalog and other Databricks-native security features. Optimize query performance by leveraging Z-Order indexing, data skipping, caching, and clustering

...experience as a Python Developer At least 3 years of experience in data analytics Strong proficiency in Python programming language Experience with data manipulation and analysis libraries such as Pandas, NumPy, and Scikit-learn Familiarity with data visualization tools like Matplotlib, Seaborn, or Plotly Knowledge of SQL and experience with relational databases Experience with data integration and ETL processes Strong problem-solving skills and attention to detail Excellent communication and teamwork abilities Bachelor's degree in Computer Science, Engineering, or a related field. Preferred Qualifications: Experience with cloud platforms like AWS, Azure, or GCP Knowledge of machine learning algorithms and frameworks Familiarity with version control systems such as Git P...

I'm looking for a small Python script that will scrape an e-commerce website for product details and prices. Key Requirements: - The script should be adept at web scraping, collecting data on various products and their prices from the sp...their prices from the specified e-commerce site. - It should be able to handle potential hurdles such as CAPTCHAs and IP bans without interruption. Ideal Skills: - Proficiency in Python, especially with libraries like Beautiful Soup, Scrapy, or Selenium. - Previous experience with web scraping, particularly from e-commerce websites. - Understanding of data extraction, transformation and loading (ETL) processes. Please note that the script's ultimate aim is to automate the task of collecting this data, making it easier for me to condu...

...applications will not be considered** **We are not working with agencies for this role. Only direct applicants will be reviewed** Work Schedule: Days: Monday to Friday Hours: 3 hours/day Timings: 8 PM to 12 AM IST Location: Remote via Zoom (Screen share and remote control required) Salary: ₹35,000 per month (INR) Job Description: We are seeking an AWS Data Engineer with expertise in DevOps, ETL, Cognos Reporting, and Cloud Infrastructure to support a Contract Management System for Medicare Advantage (MA) Divesture Tranche 2. The project involves duplicating the existing commercial contract management system and setting up reporting, data pipelines, and AWS infrastructure. As a part-time developer, you will collaborate with one of our clients who is new to their role and requ...

I'm in need of an Azure Data Factory (ADF) pipeline creation specifically for an ETL process. Key Responsibilities: - Design and implement an ADF pipeline to manage the ETL process. - Source data is currently stored in Azure Blob Storage, so familiarity with this specific cloud service is crucial. Ideal Skills: - Extensive experience with Azure Data Factory. - Proficient in designing ETL processes. - Knowledgeable in working with Azure Blob Storage. - Strong problem-solving skills and attention to detail.

Backend Developer Stack: ETL, Python, JavaScript, MongoDB, PostgreSQL, Databricks, AWS The ideal candidate will have a strong background in building scalable backend systems, working with ETL processes, and have expertise in Python, JavaScript, and modern cloud technologies such as AWS and Databricks. You will play a crucial role in enhancing our data pipelines, optimizing databases, and building powerful backend solutions that empower our data-driven applications. Key Responsibilities: Design, develop, and maintain scalable backend services and APIs. Build and optimize ETL pipelines to extract, transform, and load data efficiently. Collaborate with cross-functional teams, including data engineers and data scientists, to integrate backend solutions with data proc...

Need resource who matches on below JD and ready to work on monthly basis. ONLY MATCHES CANDIDATE WILL APPLY Please find the below JD. We have open positions for Snowflake Skills: - Hands on with Snowflake - Good knowledge of Salesfo... - Expert at SQL and complex query building. Good to have Skills - Good experience with AWS Cloud Services eg. Lambda , S3, Step functions - Building ETLs using any other tool. - Python - Power BI - Migration Project Experience. Roles and Responsibilities: - As snowflake / dataware house developer person is expected to build ETL - Data Transformation would be key part of the work - Communicate for requirements and demonstrations of compl...

i need to complete these data analyst/data engineering tasks using python, bigQuery and Tableau. Will send the brief found help

I am in need of an experienced consultant who can assist with the migration from SQL server to Postgres. The project will involve: - Data extraction and transformation - Schema conversion - Data validation and integrity checks The preferred tool for this migration is Talend Open Studio, an open-source ETL tool. Therefore, expertise and experience with this specific tool will be highly advantageous. The ideal candidate will have a strong background in ETL processes, with a focus on utilizing open-source tools. A keen attention to detail is necessary for performing data validation and ensuring the integrity of the data throughout the migration process.

...visualize sourcing risks and compliance status. ✅ Ensure data security & access control so brand owners see all data, but traders can only access their own. ✅ Automate compliance reporting (EUDR, CSDDD, CSRD). ? Required Skills & Tech Stack ? Backend: Python (FastAPI/Django) or Node.js ? Frontend: React / Vue.js / Angular ? Databases: PostgreSQL, MongoDB (for flexible storage & analytics) ? APIs & ETL Pipelines: REST, GraphQL, Apache Airflow ? Geospatial Tools: Google Earth Engine, PostGIS, Sentinel-2 ? Security & Access Control: OAuth 2.0, Role-Based Access (RBAC), GDPR compliance ? Cloud Infrastructure: AWS, GCP, or Azure ? Nice to Have ✔️ Experience in agriculture, supply chain, or traceability systems ✔️ Knowledge of EUDR, CSDDD, CSRD compliance requirem...

I'm seeking a seasoned BigQuery professional with deep expertise in SQL and ETL processes, specifically tailored to the healthcare domain. Your primary task will be to assist in data analysis on complex datasets. Key Responsibilities: - Developing and implementing intricate SQL queries to mine and interpret healthcare data. - Crafting ETL processes to ensure smooth data flow and transformation. Ideal Skills: - Profound knowledge in SQL and BigQuery. - Extensive experience in ETL process development. - Prior work in the healthcare industry is a strong plus. - Strong analytical skills to assist with data interpretation. Your role is crucial in enabling comprehensive data analysis, thereby driving informed decision-making. If you're a problem solver with a kn...

I need assistance with a Tableau ETL project to take file from ADP ETL into format for NetSuite processing, primarily focused on data transformation and cleaning. Key Requirements: - Expertise in handling and processing flat files, specifically CSV and Excel - Proficiency in Tableau ETL for data transformation - Skills in data filtering, sorting, aggregating, summarizing, merging and joining The ideal candidate would have a strong background in data analysis and visualization, with a proven track record of using Tableau ETL for similar projects.

I am in need of an experienced ETL engineer who can extract customer data from our MongoDB and present it on our website through interactive dashboards using Google Looker. Key Requirements: - Extraction of relevant customer data such as demographics, purchase history, and behavioral insights. - Creation of interactive dashboards that allow users to filter data and drill-down into specifics. Ideal Skills and Experience: - Proficiency with ETL processes and MongoDB. - Experience with Google Looker or similar data visualization tools. - Ability to create user-friendly, interactive dashboards. Please provide examples of previous work with customer data and Google Looker in your proposal.

Hi All, We need below resource who matches our JD and budget. JOB DESCRIPTION: What you will do: Process and manage complex datasets using Python, SQL, and Snowflake, ensuring accurate and efficient ETL workflows on Azure infrastructure. Automate and optimize data workflows, developing tools to empower analytics and engineering teams. Apply advanced data techniques such as machine learning and statistical analysis to drive business insights. Build interactive dashboards and visualizations in Power BI to support decision-making processes. Collaborate with stakeholders and IT teams to address data-related challenges and ensure scalable solutions. What We're Looking For Minimum 3 years of experience in a similar role (Data Engineer, Data Scientist or BI). Strong knowledg...

Overview We are looking for a Certified Azure Databricks Engineer to provide ongoing support, troubleshooting, and optimization of our Azure-based big data environment. The ideal candidate must have both Azure and Databricks certifications, demonstrating deep expertise in data engineering, ETL processes, and performance tuning on Databricks. We are offering two engagement options: 1️⃣ Hourly-Based: $32 per hour, capped at $3,000 per month (Daily timesheets required, no fixed contract). 2️⃣ Retainer-Based: $3,000 per month, with a Monday to Friday commitment (Fixed 3-month agreement, extendable). Mandatory Requirements ✔ Azure Certification (e.g., Azure Data Engineer Associate, Azure Solutions Architect, or equivalent) ✔ Databricks Certification (e.g., Databricks Certified...

We are seeking a skilled freelancer to connect Close CRM with Big Query and Meta using Airbyte. The ideal candidate will have experience in data integration and ETL processes to build and manage tables that facilitate seamless data flow. Your expertise will help us leverage our CRM data for deeper insights and analytics. If you have a strong background in data engineering and are familiar with these tools, we would love to hear from you.

I am in need of an experienced ETL engineer who can extract customer data from our MongoDB and present it on our website through interactive dashboards using Google Looker. Key Requirements: - Extraction of relevant customer data such as demographics, purchase history, and behavioral insights. - Creation of interactive dashboards that allow users to filter data and drill-down into specifics. Ideal Skills and Experience: - Proficiency with ETL processes and MongoDB. - Experience with Google Looker or similar data visualization tools. - Ability to create user-friendly, interactive dashboards. Please provide examples of previous work with customer data and Google Looker in your proposal.

...Responsibilities Design and implement efficient data collection solutions Create robust data models using Prisma Optimize application performance and reliability Implement proper error handling and monitoring Write clean, maintainable, and well-documented code Provide regular progress updates What Sets You Apart Strong analytical and problem-solving abilities Experience with data processing and ETL workflows History of delivering independent projects with minimal oversight Deep understanding of JavaScript best practices Ability to optimize application performance Working Environment Independent project work with engineering support Code review process ensuring quality standards Freedom to propose and implement optimal technical solutions Clear project scope and deliverables ...

I'm looking for an experienced Data Engineer to assist with various tasks related to data processing and analysis. Ideal Skills and Experience: - Extensive knowledge of data engineering principles - Proficient in data pipeline setup, data warehousing, and data transformation/ETL - Experience with relational databases, APIs, and CSV/Excel files - Familiarity with tools such as Apache Hadoop, Amazon Redshift, and Google BigQuery Please note that specific tasks may vary and will be discussed further. Your expertise and flexibility in handling different aspects of data engineering will be key to the success of this project.

...Architectural Planning: Design a scalable, secure, and efficient data architecture with Snowflake as the core repository. Develop detailed technical documentation outlining integration points and data flows across Epiko Hub modules. Data Ingestion & Processing: Set up and configure Snowpipe for real-time data ingestion from game servers, blockchain APIs, and external marketplaces. Develop and optimize ETL pipelines using Snowpark for processing streaming data and generating live analytics. Implement data storage strategies using Hybrid Tables for both real-time and historical data. Analytics and AI Integration: Integrate Snowflake Cortex to enable AI/ML models for personalized insights, churn prediction, and reward optimization. Ensure processed data feeds into real-time da...

...Architectural Planning: Design a scalable, secure, and efficient data architecture with Snowflake as the core repository. Develop detailed technical documentation outlining integration points and data flows across Epiko Hub modules. Data Ingestion & Processing: Set up and configure Snowpipe for real-time data ingestion from game servers, blockchain APIs, and external marketplaces. Develop and optimize ETL pipelines using Snowpark for processing streaming data and generating live analytics. Implement data storage strategies using Hybrid Tables for both real-time and historical data. Analytics and AI Integration: Integrate Snowflake Cortex to enable AI/ML models for personalized insights, churn prediction, and reward optimization. Ensure processed data feeds into real-time da...

...Architectural Planning: Design a scalable, secure, and efficient data architecture with Snowflake as the core repository. Develop detailed technical documentation outlining integration points and data flows across Epiko Hub modules. Data Ingestion & Processing: Set up and configure Snowpipe for real-time data ingestion from game servers, blockchain APIs, and external marketplaces. Develop and optimize ETL pipelines using Snowpark for processing streaming data and generating live analytics. Implement data storage strategies using Hybrid Tables for both real-time and historical data. Analytics and AI Integration: Integrate Snowflake Cortex to enable AI/ML models for personalized insights, churn prediction, and reward optimization. Ensure processed data feeds into real-time da...

I am looking for a seasoned data professional with expertise in GCP, Scala, BigQuery, Airflow, and DB...professional with expertise in GCP, Scala, BigQuery, Airflow, and DBT. The primary focus of this project will be on data processing, specifically in the realm of data transformation and enrichment. The tasks involved will mainly revolve around: - Transforming raw data into a more usable format - Enriching datasets to provide more insights and value The ideal candidate will have: - Proven experience with ETL operations and data processing - Proficiency in using tools such as GCP, BigQuery, and DBT - Strong skills in Scala - Familiarity with Airflow for pipeline automation I am looking for someone who can efficiently handle data processing tasks and deliver high-quality, enriched...

Project Name: Data Engineer | ETL Pipelines & Sentiment Analysis for Crypto Project Description: We are looking for a Data Engineer to develop ETL pipelines and build a multi-language news scraper for sentiment analysis in the crypto market. The ideal candidate should have experience handling large datasets, optimizing storage solutions, and implementing efficient data pipelines. Key Responsibilities: • Create ETL pipelines for historical OHLCV + social media data. • Develop a multi-language news scraper (20+ languages). • Optimize vector database storage for efficient retrieval. • Implement real-time streaming data processing. Tech Stack: • Data Processing: Apache Airflow, Apache Kafka, Spark • Database: PostgreSQL, Pinecone ...

...Looker) Predictive Analytics and Machine Learning Insights Data Warehousing and ETL Processes Performance Metrics and KPI Development 4. Target Market Retail Sector: E-commerce businesses, brick-and-mortar stores, supply chain companies. Finance Sector: investment firms, fintech startups, insurance broker firms. 5. Competitive Advantage Industry-specific BI solutions tailored for retail and finance. Expertise in leading BI tools and cloud-based analytics platforms. Strong focus on actionable insights that drive business growth. Scalable solutions adaptable to various business sizes and needs. 6. Technology Stack BI Tools: Power BI, Tableau, Looker Data Processing: SQL, Python, R Cloud Platforms: Google Cloud, AWS, Azure ETL Solutions: Talend, Apache NiFi, Alteryx 7. Marke...

I'm working on an assignment related to data warehouses and Apache NiFi. I'm specifically looking for guidance on the ETL process. My ideal tutor would have significant experience with Apache NiFi and the ETL process in data warehousing. Key Requirements: - In-depth understanding of Apache NiFi and its application in data extraction. - Solid background in data warehousing and ETL processes. - Excellent teaching and communication skills to guide me through the assignment.

I'm in need of an Azure Data Engineer with substantial experience working with SAP HANA as a data source in Azure Data Factory. The primary focus of this project is to use the ODBC connector in ADF to read the source data. I look forward to your bids. Please ensure you meet the requirements before applying. The expected duration of the project is less than 1 day. The project requires basic ETL transformations.

... disaster recovery and business continuity strategies: • Acts on Disaster recovery tests to process fail over and fail back. • Interventions scheduled on weekends and non-working hours. Profile Skills required: • Technical Skills for Production Support Engineer • Linux REDhat since 6.2 version mandatory, CentOS • Scripting KSH Skill and SSH • Windows servers since 2012 version • Experience on ETL tools (Informatica) • Experience on Databases (Oracle, Teradata, PostgreSql) • Experience on BI Tools (MSBI, Business Object, Power BI) • Experience on Files Transfer Tool (Tom, SFTP,) • Experience on scheduling tools (Control-M, Jenskins) • Experience on automation (Jenskins, Ansible, Nexus , Terraform) • Experience on n...

I am looking for a skilled professional who can assist with using Python and Terraform. The project involves automating infrastructure on the cloud, specific...cloud-based infrastructure using Terraform - Process data using Python - Ensure efficient and streamlined operations Ideal Skills: - Strong proficiency in Python - Extensive experience with Terraform - Familiarity with AWS - Background in data processing Please bid if you have the relevant experience and skills. The expected timeline for this project is less than 1 month. The project will involve ETL (Extract, Transform, Load) processes for data handling. The expected volume of data to be processed is small. Provide regular progress updates during the automation process. Provide daily progress updates during the automatio...

...robust data pipelines for structured, semi-structured, and unstructured data. The ideal candidate will have extensive experience in ETL processes, database architecture, distributed data processing, and cloud platforms (AWS, Azure, GCP). This role is a full-time freelance position, requiring 8-hour workdays as part of a Scrum team, with occasional meetings in Pacific Time but most work done in IST. Key Responsibilities: Data Pipeline & Architecture: Design and architect scalable data pipelines for various data types. Develop, manage, and optimize databases, including RDMS (MySQL, PostgreSQL), NoSQL (MongoDB), and data lakes (S3). Implement efficient ETL processes using PySpark and Hadoop to transform and prepare data for analytics and AI use cases. Optimize dat...

...pull data from multiple sources, including different projects, documents, cloud storage services (Dropbox, S3), URLs, and media content (YouTube videos). The agent will process, clean, and store the extracted data for further analysis, search, and retrieval. 2. Objectives Develop an AI-driven system to aggregate data from various sources. Enable automated extraction, transformation, and loading (ETL) processes. Ensure seamless integration with cloud storage, document repositories, and media platforms. Implement Natural Language Processing (NLP) for document comprehension and YouTube transcription. Provide an API interface for querying and accessing the processed data. Support integration with various Large Language Models (LLMs), allowing flexibility in model selection. 3...

I'm embarking on my first IT project and I need a seasoned data engineer to assist me. The project revolves around extensive data engineering tasks with a focus on DBT and Snowflake data ware...Snowflake for Data Warehousing - Implementing ELT strategies - Utilizing DBT for data transformations - Overseeing the entire Data Warehouse (DWH) setup Data Type: - Primarily dealing with transactional data The ideal candidate would have: - Extensive experience with DBT and Snowflake - Deep understanding of data engineering principles - Proven track record in setting up databases and creating ETL pipelines - Skills in optimizing data queries - Ability to work with transactional data Please let me know if you're the right fit for this project. Thank you! The expected project tim...