Azure Data Factory +Synapse Analytics End to End ETL project
Azure Data Factory +Synapse Analytics End to End ETL project
Complete end to end Project on OTT platform using Azure Data Factory and Azure Synapse Analytics from Scratch with Labs
Created by Shanmukh Sattiraju | 5 hours on-demand video course
Are you looking to build an end-to-end ETL project using Azure data engineering services? Look no further than this comprehensive course on Azure Data Factory and Azure Synapse Analytics. This course is designed to guide you through the process of creating all the necessary services from scratch, before building an ETL project from start to finish.
Throughout this course, you’ll gain practical, hands-on experience with Azure Data Factory and Azure Synapse Analytics, learning how to use these powerful data engineering tools to create a highly effective ETL solution. You’ll explore the many features and capabilities of these platforms, as well as their integration with other Azure services like
- Azure SQL Database
- Azure Synapse Analytics
- Azure Key Vault
- Azure Data Factory for Orchestration,
- Azure Storage solutions (Azure Datalake Gen2)
- Microsoft Power BI
- Azure Logic Apps
In addition to the core ETL project, this course also includes an additional section on CICD (Continuous Integration and Continuous Deployment) on Azure Data Factory, helping you to fully automate your data engineering workflow.
Whether you’re a beginner or an experienced data engineer, this course is designed to help you gain a comprehensive understanding of Azure Data Factory and Azure Synapse Analytics. By the end of the course, you’ll be able to confidently create and manage your own ETL projects on the Azure cloud.
What you’ll learn
- Create incremental pipeline based on last modified date
- Calling Azure Synapse notebook from Azure Data Factory
- Configure CICD from Scratch in Azure Data Factory
- Build end to end project using Azure Data Engineering Services
- Create a Self Hosted Integration Runtime and ingest data from On-premise Environment
- How to transform data with Azure Synapse Analytics
- Create and configure an Azure Data Lake
- Project Oriented Learning of Azure Data Engineering services
- Create an Azure Key vault service
- Using Secrets of Key vault in Linked services
- Create and configure an Azure SQL Database
- Writing PySpark transformation logic in Synapse Notebook
- Use of Triggers in Azure Data Factory
- Create an incremental data load pipeline to ingest file in daily basis
- Build an Automated pipeline in Azure Data Factory
- Setup CICD environment from Scratch
- Create incremental pipeline based on file name and pick today’s file
- Perform Data Analysis Using PySpark Code using Synapse Notebook
- Create and configure an Azure Data Factory
- Create and configure an Azure Synapse Analytics
- Orchestrate all pipelines in Azure Data Factory
- Loading Data into Azure SQL Database
- Ingest Data from On-premise to Azure Datalake in Azure Cloud
- Send an Automatic Alert email when a pipeline is failed
- Transform only Today’s file in Azure Synapse Analytics
- Report transformed data in Power BI
- Complete hands-on project using Azure Data Engineering Services
- Provides insights on services that needed to clear DP-203
- Understand how GetMetaData, ForEach, If Condition Activities works
Recommended Course
