Masterclass on Azure Data Factory Data Engineer for 2024

Masterclass on Azure Data Factory – Data Engineer for 2024, available at $69.99, has an average rating of 4.54, with 71 lectures, based on 228 reviews, and has 5274 subscribers.

You will learn about You will learn how to build a real time data pipeline in Azure Data Factory (ADF). You will learn how to transform data using Data Flows in Azure Data Factory (ADF) and load into ADLS2, Blob storage. You will learn how to ingest JSON data from SQL Server to API end point. You will learn how to build production ready pipelines and good practices and naming standards. You will learn how to monitor pipelines using Azure Data Factory (ADF), Azure Monitor and Log Analytics with a real-world project. You will learn about manual/Triggers in Azure Data Factory (ADF) and how to use them to schedule the data pipelines. Create Data driven, fully integrated, dynamic and automated production grade pipeline creation and orchestration. Data Warehousing Concepts like Fact & dimension tables, SCD type1, type2, Incremental loading and implement using ADF. Connect & copy data from On Premise data stores, On premise SQL server via Self Hosted Integration Runtime to cloud. How to ingest data from sources such as REST API, Azure Blob Storage, SQL DB into Azure Data Lake Gen2 using Azure Data Factory (ADF) Learn Azure Data Factory(ADF) with real time projects – ADF, SQL Server, Blob Storage, Datalake G1,G2, REST API. This course is ideal for individuals who are For all levels. or Students, Working Professionals It is particularly useful for For all levels. or Students, Working Professionals.

Enroll now: Masterclass on Azure Data Factory – Data Engineer for 2024

Summary

Title: Masterclass on Azure Data Factory – Data Engineer for 2024

Price: $69.99

Average Rating: 4.54

Number of Lectures: 71

Number of Published Lectures: 69

Number of Curriculum Items: 71

Number of Published Curriculum Objects: 69

Original Price: ?6,500

Quality Status: approved

Status: Live

What You Will Learn

You will learn how to build a real time data pipeline in Azure Data Factory (ADF).

You will learn how to transform data using Data Flows in Azure Data Factory (ADF) and load into ADLS2, Blob storage.

You will learn how to ingest JSON data from SQL Server to API end point.

You will learn how to build production ready pipelines and good practices and naming standards.

You will learn how to monitor pipelines using Azure Data Factory (ADF), Azure Monitor and Log Analytics with a real-world project.

You will learn about manual/Triggers in Azure Data Factory (ADF) and how to use them to schedule the data pipelines.

Create Data driven, fully integrated, dynamic and automated production grade pipeline creation and orchestration.

Data Warehousing Concepts like Fact & dimension tables, SCD type1, type2, Incremental loading and implement using ADF.

Connect & copy data from On Premise data stores, On premise SQL server via Self Hosted Integration Runtime to cloud.

How to ingest data from sources such as REST API, Azure Blob Storage, SQL DB into Azure Data Lake Gen2 using Azure Data Factory (ADF)

Learn Azure Data Factory(ADF) with real time projects – ADF, SQL Server, Blob Storage, Datalake G1,G2, REST API.

Who Should Attend

For all levels.

Students, Working Professionals

Target Audiences

For all levels.

Students, Working Professionals

Welcome to the Course “Azure Data Factory – Data Engineer : Real Time Projects“.

Learn ETL and Big Data Processing on Azure with Azure Data Factory V2

Azure Data Factory (ADF) is one of the in-demand data engineering tool in the Cloud. This course has been taught with covering a data engineering solutions using ADF for real world scenarios.

Once you complete this course including assignments, quizzes, you will be in a position to take up or handle confidently any kind of projects one equal to 10+ years of experience.

Entire course built based on real time scenarios which are going to use very frequently and common to any type of projects on various domains.

This course will gives you necessary skills and help you greatly to pass for Certification exams like DP200 and DP203.

What this course will teach you?

You will learn how to build a real time data pipeline in Azure Data Factory (ADF).

You will learn how to transform data using Data Flows in Azure Data Factory (ADF) and load into ADLS2, Blob storage.

You will learn how to ingest JSON data from SQL Server to API end point.

You will learn how to build production ready pipelines and good practices and naming standards.

You will learn how to monitor pipelines using Azure Data Factory (ADF), Azure Monitor and Log Analytics with a real-world project.

You will learn about manual/Triggers in Azure Data Factory (ADF) and how to use them to schedule the data pipelines.

Create Data driven, fully integrated, dynamic and automated production grade pipeline creation and orchestration.

Data Warehousing Concepts like Fact & dimension tables, SCD type1, type2, Incremental loading and implement using ADF.

Connect & copy data from On Premise data stores, On premise SQL server via Self Hosted Integration Runtime to cloud.

How to ingest data from sources such as REST API, Azure Blob Storage, SQL DB into Azure Data Lake Gen2 using Azure Data Factory (ADF)

Course Curriculum

Chapter 1: Initial Concepts

Lecture 1: Data Factory creation

Lecture 2: Resource Group

Lecture 3: Top Level Concepts in ADF

Lecture 4: Linked Service and Datasets

Lecture 5: Blob Storage

Lecture 6: Variables in ADF

Lecture 7: Parameters

Lecture 8: JSON Structure

Lecture 9: Integration Runtime

Chapter 2: Pipeline Activities

Lecture 1: Copy Activity

Lecture 2: Append Variable Activity

Lecture 3: Delete Activity

Lecture 4: Execute Pipeline Activity

Lecture 5: Fail Activity

Lecture 6: Get Metadata Activity

Lecture 7: Lookup Activity

Lecture 8: Set Variable Activity

Lecture 9: Wait Activity

Chapter 3: Conditional and Loop Activities

Lecture 1: Switch Activity

Lecture 2: Filter Activity

Lecture 3: ForEach Activity

Lecture 4: Until Activity

Lecture 5: If Condition

Chapter 4: Data Flow Transformations

Lecture 1: Conditional Split Transformation

Lecture 2: Exists Transformation

Lecture 3: Union Transformation

Lecture 4: Join Transformation

Lecture 5: Aggregate Transformation

Lecture 6: Surrogate Key Transformation

Lecture 7: Select Transformation

Lecture 8: Lookup Transformation

Lecture 9: Derived Column Transformation

Lecture 10: Pivot Transformation

Lecture 11: Unpivot Transformation

Lecture 12: Rank Transformation

Lecture 13: Window Transformation

Lecture 14: Flatten Transformation

Lecture 15: Filter Transformation

Chapter 5: Real Time Scenarios

Lecture 1: Part 1 – Copy files dynamically from one storage location to other.

Lecture 2: Part 2 – Copy files dynamically from one storage location to other.

Lecture 3: Set Variable : self assignment handling.

Lecture 4: Fetch file names from folder and assign to variable

Lecture 5: Copying specific file pattern files from Source to Sink

Lecture 6: ADF pipeline logging

Lecture 7: Process CSV file and split single column into multiple columns

Lecture 8: Copy Missed files from Source to Destination

Lecture 9: Run ADF pipeline manaully-Call pipeline from other ADF pipeline (with REST API)

Chapter 6: Email Notification with ADF and Logic App

Lecture 1: Email notification theory

Lecture 2: Logic App Creation in Azure

Lecture 3: Logic App and ADF Configuration for Email notification

Lecture 4: Email notification configuration before and after pipeline with Stored procedure

Chapter 7: Data Load for Multiple Tables and multiple approaches

Lecture 1: Full Load and Incemental load Theory

Lecture 2: Copying multiple tables from database to database using ADF array.

Chapter 8: Slowly Changing Dimensions

Lecture 1: Fact and Dimension tables introduction

Lecture 2: Slowly Chaning Dimension – Type 1

Lecture 3: Slowly Changing Dimension – Type 2

Lecture 4: Slowly Changing Dimension – Type 6

Chapter 9: Logs Maintenece in ADF

Lecture 1: Logs Table Creation

Lecture 2: Starting Logs Stored Procedure Creation

Chapter 10: Azure Key Vault

Lecture 1: Azure Key Vault – Creation

Chapter 11: Azure Databricks – PySpark

Lecture 1: Azure Databricks Service -Creation

Lecture 2: JDBC Connection -How to connect to SQL server & read table from Azure Databricks

Lecture 3: Mount to ADLS or Blob from Azure Databricks using PySpark

Lecture 4: Read CSV file from DBFS with schema and header

Lecture 5: withColumn() Transformation Function

Chapter 12: Azure Databricks – Delta Lake & Delta Table

Lecture 1: Delta lake introduction

Lecture 2: Datawarehouse Vs Data Lake Vs Delta Lake

Lecture 3: Delta Table Architecture

Lecture 4: Delta Table creation and observing its structure

Instructors

Atchyut Kumar
Azure Data Engineer and Instructor

Rating Distribution

1 stars: 6 votes

2 stars: 7 votes

3 stars: 30 votes

4 stars: 84 votes

5 stars: 101 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!