Mastering the Data Lakehouse: Analytics with Azure Databricks

Duration

2 Days

Skills Gained

By the end of this course, participants will be able to:

• Master the foundational concepts of Big Data, Data Warehousing, and ETL/ELT processes

• Explain the role and architecture of Apache Spark and the overall structure and components of the Azure Databricks environment

• Utilize Azure Databricks Workspaces and Notebooks to manage compute clusters and execute queries using Databricks SQL and Magic Commands

• Implement the Data Lakehouse architecture and apply Unity Catalog for robust data governance, access, and discoverability

• Work with Delta Lake, demonstrating proficiency in creating and manipulating data objects (tables, views, UDFs), performing DML operations, and leveraging versioning and Time Travel

• Conduct Exploratory Data Analysis (EDA), create and share AI/BI Dashboards, and automate data workloads using Databricks Jobs and Pipelines

Who Can Benefit

This course is designed for:

• Data Engineers

• Data Analysts

• Professionals building cloud data engineering skills with Azure Databricks (beginner to intermediate)

Prerequisites

Participants should enter this course with:

• Basic understanding of SQL (helpful, not required)

• Basic understanding of Python (helpful, not required)

Organizational Objectives

This course assists organizations to:

• Build a team-wide working knowledge of Azure Databricks and the Data Lakehouse architecture in two days

• Reduce time spent on ad hoc data engineering by codifying pipeline and job automation practices

• Improve data governance and discoverability through consistent Unity Catalog adoption

• Lower cloud compute costs through applied performance monitoring and optimization techniques

Software

• A computer with an internet connection is required

• A remote lab VM with an Azure account will be provided to the participants

Module 1 - Big Data and Data Warehousing

By the end of this module, you will be able to define Big Data, compare traditional and cloud data warehousing, and describe the data storage types, roles, and ETL/ELT processes that underpin a modern data platform.

• What is Big Data?

• Traditional Data Warehouse vs Cloud Data Warehouse

• Data Storage Types and Formats

• Data Roles and Tools

• ETL and ELT Processes

• Hands-on Lab: Classify a set of sample datasets by storage type and format, and map out an ETL vs. ELT process for one of them.

Module 2 - Apache Spark and Azure Databricks

By the end of this module, you will be able to describe how Apache Spark processes data, explain its architecture and workflow, and place Azure Databricks within that architecture.

• What is Spark?

• Spark Architecture and Workflow

• Azure Databricks Overview

• Azure Databricks Architecture and Data Sources

• Hands-on Lab: Trace a sample Spark job through its architecture and workflow stages inside the Azure Databricks workspace.

Module 3 - Azure Databricks Environment

By the end of this module, you will be able to navigate the Azure Databricks workspace, configure compute clusters, and execute queries using Databricks SQL and Magic Commands.

• The Azure Databricks Main Page

• Compute Options and Clusters

• Workspaces and Notebooks

• Databricks SQL and Magic Commands

• Hands-on Lab: Spin up a compute cluster, create a notebook, and run a query using both Databricks SQL and Magic Commands.

Module 4 - Data Lakehouses and Unity Catalog in Azure Databricks

By the end of this module, you will be able to explain the Data Lakehouse architecture and apply Unity Catalog to govern data access and discoverability.

• Understanding Data Lakehouse

• Unity Catalog and Data Governance

• Data Access and Discoverability

• Hands-on Lab: Register a sample dataset in Unity Catalog and configure access permissions for two different roles.

Module 5 - Azure Databricks Databases and Tables

By the end of this module, you will be able to navigate the data object hierarchy, apply table partitioning, and create tables using PySpark.

• Data Object Hierarchy and Tables

• Table Partitioning

• Creating Tables with PySpark

• Hands-on Lab: Create a partitioned table with PySpark and inspect the resulting data object hierarchy.

Module 6 - Delta Lake in Azure Databricks

By the end of this module, you will be able to explain how Delta Lake extends Parquet with a transaction log and ACID guarantees, and use versioning and Time Travel to recover or audit prior table states.

• Delta Lake and Parquet

• Delta Transaction Log and ACID

• Versioning and Time Travel

• Hands-on Lab: Modify a Delta table, review its transaction log, and use Time Travel to recover a prior version.

Module 7 - Azure Databricks Objects

By the end of this module, you will be able to create and use views, materialized views, and user-defined functions, and apply built-in functions and security controls to Databricks objects.

• Views and Materialized Views

• User Defined Functions (UDFs)

• Built-in Functions and Security

• Hands-on Lab: Create a view, a materialized view, and one UDF against a shared sample dataset.

Module 8 - DML Operations in Azure Databricks

By the end of this module, you will be able to perform INSERT, UPDATE, and DELETE operations, join tables using SQL and PySpark, and account for key SQL differences between T-SQL and Databricks.

• Performing INSERT, UPDATE, DELETE

• Joining Tables with SQL and PySpark

• SQL Differences in T-SQL and Databricks

• Hands-on Lab: Perform INSERT, UPDATE, and DELETE operations on a sample table, then join it to a second table using both SQL and PySpark.

Module 9 - EDA and AI/BI Dashboards in Azure Databricks

By the end of this module, you will be able to conduct exploratory data analysis, choose appropriate data visualization options, and create and share AI/BI dashboards.

• Data Visualization Options

• Exploratory Data Analysis (EDA)

• Creating and Sharing Dashboards

• Hands-on Lab: Conduct exploratory data analysis on a sample dataset and publish the results as a shared AI/BI dashboard.

Module 10 - Automation with Jobs & Pipelines in Azure Databricks

By the end of this module, you will be able to automate data workloads by creating and scheduling jobs and pipelines for real business use cases.

• Pipeline and Job Automation

• Creating and Scheduling Jobs

• Sample Code and Business Cases

• Hands-on Lab: Build and schedule a job that automates a sample data pipeline end-to-end.

Module 11 - Azure Databricks Performance Monitoring and Optimization

By the end of this module, you will be able to monitor job performance using the Spark UI and apply optimization techniques to improve cost and runtime efficiency.

• Monitoring with Spark UI

• Performance Optimization Techniques

• Hands-on Lab: Use the Spark UI to diagnose a slow-running query and apply at least one optimization technique to improve it.

Module 12 - Azure Databricks Assistant

By the end of this module, you will be able to use the Databricks Assistant inside notebooks and dashboards to accelerate everyday development tasks.

• Understanding the Assistant

• Using the Assistant in Notebooks and Dashboards

• Hands-on Lab: Use the Databricks Assistant to accelerate a notebook task and a dashboard task.