Machine Learning with Apache Spark on Azure Databricks

Duration

2 Days

Skills Gained

By the end of this course, participants will be able to:

• Explain Spark’s distributed architecture and how it differs from traditional single-machine and Hadoop MapReduce processing

• Perform core Spark operations including data loading, transformations, actions, and caching

• Query and manipulate data using Spark SQL and DataFrames

• Apply Spark MLlib classification and regression algorithms to build predictive models

• Engineer features and tune hyperparameters for Spark ML models

• Apply stream-processing concepts and windowing operations to real-time data

Who Can Benefit

This course is designed for:

• Data Engineers

• Data Scientists & ML Engineers

• Technical Analysts with Python and Spark experience

Prerequisites

Participants should enter this course with:

• Practical Python experience

• Basic understanding of SQL

• Recommended (not required): Mastering the Data Lakehouse: Analytics with Azure Databricks, or equivalent Databricks experience

Organizational Objectives

This course assists organizations to:

• Build in-house machine learning capability for health insurance data — claims, risk, and fraud — in two days

• Reduce time-to-model by standardizing on Spark MLlib pipelines and feature engineering practices

• Improve model development velocity through reusable feature engineering and hyperparameter tuning workflows

• Establish a shared Spark and PySpark technical vocabulary across data engineering and data science teams

Software

• A computer with an internet connection is required

• A remote lab VM with an Azure account will be provided to the participants

Module 1 - Introduction to Spark: Overview and Architecture

By the end of this module, you will be able to explain the big data characteristics that motivate distributed computing, contrast Spark with Hadoop MapReduce, and describe Spark’s cluster manager, driver, and executor architecture within Azure Databricks.

• Big Data fundamentals: volume, velocity, variety, and veracity

• Scaling and distributed computing: bringing data to compute vs. compute to data

• History of Spark and Hadoop, and Spark vs. Hadoop MapReduce

• Supported languages and the Pandas API on Spark

• Spark architecture: cluster manager, driver process, and executor/worker processes

• Databricks cloud and compute clusters

• Hands-on Lab: Provision a Databricks compute cluster and trace how a sample health insurance claims dataset would move through Spark’s driver and executor architecture.

Module 2 - Basic Spark Operations and Transformations

By the end of this module, you will be able to initialize a Spark session, load data, apply actions and transformations, and use persistence and caching to improve performance.

• Spark session and context

• Loading data into Spark

• Actions vs. transformations

• Persistence and caching

• Hands-on Lab: Load a sample member enrollment dataset into a Spark session, apply a sequence of transformations and actions, and cache the result for reuse.

Module 3 - Introduction to Spark SQL

By the end of this module, you will be able to create and manipulate DataFrames, query data with Spark SQL, connect to cloud storage, JDBC, and Hive sources, and reason about Spark SQL’s performance and fault-tolerance characteristics.

• Spark SQL and the Catalyst Optimizer

• Uniform data access: cloud storage, JDBC sources, and Hive integration

• Creating DataFrames (createDataFrame, Row, toDF, registerTempTable, spark.read)

• Common DataFrame methods, grouping, and aggregation

• Converting between RDDs and DataFrames

• Reading and writing JSON data

• Performance, scalability, and fault tolerance of Spark SQL

• Hands-on Lab: Create a DataFrame from a sample claims JSON file, register it as a temp table, and use Spark SQL to group and aggregate claims by provider.

Module 4 - Spark MLlib: Classification and Regression

By the end of this module, you will be able to select and apply appropriate Spark MLlib classification and regression algorithms for a given prediction task.

• Spark MLlib overview and features

• Classification: binary, multi-class, multi-label, and imbalanced classification

• Regression: linear, multiple linear, and polynomial regression

• Regression: support vector, decision tree, and random forest regression

• Hands-on Lab: Train a binary classification model to predict claim denial likelihood on a sample health insurance dataset, then compare it against a regression model predicting claim cost.

Module 5 - Feature Engineering and Hyperparameter Tuning in Spark MLlib

By the end of this module, you will be able to engineer features from raw and text data using Spark MLlib transformers, and tune model hyperparameters using grid search, random search, and Bayesian optimization.

• Feature engineering processes and extraction methods

• Text feature extraction: TF-IDF, Word2Vec, and Count Vectorizer

• Feature transformers: Tokenizer, Stopwords Remover, N-gram, and Binarizer

• Hyperparameter tuning methods: random search, grid search, and Bayesian optimization

• Hands-on Lab: Engineer features from a sample of free-text claims notes using Tokenizer, Stopwords Remover, and TF-IDF, then tune a model’s hyperparameters using grid search.

Module 6 - Streaming Analytics

By the end of this module, you will be able to describe the streaming analytics landscape, differentiate timestamp types in stream processing, and apply windowing operations to streaming data.

• The state of data in data science: batch vs. streaming

• Stream analytics tools: Kafka, Storm, Flink, and Spark Streaming

• Timestamps in stream analytics: event, arrival, and processing timestamps

• Windowing operations: tumbling window, sliding window, and session window

• Hands-on Lab: Apply a tumbling window to a simulated stream of incoming health insurance claims and compute a rolling claims-volume metric.