Databrics Live Training

Course Description

Demo Video:

Demo session - Databrics Live training

MODULE 0: Foundation & Readiness (Sessions 1–2)

Goal: Align on Databricks fundamentals, Lakehouse concept, and workspace navigation

Session 1: Databricks & Lakehouse Architecture

·       Problems with traditional DW + Data Lake

·       Lakehouse architecture explained

·       Databricks vs Synapse, Snowflake, BigQuery

·       Core components: Workspace, Clusters, Notebooks, DBFS / Unity Catalog (intro)
Hands-On: Login, explore workspace, create first notebook
Outcome: Understand why Databricks exists and its positioning

Session 2: Clusters, Runtime & Cost Basics

·       Cluster types (all-purpose vs job clusters)

·       Databricks Runtime, Autoscaling & auto-termination

·       Cost drivers & best practices
Hands-On: Create & configure cluster, attach notebook, run commands
Outcome: Confidently create and manage clusters

MODULE 1: Apache Spark Core (Sessions 3–8)

Goal: Build strong Spark fundamentals

Session 3: Spark Architecture Deep Dive

·       Driver vs Executors, Jobs, Stages, Tasks

·       Lazy evaluation & DAG
Hands-On: Visualize Spark UI, run jobs, inspect stages
Outcome: Understand how Spark executes code

Session 4: DataFrames & RDDs

·       DataFrames vs RDDs

·       Reading data: CSV, JSON, Parquet

·       show(), select(), filter(), withColumn()
Hands-On: Load dataset, apply transformations, register temp views & query using SQL

Session 5: Transformations vs Actions

·       Narrow vs wide transformations

·       Shuffle explained

·       collect(), count(), take()
Hands-On: Write transformations, observe execution in Spark UI

Session 6: Joins, Aggregations & Window Functions

·       Join types, groupBy & aggregations

·       Window functions in Spark
Hands-On: Multi-table joins, running window functions for analytics

Session 7: Performance Basics in Spark

·       Partitioning, Repartition vs Coalesce

·       Broadcast joins
Hands-On: Tune a slow job, compare execution times

Session 8: Spark SQL Optimization

·       Catalyst Optimizer, Tungsten, Explain plan
Hands-On: Use EXPLAIN, optimize SQL queries
Outcome: Write efficient Spark code, not just working code

MODULE 2: Delta Lake & Lakehouse Design (Sessions 9–13)

Goal: Master Delta Lake – Databricks’ key differentiator

Session 9: Delta Lake Fundamentals

·       ACID transactions, Delta vs Parquet, Delta log
Hands-On: Create Delta table, insert & query data

Session 10: Delta Table Operations

·       UPDATE / DELETE / MERGE, Upserts (CDC patterns)
Hands-On: Implement MERGE for incremental load

Session 11: Time Travel & Versioning

·       Time travel, version history, rollbacks
Hands-On: Query old versions, restore tables

Session 12: Schema Evolution & Enforcement

·       Schema enforcement, auto-merge, bad data handling
Hands-On: Load evolving schema data, observe behavior

Session 13: Bronze–Silver–Gold Architecture

·       Medallion architecture, data modeling, fact & dimension placement
Hands-On: Build Bronze → Silver → Gold pipeline
Outcome: Design production-ready Lakehouse models

MODULE 3: Data Engineering Pipelines (Sessions 14–18)

Goal: Build scalable, automated ingestion pipelines

Session 14: Batch Ingestion Patterns

·       Full load vs incremental, file-based ingestion patterns
Hands-On: Build batch ingestion pipeline

Session 15: Incremental Loads & CDC

·       Watermarking, CDC patterns, MERGE strategies
Hands-On: Incremental pipeline with MERGE

Session 16: Auto Loader

·       Auto Loader architecture, schema inference, cloud file notifications
Hands-On: Implement Auto Loader pipeline

Session 17: Structured Streaming Basics

·       Streaming vs batch, micro-batch model, checkpoints
Hands-On: Streaming ingestion to Delta

Session 18: Streaming with Delta

·       Exactly-once processing, streaming aggregations
Hands-On: Build streaming Bronze → Silver pipeline
Outcome: Build real-world ingestion pipelines

MODULE 4: Orchestration, DevOps & Productionization (Sessions 19–23)

Goal: Make pipelines production-ready

Session 19: Databricks Jobs

·       Job types, parameters, scheduling
Hands-On: Convert notebook to job

Session 20: Error Handling & Logging

·       Try/except patterns, audit tables, reprocessing strategies
Hands-On: Implement logging framework

Session 21: CI/CD for Databricks

·       Git integration, repo structure, Dev/Test/Prod separation
Hands-On: Connect Databricks to Git, commit & deploy code

Session 22: Configuration & Secrets

·       Databricks secrets, Key Vault integration
Hands-On: Secure credentials

Session 23: Performance Tuning in Production

·       Caching, Z-Ordering, OPTIMIZE
Hands-On: Tune Gold tables
Outcome: Think like a production data engineer

MODULE 5: Governance, Security & Unity Catalog (Sessions 24–26)

Goal: Enterprise-grade governance & security

Session 24: Unity Catalog Fundamentals

·       Metastore, Catalog/Schema/Table, access control
Hands-On: Create catalogs & schemas

Session 25: Data Security & RLS/CLS

·       Table-level security, row-level security, column masking, security policies

Session 26: Data Lineage & Auditing

·       Lineage tracking, audit logs
Hands-On: Explore lineage in Unity Catalog
Outcome: Support governance & compliance teams

MODULE 6: Advanced Topics & Capstone (Sessions 27–30)

Goal: Differentiate senior engineers from average

Session 27: Databricks SQL & BI Integration

·       SQL Warehouses, Power BI / Tableau connectivity
Hands-On: Connect BI tool to Gold tables

Session 28: ML & Feature Engineering (Overview)

·       MLflow basics, feature tables
Hands-On: Track experiments with MLflow

Session 29: Cost Optimization & Best Practices

·       Cluster sizing, spot instances, query optimization
Hands-On: Cost optimization exercises

Session 30: Capstone Project & Review

·       End-to-end architecture review
Hands-On Build: Ingestion, transformation, Gold tables, job scheduling, governance
Outcome: Participants leave with real project experience

Final Learning Outcomes

·       Design enterprise Lakehouse architectures

·       Build batch & streaming pipelines

·       Optimize Spark & Delta workloads

·       Implement governance using Unity Catalog

·       Productionize Databricks solutions