End-to-End Medallion Architecture Implementation for GlobalMart

Learning Objectives

Understand the step-by-step workflow of the Bronze, Silver, and Gold layers in Databricks.

Perform data ingestion and create datasets in ADLS with proper configuration.

Build Delta tables in the Bronze and Silver layers, applying schema constraints and validations.

Generate aggregated insights in the Gold layer optimized for analytics.

Overview

This project guides you through building a complete Medallion Architecture (Bronze–Silver–Gold) pipeline in Databricks.

You’ll start by ingesting data into ADLS, organize it in the Bronze layer, perform cleaning and transformation in the Silver layer, and finally create aggregated outputs in the Gold layer.

By the end of this 3-hour hands-on session, you’ll gain practical experience in managing each layer efficiently using PySpark and SQL.

Prerequisites

Basic knowledge of PySpark and SQL in Databricks
Understanding of storage accounts and data upload process
Familiarity with Delta tables and Medallion architecture concepts
Understanding of Databricks workspace and clusters