Databricks lakehouse architecture (Medallion Architecture)

 The Databricks Lakehouse architecture combines the best elements of data lakes and data warehouses into a single unified platform.

Historically, companies had to store raw, unstructured data in a low-cost Data Lake (like AWS S3 or Azure ADLS) and then move subsets of that data into a costly Data Warehouse (like Snowflake) for fast SQL queries and business intelligence. The Lakehouse eliminates this dual-system complexity by adding a warehouse-like management layer directly on top of cheap cloud storage.

---

The Core Layers of the Architecture

A Databricks Lakehouse is structured into four primary layers:

1. Storage Layer (Cloud Object Storage): All data remains in your own cloud account (e.g., Azure Data Lake Storage, AWS S3, or Google Cloud Storage). Data is stored in open-source file formats, predominantly Parquet.
2. Storage Management Layer (Delta Lake): This is the "secret sauce." Delta Lake is an open-source storage layer that sits on top of your cloud files. It brings ACID transactions, data versioning (Time Travel), and schema enforcement to your data lake.
3. Compute Engine (Photon): Databricks uses a decoupled vectorised query engine called Photon. It executes SQL queries and data processing jobs directly on the cloud object storage at extreme speeds.
4. Consumption Layer (Unity Catalog & Tools): Data governance and security are managed via Unity Catalog. From here, users can consume data using Databricks SQL for BI dashboards, Spark/Python for data engineering, and MLflow for machine learning.

---

The Medallion Architecture (Data Flow Example)

Databricks organizes data logically using the Medallion Architecture, which refines data step-by-step as it flows through three progressive tables: Bronze \(\rightarrow \) Silver \(\rightarrow \) Gold.

[ Raw Data Sources ] 
        │
        ▼
┌────────────────────────────────────────────────────────┐
│ 1. BRONZE LAYER (Raw Data / Append-Only)               │
│    Example: Raw JSON log files loaded via Auto Loader. │
└────────────────────────┬───────────────────────────────┘
                         │
                         ▼
┌────────────────────────────────────────────────────────┐
│ 2. SILVER LAYER (Cleaned / Conformed / Enriched)       │
│    Example: Deduplicated records, dates formatted,    │
│    null values handled, tables joined.                 │
└────────────────────────┬───────────────────────────────┘
                         │
                         ▼
┌────────────────────────────────────────────────────────┐
│ 3. GOLD LAYER (Aggregated / Business-Level)            │
│    Example: Daily revenue summaries for Power BI.      │
└────────────────────────────────────────────────────────┘

Code Example: Building a Lakehouse Pipeline

Here is a practical Python/PySpark example demonstrating how data is transformed across the Medallion layers inside a Lakehouse using Delta tables:

python

# ==========================================
# 1. BRONZE: Ingest raw JSON into a Delta Table
# ==========================================
spark.readStream \
    .format("cloudFiles") \
    .option("cloudFiles.format", "json") \
    .load("/mnt/raw-logs/") \
    .writeStream \
    .format("delta") \
    .outputMode("append") \
    .table("hive_metastore.default.bronze_orders")

# ==========================================
# 2. SILVER: Clean and enrich the data
# ==========================================
from pyspark.sql.functions import col, current_timestamp

bronze_df = spark.read.table("hive_metastore.default.bronze_orders")

silver_df = bronze_df \
    .filter(col("order_id").isNotNull()) \
    .dropDuplicates(["order_id"]) \
    .withColumn("processed_time", current_timestamp())

silver_df.write \
    .format("delta") \
    .mode("overwrite") \
    .saveAsTable("hive_metastore.default.silver_orders")

# ==========================================
# 3. GOLD: Aggregate for business dashboards
# ==========================================
silver_df.groupBy("customer_id") \
    .sum("order_amount") \
    .withColumnRenamed("sum(order_amount)", "total_spent") \
    .write \
    .format("delta") \
    .mode("overwrite") \
    .saveAsTable("hive_metastore.default.gold_customer_sales")

---

Why Companies Use It (Key Benefits)

• Single Source of Truth: Data engineering, BI reporting, and AI model training all read from the exact same files. You no longer need to sync data between a lake and a warehouse.
• Low Cost: Computing is decoupled from storage. You pay standard low-cost rates for cloud object storage and only pay for compute resources when actively running queries.
• Open Formats: You are never locked into a proprietary vendor format. Your data belongs to you in open Parquet/Delta formats.

Medallion Architecture Interview Questions

Q1: What is the Medallion Architecture, and what problem does it solve?

Answer: The Medallion Architecture is a data design pattern that organizes data into three distinct layers—Bronze, Silver, and Gold—to progressively improve data quality and structure as it flows through a lakehouse.

• The problem it solves: Historically, raw data lakes turned into unmanaged "data swamps" full of messy, unreliable data. The Medallion pattern introduces a structured, step-by-step pipeline ensuring data is validated, cleaned, and production-ready before reaching business users.

Q2: Can you describe the primary purpose of the Bronze, Silver, and Gold layers?

Answer:

• Bronze (Raw Layer): Act as the landing zone. It stores raw data exactly as it comes from source systems (APIs, databases, IoT logs). It is typically append-only and retains the original schema and historical data.
• Silver (Cleaned/Enriched Layer): Acts as the enterprise source of truth. Data from Bronze is deduplicated, cleaned, checked for nulls, conformed to unified formats, and enriched by joining other tables.
• Gold (Aggregated/Business Layer): Acts as the consumption layer. It contains highly optimized, aggregated data tailored to specific business use cases, dashboards, or machine learning models (e.g., daily revenue summaries).

Q3: Why do we store raw data in the Bronze layer instead of transforming it immediately upon ingestion?

Answer: Storing raw data provides an immutable audit trail. If a bug is discovered in the transformation logic later, or if business requirements change, you can easily wipe out the Silver and Gold tables and reprocess everything from scratch using the intact Bronze data.

Q4: How does Delta Lake enable the Medallion Architecture?

Answer: A Medallion architecture relies heavily on Delta Lake features to ensure reliability across layers:

• ACID Transactions: Ensures that if a pipeline fails midway through updating Silver or Gold tables, the transaction rolls back, preventing data corruption.
• Schema Enforcement: Prevents bad data from breaking the Silver/Gold tables by enforcing structural rules.
• Time Travel: Allows auditing by letting developers query exactly how a table looked at a specific point in time or version.

Q5: Should you use Streaming (Structured Streaming) or Batch processing to move data between the Medallion layers?

Answer: It depends on the business SLA, but Structured Streaming paired with Auto Loader (cloudFiles) is highly recommended for moving data from Bronze to Silver. Even if data arrives in batches once a day, using a streaming pattern with the Trigger(availableNow=True) option ensures incremental, state-tracked loading without scanning the entire directory every time, significantly lowering cloud compute costs.

Q6: Is it an anti-pattern to let BI tools or business users query the Silver layer directly?

Answer: Generally, yes, it is discouraged. The Silver layer is meant for data engineers and data scientists who need access to granular, atomic enterprise data. Business analysts using tools like Power BI or Tableau should query the Gold layer. Direct queries on Silver can cause performance bottlenecks because the data is not pre-aggregated or optimized for fast dashboard queries.

Q7: How do you handle GDPR compliance or "Right to be Forgotten" requests in a Medallion Architecture?

Answer: Because Delta Lake supports standard SQL commands, you must run DELETE operations across all three layers based on a unique identifier (like a user ID).

• Crucial Step: Running a standard DELETE only marks the data as deleted in the newest log version; the data still physically exists in older Parquet files due to Time Travel. To completely purge the data for compliance, you must run the VACUUM table_name RETAIN 0 HOURS command after the deletion to physically remove the old files from cloud storage (requires enabling the override configuration flag).

---