📊 Table Formats

🏠 Home

Overview

Modern Table Formats bring ACID transactions, schema evolution, and time travel to data lakes, enabling lakehouse architectures.

Format Comparison

Delta Lake

Origin: Databricks (2019)

Storage: Parquet + transaction log

Strengths: Mature ecosystem, streaming

Best For: Spark-heavy workloads

Apache Iceberg

Origin: Netflix → Apache (2018)

Storage: Any format + metadata tree

Strengths: Engine agnostic, hidden partitioning

Best For: Multi-engine environments

Apache Hudi

Origin: Uber → Apache (2016)

Storage: Base + log files

Strengths: Incremental processing, CDC

Best For: Real-time updates

Key Capabilities

Feature Delta Lake Apache Iceberg Apache Hudi
ACID Transactions ✅ Full ACID ✅ Full ACID ✅ Full ACID
Time Travel ✅ Version & timestamp ✅ Snapshot-based ✅ Point-in-time
Schema Evolution ✅ Add/rename columns ✅ Full evolution ✅ Schema registry
Hidden Partitioning ❌ Manual partitioning ✅ Automatic ✅ Timeline-based
Streaming Support ✅ Native streaming ✅ Via Flink/Spark ✅ Real-time ingestion
Engine Support Spark, Presto, Athena Spark, Flink, Trino, Athena Spark, Presto, Hive

Architecture Patterns

Common Patterns

  • Medallion Architecture Bronze→Silver→Gold
  • Compaction Jobs Small file optimization
  • Vacuum Operations Old version cleanup
  • Z-Ordering Data clustering
  • Liquid Clustering Auto-optimization

Use Case Fit

  • Delta Lake: Databricks ecosystem, streaming
  • Iceberg: Multi-cloud, engine flexibility
  • Hudi: CDC, incremental processing
  • All Formats: Replace Hive tables

Implementation Considerations

Key Decisions
  • File Size: Target 128MB-1GB per file
  • Partitioning: Balance query performance vs small files
  • Compaction: Schedule during low-usage windows
  • Retention: Configure vacuum/expire policies
  • Clustering: Choose columns for Z-order/liquid clustering

Real-World Examples

🏠 Back to Home