Iceberg vs. Delta Lake

Metadata Tree vs. Transaction Log: The Split That Explains Everything Else

Apache Iceberg's own spec describes a table as a chain of atomic swaps: every change writes a new metadata.json file and replaces the old one in a single atomic operation. That metadata file points to a snapshot, the snapshot points to a manifest list carrying partition stats and file counts, and the manifest list points to one or more manifests, each holding a row per data file with that file's partition data and column-level metrics. No single engine owns this structure; it's a spec any compliant reader implements the same way.

Delta Lake takes a different physical approach to the same goal. Per Delta's own protocol spec, every write becomes a single-line JSON commit appended to a `_delta_log` directory, sequentially numbered, periodically compacted into Parquet checkpoints for fast replay. Delta layers on two independent versioning dimensions: protocol versions that set minimum reader/writer capability, and, for newer tables, named table features that gate specific capabilities like deletion vectors or column mapping.

Both models deliver serializable-isolation optimistic concurrency and full time travel. The practical difference shows up at the edges: Iceberg's spec-first design means no engine is privileged over another by construction, while Delta's log-and-features model means an engine on an older protocol version can be locked out of a table the moment a newer table feature gets turned on — not a bug, just a direct consequence of a protocol built to be extended quickly by whoever controls it.

Partitioning: Iceberg Hides It, Delta Replaced It With Liquid Clustering

Iceberg's answer to Hive's oldest partitioning problem is hidden partitioning: per Iceberg's own docs, the engine "produces partition values by taking a column value and optionally transforming it" and keeps track of that relationship itself, so a query never needs to know or filter on a physical partition column the way Hive requires. Hive can't validate a partition value at write time, so a small formatting mistake produces silently wrong query results instead of an error; Iceberg removes that failure mode by construction. When a table's partitioning scheme needs to change, Iceberg's evolution docs are explicit that old data stays in its old layout and new data lands in the new one, with both layouts planned separately in the same table as a pure metadata operation, no rewrite required.

Delta's current answer isn't partitioning at all — it's Liquid Clustering, which Databricks' own docs describe as a technique that "replaces table partitioning and ZORDER," letting clustering keys be redefined without rewriting existing data. It's a genuinely different mechanism aimed at the same underlying pain point: query patterns change, and a data layout chosen a year ago shouldn't require an expensive migration to fix. Liquid Clustering has real, stated limits worth knowing before you commit to it — a maximum of four clustering keys, no complex types like structs, maps, or arrays, and it's explicitly incompatible with traditional partitioning or ZORDER on the same table.

Functionally, Iceberg's partition evolution and Delta's Liquid Clustering are solving the identical problem from opposite directions: one hides and evolves partition metadata, the other replaces partitioning with a clustering key that can move.

Row-Level Deletes: Both Sides Converged on Merge-on-Read

Update-heavy and delete-heavy workloads were the weak spot of early table formats, and both projects solved it the same conceptual way: mark a row as deleted without rewriting the entire file it lives in, then resolve the deletion at read time. Iceberg's v2 spec introduced delete files that encode which rows in an existing data file are gone; v3 — now a complete, adopted spec version — replaced that with more compact binary deletion vectors. Delta Lake's deletion vectors, per Delta's own 2023 announcement and Databricks' current docs, encode deleted-row positions in RoaringBitmap, a highly compressed bitmap format, applied at query time so `DELETE`, `UPDATE`, and `MERGE` operations skip rewriting the underlying Parquet file entirely.

Both formats treat this as a temporary state, not a permanent one: Databricks' own docs are explicit that ordinary file compaction "don't have strict guarantees for resolving changes recorded in deletion vectors" — you need an explicit `OPTIMIZE` or `REORG ... APPLY (PURGE)` to actually rewrite the base files and clear the deletion vectors out. Skip that maintenance step on a delete-heavy table on either format, and you accumulate write-amplification debt that eventually shows up as degraded read performance, quietly, with no error to flag it.

The Maintenance You Can't Skip — And What Happens If You Do

Both projects' own documentation carries surprisingly blunt warnings about maintenance, worth reading before assuming either format is low-touch. Iceberg has three named operations: `rewriteDataFiles` (compaction, because small files cause "an unnecessary amount of metadata and less efficient queries from file open costs"), `expireSnapshots` (recommended to keep metadata size bounded, since data files persist as long as any live snapshot references them), and `deleteOrphanFiles`, which carries the sharpest warning in Iceberg's own docs: removing orphan files with too short a retention window "might corrupt the table if in-progress files are considered orphaned and are deleted," which is why the default retention sits at three days. Delta's equivalent is `OPTIMIZE` for compaction and `VACUUM` for retention cleanup, defaulting to seven days, with an almost identically worded warning — a shorter retention risks concurrent readers failing or, worse, the table being corrupted if `VACUUM` deletes files that haven't been committed yet, and it permanently forfeits time travel to anything older than the new retention window.

The risk this maintenance actually guards against is real and Iceberg-specific in one important way: metadata itself can become the bottleneck. An illustrative case published by e6data — not a named customer incident, worth treating as an architecturally plausible scenario rather than a verified postmortem — describes a 50 TB table built from 45 million small files under streaming ingestion, where a 5 TB metadata footprint across 200,000 manifests caused Spark and Trino query planners to crash with out-of-memory errors while trying to read stats for 22 million files during planning. The root cause tracks directly back to Iceberg's manifest-based planning model: cost scales with file and manifest count, not raw data volume, so any streaming or CDC-heavy Iceberg table needs a compaction cadence tight enough to keep that count bounded before problems appear, not after.

The Catalog Layer Is Where the Real Decision Lives Now

This is the section where "Iceberg vs. Delta" stops being a pure format question. Both projects solved the same interoperability problem from opposite directions, and both solutions are now shipping, GA infrastructure rather than roadmap promises.

On the Iceberg side, the REST Catalog spec puts catalog logic behind a common, OpenAPI-defined HTTP API, so any engine implements one client and gets access to any compliant catalog — Apache Polaris, Project Nessie, AWS Glue, Unity Catalog, and others all speak it. Snowflake built Polaris and donated it to the Apache Software Foundation in August 2024; it graduated to a full top-level ASF project on February 18, 2026, putting Iceberg's most vendor-neutral catalog under the same governance model as Iceberg itself. On the Delta side, Delta Universal Format — UniForm — asynchronously generates Iceberg (and separately Hudi) metadata from ordinary Delta commits, using the same compute that ran the transaction, with Delta's own docs stating "negligible Delta write overhead." It requires column mapping enabled, minimum reader/writer protocol versions, and Delta Lake 3.1 or newer, but where those prerequisites are met, a Delta table becomes directly readable by an Iceberg client without a migration.

Meanwhile every major cloud has shipped native, first-party Iceberg support as of this writing: Snowflake's Iceberg Tables (Snowflake-managed or externally-managed via a REST catalog), AWS S3 Tables (a dedicated managed Iceberg storage layer AWS claims delivers up to 3x faster queries and 10x higher transactions per second than self-managed Iceberg on standard S3 — a vendor number with no independently published methodology found, worth treating as a claim rather than a verified benchmark), and Google BigQuery's managed Iceberg tables. Delta's equivalent first-party distribution outside Databricks itself remains comparatively thinner. That asymmetry, not any remaining spec-level feature gap, is the more decisive input to a real format decision today.

The Tabular Acquisition: What Actually Changed

On June 4, 2024, Databricks announced it was acquiring Tabular — the company founded by Iceberg's original creators — for a reported $1 billion-plus. Databricks' own blog post framed the rationale directly: "This acquisition brings the original creators of Apache Iceberg and those of Linux Foundation Delta Lake, the two leading open source lakehouse formats, together," with the stated long-term goal of "evolving toward a single, open, and common standard of interoperability." One day later, at the same event, Databricks open-sourced Unity Catalog under Apache 2.0 through the Linux Foundation's LF AI & Data, explicitly built to support "Delta Lake, Apache Iceberg via UniForm... and all the formats out there," implementing both the Iceberg REST Catalog API and the Hive Metastore API in the same catalog.

Read together, these two moves are the clearest signal available that the era of a genuine, permanent Iceberg-versus-Delta rivalry is largely over at the format layer. What it doesn't resolve is governance concentration: Databricks now controls Delta's primary commercial distribution and employs the engineers who built Iceberg's reference implementation, even though Iceberg itself remains governed independently by the Apache Software Foundation. Snowflake's parallel move — donating Polaris to the ASF rather than building a Databricks-equivalent acquisition — reads as a direct hedge against exactly that concentration.

Neither company's public statements change the underlying open-source licensing of either format; Apache 2.0 for Iceberg and the Linux Foundation's governance model for Delta are the durable guarantees to watch, not the acquisition headlines themselves.

So Which One Should You Actually Choose?

Start from your actual engine mix, not an abstract feature checklist. If your organization is Databricks-centric and Spark is the dominant write path, Delta remains the simplest choice — it's the most mature, most deeply integrated option in that specific environment, and UniForm gives you a credible path to multi-engine read access later without a migration now. If your estate spans multiple engines and clouds — Snowflake plus Trino plus BigQuery plus Spark, none of which should have privileged access to the data — Iceberg via a REST catalog is the more defensible default, precisely because no single vendor's engine is structurally favored by the spec.

If you're already running Iceberg and ingesting from streaming or CDC sources, treat metadata scaling as a first-class operational concern from day one, not something to discover after a planner starts throwing out-of-memory errors. And if you're inheriting an existing deployment rather than choosing greenfield — which is the more common starting point for most engagements — the higher-value work usually isn't a format migration at all. It's auditing whether compaction cadence, snapshot expiration, and VACUUM retention were ever tuned past the defaults, because both projects' own documentation is unambiguous that skipping that maintenance is where real production incidents come from, regardless of which format sits underneath.

Related: Data Engineering