Who this is for#
You already know Delta Lake is "Parquet plus a transaction log." This lesson goes under that log: how commits achieve atomicity on OneLake, how the SQL analytics endpoint and Direct Lake actually read that log, and the concrete mechanics — and measured costs — of V-Order, deletion vectors, Optimized Write, and compaction.
The transaction log is the table#
A Delta table on OneLake is immutable Parquet objects plus a write-ahead log in _delta_log. The log, not a directory listing, is the single source of truth for which Parquet objects currently belong to the table — this is what lets object storage (ADLS Gen2-style, which OneLake is built on) offer ACID table semantics it doesn't natively have [S1]. Each commit is a zero-padded, sequentially numbered JSON file (000003.json) containing an array of actions: add/remove record which Parquet objects entered or left the table, metaData/protocol record schema and format-version changes, commitInfo carries audit data, and an application-defined txn action lets streaming writers track exactly-once progress [S1].
Concurrency is optimistic, not lock-based: a writer stages new Parquet objects independently, then atomically creates the next-numbered log file. Only one writer can win that commit number; a loser detects the collision immediately, rereads the latest state, and retries rather than blocking [S1]. That atomic "only one writer wins this number" guarantee is implemented per storage backend — ADLS Gen2 uses an atomic rename that fails if the target name already exists [S1]. Readers get snapshot isolation by pinning to the newest fully-committed log record at query start and never observing later commits [S1].
To keep metadata reads fast as history accumulates, the engine periodically compacts the log into a Parquet checkpoint (every 10 commits by default), folding in still-relevant actions and dropping superseded ones; readers then reconstruct state from the latest checkpoint plus a handful of trailing JSON files instead of replaying from commit zero [S1]. Checkpoint writing is best-effort and separate from the atomic commit — a crash mid-checkpoint leaves the table uncorrupted, and readers just fall back to an older checkpoint or raw commit files [S1]. This log-based discovery is roughly two orders of magnitude faster at scale than listing Parquet footers directly: a 1,000,000-partition benchmark table took over an hour for non-Delta engines versus 108 seconds for the Delta-backed engine (17 seconds with the log cached on SSD) [S1], achieved by parallelizing file-listing and statistics collection across a cluster rather than serializing on one driver — avoiding the ~1,000-key LIST-call ceiling that plain object stores hit [S1].
Data skipping and its layout dependency#
Writers record per-column min/max statistics and null counts in each object's add action, letting engines exclude whole Parquet objects from I/O before touching them [S1]. But skipping is only as good as physical layout: sorting or partitioning by one column makes skipping strong on that column and nearly useless on others, since unsorted columns' min/max ranges collapse toward the whole table's range — the direct motivation for Z-ordering across multiple columns [S1].
V-Order: write-time Parquet reshaping#
V-Order reorganizes physical Parquet layout at write time — row-group distribution, encoding choices, compression — without changing logical data, so it accelerates reads without altering query semantics [S2]. Files remain fully spec-compliant Parquet; non-Fabric engines simply don't get the acceleration [S2]. Microsoft's stated tradeoff: writes run roughly 15% longer on average for reads that can improve significantly depending on workload [S2]. V-Order is file-level, so a replaceWhere overwrite applies it only to rewritten files, and it composes with Z-Order, compaction, vacuum, and time travel [S2]. Session- or write-level settings override the table property — enabling V-Order for a session applies it to every Parquet write in that session, even tables with delta.parquet.vorder.enabled=false [S2]. For Direct Lake specifically, V-Order pays off twice: better RLE compression quality streams columns into memory faster during transcoding, and VertiScan can compute directly on compressed data without decompressing [S3].
Direct Lake's transcoding path#
Direct Lake doesn't query Delta through Spark or SQL — it transcodes Parquet encodings straight into VertiPaq in-memory structures. On a cold query it merges each needed column's per-row-group local dictionaries into one global dictionary, so more row groups means slower transcoding [S3]. Transcoding is a cheap ID remap when both sides use RLE/Bit-Packing; if a file used plain or delta encoding instead, Direct Lake must re-encode, slowing column loads [S3]. Row-group count maps one-to-one to VertiPaq segment count per column, so many tiny row groups produce many tiny segments and hurt performance [S3]. Cold multi-table DAX queries additionally require building join indexes from relationship keys and loading any deletion vectors to exclude deleted rows [S3]. Incremental framing is cheaper: it reads the Delta log and evicts only column segments tied to removed row groups, keeping and merely extending dictionaries to avoid re-transcoding [S3]. Notably, Direct Lake does not use Delta or Parquet statistics for row-group skipping when loading columns — pruning that helps other engines does nothing for its transcoding cost [S3]. Framing can outright fail if a table exceeds capacity guardrails such as 10,000 Parquet files [S4].
Deletion vectors: copy-on-write becomes merge-on-read#
Deletion vectors shift mutations from copy-on-write to merge-on-read: instead of rewriting a whole Parquet file for any changed record, the engine records affected row positions in a small binary file and filters them at query time [S5]. Mechanically, vector metadata is a UUID-named .bin file; the Delta log's add entry for the affected Parquet file gets storageType, path, offset, sizeInBytes, and cardinality fields pointing to it [S5]. For UPDATE, the old row version is soft-deleted via a new .bin record while only the changed values are written to a new, smaller Parquet file — no full rewrite [S5].
Benchmarked (100M-row synthetic dataset, author's controlled test, directional): deletion vectors cut single-row delete time ~8x, a 33% mass deletion ~2.5x, sped OPTIMIZE ~2x, and cut VACUUM ~1.7x, roughly halving cumulative write-plus-maintenance time [S5]. Before OPTIMIZE, deletion vectors used ~3x less storage (2,351 MB vs 7,705 MB) and ~1.5x fewer files than copy-on-write [S5]. But unmaintained vectors erode reads: one scenario (53% non-append changes, no compaction) ran 2.3x slower, and a MERGE-only scenario with 5M new records ran 1.5x slower than copy-on-write [S5]. After OPTIMIZE, read performance matched copy-on-write exactly [S5]. With regular OPTIMIZE and VACUUM in the loop, deletion-vector tables had the lowest cumulative processing time across all operations — net-positive for write-heavy pipelines with real maintenance [S5]. One caveat: UPDATE/MERGE combined with deletion vectors still tends to produce small files even with Optimized Write active, since those operations touch many existing files and write new ones for changed records [S6]. And portability has a rough edge — Polars via delta-rs raises DeltaProtocolError on deletion-vector tables, while Spark and DuckDB 1.2+ handle them correctly [S7].
Optimized Write and compaction mechanics#
Optimized Write addresses the small-file problem by shuffling data across executors before writing, so each partition is handled by one executor and produces fewer, larger files [S6]. Target size is governed by BinSize; Fabric's default is 1 GB, with 256 MB recommended for general workloads and 128 MB for small workloads to avoid over-consolidation [S6]. It's not universally beneficial: on a non-partitioned table, disabling Optimized Write produced 35% faster writes and 2x faster queries, because 96 output files gave better read parallelism across executor cores than the 15 files the feature produced [S6]. On a highly partitioned table (1,823 partitions), enabling it cut write time from 6:43 to 53 seconds and file count from 175,008 to 1,823 — a 19x downstream query improvement (author's controlled environment) [S6]. Fragmentation also costs compression: the feature-disabled variant produced ~33% more data on disk in a partitioned benchmark [S8].
Auto Compaction runs on three tunables: maxFileSize (default 128 MB, the compaction target), minFileSize (unset by default, computed as half of maxFileSize at runtime — the threshold below which files are candidates), and minNumFiles (default 50, the undersized-file count that triggers a run) [S9]. It's distinguishable from manual OPTIMIZE in the log: operationParameters carries an auto: true flag for system-triggered runs, useful for audit [S9]. An earlier Fabric Spark Runtime bug counted already-compacted files toward minNumFiles, causing excessive triggering on tables over 1 GB; Microsoft has since fixed it, though the earlier guidance was to fall back to scheduled OPTIMIZE for large tables until the fix shipped [S9]. Benchmarked across five configurations (200 iterations of 1K-row batches): Auto Compaction plus Optimized Write together won on every metric — lowest total runtime (12.77 min) and lowest file-count variance (stdev 14) [S9]. Running no compaction at all was worst by a wide margin: 33.27 minutes and a file-count stdev of 864 [S9]. Optimized Write alone cut per-iteration output from ~16 files to 1 with negligible overhead, but periodic compaction is still needed to consolidate into the 500 MB–1 GB range for long-term reads [S9].
Downstream consumers of the log#
The SQL analytics endpoint's background sync reads each table's Delta transaction log under /Tables in OneLake rather than re-scanning Parquet data, handling table discovery, schema-change detection, and data-change detection as one process [S4]. A newer, opt-in metadata sync (preview, announced May 2026) replaces log re-scanning with an external-tables-based approach, exposing health through sys.dm_db_external_tables_log_status (last update, Delta log version processed, latest checkpoint applied, whether the last attempt was blocked); it doesn't support Delta's deprecated multi-part checkpoint feature and can't be enabled on workspaces with workspace private link [S4]. Under either sync, unmaintained small-file growth degrades both metadata scan and query latency, since Parquet immutability means every update/delete adds files rather than rewriting them — the fix is the same scheduled OPTIMIZE/compaction discipline [S4]. High-cardinality partition columns compound this by fragmenting the table into many partitions and slowing discovery; Microsoft recommends lower-cardinality columns yielding partitions of roughly 1 GB or larger [S4]. Column mapping by name is supported (preview); mapping by ID is not, which can hide some Spark-visible columns from SQL [S10].
Fabric's Native Execution Engine adds another consumer-side gain: it supports parallel Delta snapshot loading and accelerates reads on tables organized with Z-ordering or Liquid Clustering, stacking on top of raw operator vectorization when layout is optimized [S11]. In Warehouse, every DML statement appends a JSON commit to the same kind of log, with automatic checkpointing cutting metadata I/O on frequently updated tables [S12]; a background compaction service merges small files and removes logically deleted rows, and since October 2025 it checks for shared locks held by user queries and waits or aborts rather than committing a conflicting write [S12]. V-Order is on by default for all warehouses and, once disabled, cannot be re-enabled — a common pattern is a V-Order-disabled staging warehouse feeding a V-Order-enabled warehouse for reads [S12]. Mirroring layers one more latency source on top: the SQL endpoint adds roughly 30–60 seconds of sync delay beyond underlying Delta replication, so reading the Delta layer directly via Spark is preferable when freshness matters most [S13].
What goes wrong#
- Treating Direct Lake framing as free: a table crossing 10,000 Parquet files, or accumulating high row-group counts, silently increases both framing time and cold-query transcoding cost [S3] [S4].
- Enabling deletion vectors without a maintenance cadence: reads degrade 1.5x–2.3x until OPTIMIZE runs, and some non-Spark/DuckDB<1.2 readers fail outright [S5] [S7].
- Assuming Optimized Write always helps: on non-partitioned tables it can produce fewer, less-parallelizable files and net-slower queries [S6].
- Letting UPDATE/MERGE-heavy workloads run with deletion vectors and no compaction, expecting Optimized Write alone to control file count [S6].
- High-cardinality partition columns, which slow SQL endpoint metadata discovery independent of any Spark-side symptoms [S4].
- Disabling warehouse V-Order for a one-off ingestion job — it cannot be re-enabled afterward [S12].
AI-generated deep dive (beyond the verified knowledge base)#
The section below is AI-generated from model knowledge, not from verified Fabric Codex claims. It is believed factual; verify specifics against current documentation before relying on them.
Checkpoint and log-replay economics at scale#
Snapshot construction cost is driven by the number of live add actions, not the table's byte size: a reader resolves _last_checkpoint, loads the checkpoint Parquet, then applies trailing JSON commits. Two things bloat this. First, per-file column statistics are embedded in each add action (as a JSON string, optionally also as a parsed struct), so a wide table multiplies checkpoint size by its column count; delta.dataSkippingNumIndexedCols caps how many leading columns get stats and is the standard lever when checkpoints grow disproportionate to data. Second, millions of small files mean millions of live actions — another reason compaction is metadata hygiene, not just read optimization. The protocol has evolved here: classic multi-part checkpoints are deprecated (as the Fabric metadata-sync note above reflects), and the newer V2 checkpoint design uses a manifest plus sidecar files so engines can reuse unchanged checkpoint portions incrementally rather than rewriting the whole thing every ten commits. Whether a given engine writes or merely reads V2 checkpoints varies — check each writer in your estate.
Column mapping's second-order effects#
Column mapping (none/name/id) decouples logical column names from physical Parquet field names, making RENAME COLUMN and DROP COLUMN metadata-only commits. Two consequences matter operationally. Physical names become opaque identifiers, so any tool that reads the Parquet files directly — bypassing the Delta log — sees gibberish column names; only log-aware readers reconstruct the logical schema. And a dropped column is logical: the bytes remain in existing files until they are rewritten and vacuumed, which matters for GDPR-style purge obligations — a compliance delete needs rewrite + vacuum, not just DDL.
UniForm and cross-format interop#
Delta UniForm asynchronously generates Iceberg (and optionally Hudi) metadata over the same Parquet data files after commits, letting Iceberg-native engines read the table without a copy. Because it is one physical dataset with parallel metadata trees, the constraint surface is the union of both formats' assumptions — historically, features like deletion vectors have had compatibility restrictions with UniForm, and the metadata generation is eventually consistent with the Delta log rather than transactional with it. Interop matrices change fast; verify current feature compatibility before committing an architecture to it.
Concurrency conflict classes#
Optimistic commits fail with distinct, diagnosable exceptions: ConcurrentAppendException (a competing commit added files your operation's read predicate might have matched), ConcurrentDeleteReadException / ConcurrentDeleteDeleteException (someone removed files you read or also deleted), MetadataChangedException (schema/property change raced you), ProtocolChangedException (table upgrade raced you), and ConcurrentTransactionException (duplicate streaming txn identity). The design levers: blind appends don't conflict with each other under the default WriteSerializable isolation; partition-disjoint replaceWhere writes commute; and MERGE conflict rates fall sharply when the ON clause includes the partition or clustering predicate so the read set narrows. When a pipeline shows chronic commit retries, the fix is almost always narrowing read predicates or serializing writers per partition — not disabling isolation.