AI-generated content. This lesson was produced from model knowledge plus the cited public sources; its claims have not yet passed Fabric Codex human verification. Verify limits and feature support against current Microsoft documentation before relying on them.

The Native Execution Engine: what actually changes#

Fabric's Native Execution Engine (NEE) replaces the execution layer of Spark SQL — not the planner, not the API — with vectorized C++ code. It is built from two open-source components: Velox, Meta's C++ database acceleration library that performs the vectorized execution itself, and Apache Gluten (incubating), Intel's middle layer that offloads work from JVM-based SQL engines to native ones [S1].

The mechanics: Catalyst still parses, analyzes, and optimizes exactly as before. Gluten then intercepts the optimized physical plan, converts the portions it supports into a Substrait plan (a cross-engine plan-interchange format), and hands those to Velox, which executes them as SIMD-accelerated, columnar C++ operators in place of the JVM implementations [S1]. Data stays in columnar batches in native (off-JVM-heap) memory through the offloaded segments, avoiding both row-at-a-time iterator overhead and JVM garbage-collection pressure on the hot path.

Spark SQL query lifecycle in Fabric

The crucial design property is per-operator fallback. When NEE meets an operator, expression, data type, or feature it does not support, only that segment of the query falls back to standard JVM Spark; the job continues correctly, just without acceleration for that portion [S1]. A single query can therefore be a patchwork of native and JVM segments, with columnar-to-row (and back) conversions at each boundary — and those transitions are themselves a cost worth watching. As of the current release, structured streaming, ANSI SQL mode, and JSON/XML sources always run on JVM Spark, while Python/Scala UDFs, complex types (arrays, maps, structs), and CSV (via a vectorized native parser) are handled natively [S1].

Trust, but verify. Whether an operator actually ran natively is observable: native operators in Spark UI or df.explain() carry suffixes like *Transformer, *NativeFileScan, or *VeloxColumnarToRowExec, and the Gluten SQL/DataFrame tab renders the execution graph with green nodes for native execution and light blue for JVM fallback [S1]. Never assume acceleration from the flag alone. For scale expectations: the GA benchmarks reported by the product team showed roughly 4x on a 1 TB TPC-DS workload and up to 6x on representative end-to-end jobs, at no additional cost over standard Spark capacity billing [S1] — treat these as vendor-reported reference points, not guarantees for your workload.

Semantic divergences. Velox is a reimplementation, not a port, and a few documented behaviors differ from vanilla Spark: DECIMAL→FLOAT casting goes directly from the internal int128_t representation instead of Spark's precision-preserving string round-trip, so rounding can differ [S1]; round() uses C++ std::round rather than Spark's own logic [S1]; collect_list()/collect_set() use ARRAY as the intermediate aggregation type where vanilla Spark uses BINARY [S1]; and with spark.sql.mapKeyDedupPolicy=EXCEPTION, NEE currently skips the duplicate-key check and silently keeps the last value instead of throwing [S1]. For numerically sensitive or exactly-reproducible pipelines, test under NEE explicitly before relying on it.

Shuffle and AQE mechanics#

Shuffle is where Spark SQL queries go to die, so it deserves precise mental models. A shuffle (exchange) materializes a stage boundary: each map task partitions its output by the join/aggregation key hash, writes sorted per-partition blocks to local disk, and downstream reduce tasks fetch their partition from every map output. Costs are threefold — serialization, disk, and network — and they scale with data volume crossing the boundary, not with cluster size.

AQE exploits exactly this materialization: because shuffle map outputs are complete before reducers start, Spark can inspect the real partition sizes and rewrite the remaining plan — coalescing small shuffle partitions, demoting a sort-merge join to a broadcast join when one side proved small, and splitting skewed partitions into multiple tasks. NEE preserves these behaviors: adaptive execution, cost-based rewrites, column pruning, and predicate pushdown all continue to apply even when parts of the plan run natively and parts fall back [S1].

Under NEE, offloaded operators exchange columnar batches rather than rows, which changes shuffle serialization characteristics; the fine-grained, Fabric-specific public documentation of its shuffle implementation is thin, so beyond "verify in the Gluten tab and Spark UI," treat shuffle-level NEE behavior as something to measure rather than assume.

The cluster topology underneath is simpler than most Spark deployments: Fabric fixes the node-to-executor ratio at 1:1 — one node dedicated to the driver, each remaining node hosting exactly one executor (single-node pools split the node between driver and executor) [S2]. The head node hosts Livy, the YARN Resource Manager, ZooKeeper, and the driver [S2]. That means executor sizing is node sizing: shuffle spill capacity, off-heap headroom for Velox, and per-executor parallelism all follow directly from the pool's node size. Dynamic allocation reserves executors at submission from the pool minimum and grows with task demand [S2], and YARN executor decommissioning is enabled by default so underused nodes shut down during scale-down [S2].

Data skipping and V-Order on the read path#

Before any operator — native or JVM — touches data, Delta's transaction-log statistics (per-file min/max) plus partition pruning decide which Parquet files are read at all; pushed-down predicates then prune row groups and pages via Parquet's own statistics. Layout optimizations compound with NEE rather than being replaced by it: NEE supports parallel Delta snapshot loading and specifically accelerates reads over tables organized with Z-ordering or Liquid Clustering [S1] — clustering makes file statistics selective, and the native scan then reads the surviving files faster.

V-Order is Fabric's write-time Parquet optimization: it applies special sorting, row-group distribution, dictionary encoding, and compression to Parquet files, primarily so that Direct Lake semantic models and other Fabric readers get fast scans. For Spark SQL itself the effect is more modest — better compression and encoding help scan throughput, but V-Order is a trade of write cost for read benefit, and heavy write-oriented Spark workloads sometimes disable it. Public documentation quantifying V-Order's interaction with NEE's native scan specifically is limited; benchmark on your own tables before attributing wins or losses to it.

Common performance failure modes#

  1. Silent JVM fallback. The flag is on, the query is slow, and the Gluten graph is light blue. Check for unsupported expressions or sources; each native↔JVM boundary also adds columnar↔row conversion cost [S1].
  2. Skewed shuffles. One hot key produces a straggler task per stage. AQE skew handling mitigates; persistent skew calls for key salting or pre-aggregation.
  3. Small files. Thousands of tiny Parquet files from over-partitioned or trickle-ingested tables dominate runtime with per-file overhead — compact (OPTIMIZE) before blaming the engine.
  4. Non-sargable predicates. Wrapping the filter column in a function defeats file skipping and pushdown; the plan shows a bare scan with a post-filter.
  5. Under-sized nodes for shuffle-heavy SQL. With the 1:1 executor model [S2], a too-small node size caps memory and spill headroom per executor; wide aggregations spill to disk long before CPU is the bottleneck.
  6. Numeric drift under NEE. Decimal casts, round(), and collect intermediates differing from vanilla Spark [S1] can surface as "wrong" reconciliation totals rather than as errors — a correctness failure mode masquerading as a data bug.

Where public detail runs out#

Be explicit about the frontier: Microsoft documents NEE's architecture, fallback rules, quirks, and observability well, but does not publish fine-grained internals such as Velox operator memory-management specifics inside Fabric, native shuffle implementation details, or V-Order×NEE interaction numbers. Where this lesson leans on general Velox/Gluten/Spark knowledge rather than Fabric-specific documentation, it says so — and so should you when you carry these claims into a design review.

Sources#