Spark Engine Internals in Microsoft Fabric: Native Execution, Compute Topology, and the Write Path

Spark Engine Internals in Microsoft Fabric: Native Execution, Compute Topology, and the Write Path#

This lesson goes under the hood of three subsystems that determine Spark performance in Fabric: the Native Execution Engine (NEE), the compute pool's node/executor topology and autoscaling mechanics, and the internals of the Delta write path (Optimized Write and compaction). Every claim here is verified at L4 (performance) or L5 (internals) depth — nothing below the mechanics is speculative, and gaps are called out explicitly rather than filled in.

NEE architecture: how a query actually gets accelerated#

NEE is not a Fabric-original engine — it is composed of two open-source components. Velox is a C++ database acceleration library from Meta that performs vectorized execution; Apache Gluten (incubating) is a middle layer from Intel that offloads execution from JVM-based SQL engines to native engines ^[S1]. Mechanically, Gluten intercepts Spark's already-optimized physical plan and converts the supported portions into a Substrait plan, a language-neutral intermediate representation; Velox then executes that Substrait plan as vectorized, SIMD-accelerated, columnar C++ code, replacing the JVM operator implementations for that portion of the query ^[S1].

This interception happens after Spark's own optimizer has already run. NEE preserves Fabric Spark's existing optimizer behaviors — adaptive query execution (AQE), cost-based rewrites, column pruning, and predicate pushdown all still apply, even to a query where some operators end up native and others stay on the JVM ^[S1]. Acceleration is per-operator, not per-query: when NEE hits an operator, expression, data type, or feature it doesn't support, that segment of the query automatically falls back to standard JVM Spark rather than failing the job — execution completes correctly, just without native acceleration for that fragment ^[S1]. This is the core mental model for reasoning about NEE performance: a single query can be a hybrid of native and JVM execution, stitched together transparently.

Coverage has moved since NEE's initial release. Python and Scala UDFs and complex types (arrays, maps, structs) are now supported natively, and CSV is now parsed through a vectorized native path instead of falling back — but structured streaming, ANSI SQL mode, and JSON/XML source formats are still never accelerated and always run on JVM Spark ^[S1]. On the data-layout side, NEE supports parallel Delta snapshot loading and accelerates reads on tables organized with Z-ordering or Liquid Clustering, so layout optimization compounds with operator vectorization rather than being redundant with it ^[S1].

NEE correctness quirks: where vectorized execution diverges from vanilla Spark#

Because Velox reimplements operator semantics in C++ rather than porting Spark's JVM code line-for-line, a handful of documented behaviors are not bit-for-bit identical to vanilla Spark:

• DECIMAL to FLOAT casts: vanilla Spark preserves precision by round-tripping the value through a string conversion; Velox casts directly from the internal int128_t representation, which can produce different rounding results for the same input ^[S1].
• round(): Velox implements it using the C++ standard library's std::round instead of replicating Spark's own rounding logic, a source of subtle numeric drift ^[S1].
• collect_list() / collect_set(): NEE uses ARRAY as the intermediate aggregation type where vanilla Spark uses BINARY, which can cause query-planning or execution compatibility issues in mixed or migrated workloads that assume Spark's default intermediate representation ^[S1].
• Map key deduplication: with spark.sql.mapKeyDedupPolicy set to EXCEPTION, vanilla Spark throws on duplicate keys passed to map(); NEE currently skips that check entirely and silently returns a result keyed on the last-seen value instead of failing ^[S1].

None of these fail loudly — they change results silently. Treat NEE as a performance layer that requires validation, not a drop-in replacement, for any pipeline where exact numeric or ordering results are load-bearing.

NEE performance: the numbers and how to verify them#

Independent GA benchmarks reported by the Fabric product team showed roughly 4x faster execution on a 1 TB TPC-DS workload versus vanilla Spark, and up to 6x end-to-end improvement on representative big-data jobs and typical aggregation/join queries — at no additional cost over standard Spark capacity billing ^[S1]. Because acceleration is per-operator and silent by default, don't assume a query ran natively just because NEE is enabled at the session or workspace level — the KB does not specify the exact verification mechanism beyond noting that NEE's operator-level nature makes this distinction meaningful for capacity planning and troubleshooting.

Compute pool topology: nodes, executors, and the driver#

A Fabric Spark pool instance is composed of a head node plus workers. The head node hosts Livy (the REST job-submission interface), the YARN Resource Manager, ZooKeeper, and the Spark driver; every node — head and worker alike — runs a Node Agent and a YARN Node Manager; each worker additionally runs the Spark Executor service ^[S2]. Fabric fixes the node-to-executor ratio at 1:1: one node is dedicated entirely to the driver, and each remaining node hosts exactly one executor — with a single exception. In single-node pools, the driver and executor split that one node's resources in half, a configuration built for small workloads with restorable high availability ^[S2].

This 1:1 fixed ratio is a load-bearing detail for capacity planning: adding nodes to a pool adds executors linearly, not through some sub-node packing scheme, and the driver's node is capacity spent regardless of workload size.

Autoscale and dynamic allocation: two independent mechanisms#

Autoscale and dynamic allocation solve different problems and operate at different layers. Autoscale governs the pool's node count, scaling the pool between a configured minimum and maximum based on activity; Fabric sets spark.yarn.executor.decommission.enabled to true by default, so underused nodes shut themselves down automatically — setting it to false makes scale-down less aggressive ^[S2].

Dynamic allocation governs executor count within an already-running application. With it enabled, Fabric reserves executors at job submission based on the pool's minimum node count, then grows the executor count as tasks outstrip current capacity, and releases executors as work completes or the application goes idle — removing the need to hand-tune executor counts per job stage ^[S2]. In combination: autoscale changes how many nodes (and therefore how many possible executors) exist; dynamic allocation changes how many of those executors are actually claimed by the running application at any moment.

The write path: Optimized Write and Auto Compaction internals#

Optimized Write addresses the small-file problem mechanically, not heuristically: it shuffles data across executors before writing, so each output partition is handled by a single executor and produces fewer, larger files instead of many small ones scattered across the cluster ^[S3]. The target file size is governed by the BinSize configuration key; Fabric's default is 1 GB, integer values are interpreted as megabytes, and the documented tuning guidance is 256 MB for general workloads or 128 MB for small workloads to avoid over-consolidation ^[S3].

This shuffle is not free, and it is not universally beneficial. In one controlled benchmark on a non-partitioned table, disabling Optimized Write produced 35% faster writes and 2x faster queries, because the resulting 96 output files gave better read parallelism across available executor cores than the 15 files produced with the feature enabled ^[S3]. On the opposite end, in a benchmark on a highly partitioned table (1,823 date-sold partitions), enabling Optimized Write cut write time from roughly 6 minutes 43 seconds to 53 seconds and reduced file count from 175,008 to 1,823 — a 19x improvement in downstream query duration ^[S3]. The mechanism is the same in both cases (pre-write shuffle to consolidate per-partition output); whether that consolidation helps or hurts depends entirely on whether the table's partitioning already constrains parallelism enough that fewer, larger files reduce read-side task fan-out, versus a non-partitioned table where more files simply means more parallel read tasks.

Auto Compaction is a separate, post-write mechanism, and it leaves an audit trail: in the Delta transaction log, the operationParameters field includes an auto flag set to true for system-triggered compactions, distinguishing them from manual OPTIMIZE runs ^[S4]. Its implementation has had a documented bug: earlier versions of the Fabric Spark Runtime counted already-compacted files toward the minNumFiles threshold rather than only undersized files, causing Auto Compaction to trigger excessively on tables exceeding 1 GB; the interim workaround was scheduled OPTIMIZE instead of Auto Compaction for tables over that size, and Microsoft has since resolved the issue in the Fabric Spark Runtime ^[S4].

Networking's effect on session startup#

One infrastructure-level internal worth knowing at this depth: enabling a managed VNet for Spark increases cold-start times to the 3–5 minute range and disables Spark starter pools entirely, and workspace regional migration is blocked once a managed VNet has been allocated to a workspace ^[S5]. This is a one-way architectural decision with a measurable, permanent latency cost.

What goes wrong#

• Assuming NEE acceleration is all-or-nothing per query, when it is actually silent and per-operator — a query can be a hybrid of native and JVM execution.
• Relying on exact numeric results (DECIMAL casts, round()) or exact intermediate types (collect_list/collect_set) without testing under NEE first.
• Trusting mapKeyDedupPolicy=EXCEPTION to catch duplicate map keys under NEE — it currently does not, and returns the last-seen value silently instead of erroring.
• Enabling Optimized Write by default on non-partitioned tables where it can actively hurt write and query performance by over-consolidating files and starving read parallelism.
• Running Auto Compaction unmodified on tables larger than 1 GB on older Fabric Spark Runtime versions without checking whether the excessive-trigger bug applies to the runtime in use.
• Treating dynamic allocation and autoscale as the same lever — they operate at different layers (executors within a run vs. nodes in the pool) and both need to be understood to reason about elastic capacity.
• Enabling a managed VNet without accounting for the loss of starter pools and the resulting 3–5 minute cold start, or attempting a regional migration afterward.

Open questions this lesson can't answer yet#

The knowledge base does not yet have verified L4/L5 claims on shuffle internals (partition sizing, spill behavior, shuffle service architecture), JVM garbage collection tuning, Spark UI/query-plan-level diagnostics beyond NEE's own verification signals, or capacity/vCore-to-executor sizing math. Those would need dedicated sourcing before a fuller performance/internals lesson can be written.

AI-generated deep dive (beyond the verified knowledge base)#

The section below is AI-generated from model knowledge, not from verified Fabric Codex claims. It is believed factual; verify specifics against current documentation before relying on them.

Shuffle mechanics and AQE#

The KB gaps called out above start at the shuffle, so here is the working model. A shuffle materializes at every wide dependency (joins, non-partition-aligned aggregations, repartition): map-side tasks sort and write their output to local disk as shuffle files partitioned by reducer, and reduce-side tasks then fetch their slices over the network. That disk materialization is what makes a shuffle a stage boundary — and a natural point for runtime adaptation. The reduce-side partition count is governed by spark.sql.shuffle.partitions (200 by default in stock open-source Spark), which is almost never the right static number for any given stage.

Adaptive Query Execution (AQE) exists to fix exactly that. Because each stage's shuffle output statistics are known before the next stage is planned, AQE re-plans at stage boundaries: it coalesces small shuffle partitions into fewer, right-sized ones; demotes a sort-merge join to a broadcast join when a side turns out to be small at runtime; and splits oversized skewed partitions in a skewed join into multiple tasks. Under NEE this still applies — as the grounded section notes, the optimizer pipeline including AQE runs before Gluten's plan interception — so native execution accelerates operators within a plan shape AQE already chose.

Memory management and the off-heap dimension#

JVM Spark uses a unified memory model: execution memory (shuffles, sorts, joins, aggregations) and storage memory (caching) share one region and borrow from each other, with execution able to evict cached blocks under pressure. When execution memory runs out, operators spill to disk — correct but slow, and visible as spill metrics per task. The expert-level wrinkle in Fabric is that NEE changes where the memory pressure lives: Velox is a C++ engine operating on off-heap, columnar memory, so a heavily native query stresses off-heap allocations rather than the JVM heap, and a hybrid query (native fragments plus JVM fallbacks) stresses both plus the transitions between row and columnar representations at the boundary. Diagnosing an OOM therefore starts with establishing which memory ran out — heap, off-heap, or overall container — before touching any configuration.

Skew mitigation and reading the Spark UI#

Skew shows up in the Spark UI as a signature: in a stage's task summary, compare the max task duration and shuffle-read size against the median — a max wildly above the 75th percentile means a handful of tasks own most of one key's data. Mitigations, in rough order of preference: let AQE's skew-join handling split the offending partitions; broadcast the smaller side to eliminate the shuffle entirely; or salt the hot keys (append a random suffix on the skewed side, explode the matching side) when the skew is extreme and the join can't be broadcast. For ongoing monitoring, the SQL/DataFrame tab is the highest-leverage view — it shows the executed plan with runtime row counts per operator, which is also where the *Transformer native-versus-fallback markers discussed above appear, alongside spill, shuffle, and GC time metrics that tell you whether the next tuning dollar goes to partitioning, memory, or plan shape.