Spark is the compute engine behind Data Engineering and Data Science work in Microsoft Fabric, and Microsoft runs it as a managed service rather than something you provision yourself [S1]. In practice that means you pick between two very different flavors of compute: a starter pool that's already warm and ready before you ask for it, or a custom pool that you configure by hand for node size and scaling behavior [S1].
That choice ripples through everything else. It changes how long you wait for a notebook to attach, what you pay for, and even whether certain networking setups are compatible with your session. Two teams running what looks like the same workload can have wildly different experiences purely because of pool configuration, runtime version, or how their workspace network is locked down.
This article is a working tour of the layer engineers actually deal with: what makes a session fast or slow to start, what a Spark pool looks like once you're inside it, how the cluster grows and shrinks on its own (and, now, how it can shrink without losing shuffle data mid-job), how to stop a Delta table from turning into thousands of tiny files, how to diagnose a failed job without leaving your notebook, and — for anyone chasing performance — how Fabric's C++-based execution engine speeds queries up and where its answers can subtly differ from ordinary Spark's.
Pools, startup time, and billing#
The two pool types are really a tradeoff between speed and control. Because Fabric keeps a fleet of Medium-sized nodes running and idle in reserve, a starter pool session usually attaches in 5 to 10 seconds — but the moment you ask for a different node size or any custom setting, Fabric switches to provisioning a cluster from scratch, which stretches that wait to roughly 2 to 5 minutes [S1]. That warm fleet is also a best-effort optimization, not a guarantee: if regional demand has exhausted Microsoft's pre-warmed capacity, a starter pool session quietly falls back to on-demand provisioning and takes the same 2-to-5-minute hit anyway [S1].
Custom pools give you five node sizes to choose from, scaling from a Small node (4 cores, 32 GB of memory) up through Medium, Large, and X-Large, topping out at an XX-Large node with 64 cores and 512 GB; note that the two biggest sizes aren't available on trial capacity, and resizing an existing pool forces any running sessions to restart [S1].
Idle sessions don't run forever — by default Fabric tears one down after 20 minutes of inactivity (you can change that window), and roughly two minutes after the last session on a pool ends, the pool itself is released; spinning a fresh custom pool back up from nothing takes on the order of three minutes because Azure has to hand over the underlying nodes [S1].
There's a second knob that affects startup and is easy to overlook: how your environment's libraries get installed. Quick mode installs them when the session starts, which can add anywhere from half a minute to five minutes; Full mode instead deploys a pre-built snapshot of the environment, adding one to three minutes — but pair Full mode with a custom pool that's kept warm, and Fabric bakes that snapshot straight onto already-running clusters, bringing startup back down to roughly five seconds [S1]. If a notebook on a starter pool still feels slow to open, check the library publishing mode before assuming the pool itself is misconfigured.
A related, less-obvious option for latency-sensitive production work: custom live pools. An admin defines a recurring active window during which a dedicated custom pool stays warm with libraries pre-hydrated, giving consistent ~5-second starts — Microsoft positions this as the better choice over starter pools for predictable, scheduled workloads like Spark Job Definitions, since a starter pool's warm fleet is shared and best-effort while a live pool's warm window is yours [S1].
The billing model tracks actual usage, not cluster uptime. Fabric only meters the wall-clock time your session is doing real work — the minutes a starter pool sits pre-warmed and idle, the time it takes Azure to acquire nodes, Spark's own startup bookkeeping, and the teardown after your job finishes are all free, on both pool types [S1]. Put differently: that near-instant starter pool experience isn't something you're quietly paying for elsewhere — Microsoft eats the cost of keeping it warm.
Delta write optimization: keeping tables from fragmenting#
Left to its defaults, Spark tends to scatter a busy Delta table's data across a large number of small files, and that fragmentation is one of the most reliable ways to make downstream reads slow. Fabric gives you three separate mechanisms to fight this, and picking the right one — and knowing when to switch it off — is a real design decision, not something you can set-and-forget.
Do this: turn on Optimized Write for partitioned and streaming tables#
Rule: Enable Optimized Write on tables with many partitions, and on any streaming job writing frequent micro-batches.
Why: Optimized Write works by redistributing rows across executors before the write happens, so that each partition ends up owned by a single executor and lands as one larger file instead of dozens of small ones [S4]. Streaming writes are especially prone to fragmentation because every micro-batch tends to spawn its own crop of small files; turning this feature on folds that cleanup into the write itself instead of requiring a separate compaction pass afterward [S5].
The payoff can be substantial when the table shape fits. One practitioner's own benchmark on a table split across 1,823 date-based partitions found that turning the feature on cut write time from about six minutes forty-three seconds down to under a minute, and collapsed the file count from over 175,000 down to matching the partition count — with downstream queries running roughly nineteen times faster [S4]. (That figure comes from one author's specific test setup, not a guaranteed outcome for every workload.)
The setting lives under different names depending on your runtime version, and the target file size itself is configurable:
# Fabric Runtime 1.3 and later
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true")
spark.conf.set("spark.databricks.delta.optimizeWrite.binSize", "256") # MB
# Fabric Runtime 1.2 used a completely different key —
# code migrated from 1.2 that keeps this old key silently does nothing:
# spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true")
That binSize setting controls how large the consolidated output files get: Fabric defaults to a 1 GB target, whole numbers are read as megabytes, and the commonly recommended values are 256 MB for typical workloads or 128 MB if the workload is small enough that a bigger target would over-consolidate it [S4]. Anyone porting code from Runtime 1.2 needs to catch that the configuration key itself was renamed for 1.3 — it's a quiet trap, since the old key doesn't error, it just silently stops doing anything [S4].
What goes wrong: Optimized Write on the wrong table shape#
Consolidating files isn't free, and the very thing that helps a partitioned table can backfire on a different one.
On a non-partitioned table, the same author found the opposite result: switching Optimized Write off made writes 35% faster and queries roughly twice as fast, because spreading the data across 96 files gave the cluster's executors more parallelism to read with than the 15 larger files the feature produced [S4]. In that case, fewer files meant less I/O overhead but a bigger loss in read concurrency — a net loss.
Optimized Write also simply doesn't mix with Liquid Clustering — the two aren't compatible, and Optimized Write should be turned off on any table that uses Liquid Clustering [S4]. That's not a performance judgment call; running both is a misconfiguration.
# Wrong: Optimized Write left on for a small, non-partitioned lookup table
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true")
df.write.format("delta").mode("overwrite").saveAsTable("dim_currency")
# -> fewer, larger files reduce read parallelism on a table that was already small
# Right: leave Optimized Write off (or scope it per-write) for small non-partitioned tables
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "false")
df.write.format("delta").mode("overwrite").saveAsTable("dim_currency")
V-Order and the Runtime 1.3 migration#
The way V-Order gets applied changed too. On current runtimes it's folded into Delta's OPTIMIZE compaction step rather than happening at write time, and Fabric Runtime 1.3 dropped the older spark.sql.parquet.vorder.enable setting entirely — anyone bringing code forward from an earlier runtime should just remove that line, since V-Order now happens through compaction instead [S3].
Auto Compaction: what it does and a bug worth knowing about#
If you want to know whether a given file consolidation was Fabric's automatic housekeeping or something you triggered manually, the Delta transaction log tells you: system-triggered Auto Compaction runs record an auto: true flag in their operationParameters, which manual OPTIMIZE calls don't set, making it straightforward to audit compaction history after the fact [S5].
Older Fabric Spark runtimes shipped with a bug where Auto Compaction would fire far more often than it should on tables larger than 1 GB — the root cause was that the logic counted files that had already been compacted toward the minNumFiles trigger threshold, instead of only counting genuinely small files [S5]. Until it was patched, the workaround was to skip Auto Compaction on tables over that size and rely on scheduled OPTIMIZE runs instead; Microsoft has since fixed the underlying issue in the runtime [S5]. If an older internal runbook tells you to avoid Auto Compaction on large tables, this is almost certainly why — worth checking whether your current runtime still needs that workaround.
Writing Delta tables like you mean it: DataFrameWriterV2#
The original DataFrameWriter API (df.write.mode("overwrite").saveAsTable(...)) has a well-known ambiguity problem: the same mode("overwrite") call can mean either "replace the whole table" or "overwrite matching data," and you can't tell which from the method call alone — that ambiguity is reason enough to flag it in code review whenever you see it [S9].
Rule: Prefer DataFrameWriterV2 (df.writeTo(...)) for table-lifecycle operations, and reserve the older DataFrameWriterV1 for simple path-based exports and plain appends.
Why: DataFrameWriterV2 replaces the single ambiguous mode() with explicit verbs — create(), append(), replace(), createOrReplace(), a predicate-scoped overwrite(condition), and overwritePartitions() for dynamic partition overwrite — so the write's intent is legible from the call itself, and it's the more natural home for modern Delta configuration such as table properties and clustering [S8] [S9].
# V1 — ambiguous: is this replacing the table, or just overwriting matching rows?
df.write.mode("overwrite").saveAsTable("sales.orders")
# V2 — explicit intent
df.writeTo("sales.orders").createOrReplace() # full table replace
df.writeTo("sales.orders").overwrite("order_date = current_date()") # predicate scope
df.writeTo("sales.orders").overwritePartitions() # dynamic partition overwrite
Two details matter once you adopt V2. First, it keeps two genuinely separate metadata stores under the hood: tableProperty() values land in the Delta transaction log and show up under SHOW TBLPROPERTIES, while option() values are per-write configuration passed to the data source and are never persisted as table metadata [S8] — confusing the two is a common source of "why didn't my setting stick" bugs. Second, don't reach for clusterBy() expecting it to configure Delta clustering yet: as of Spark 4.0 the method exists on the writer, but Delta Lake doesn't honor clustering hints supplied through DataFrame writers, so the documented workaround is a SQL CREATE OR REPLACE TABLE ... CLUSTER BY statement instead, with the underlying gap tracked on the delta-io/delta GitHub issue tracker [S9].
Inference: the two newer capabilities below extend DataFrameWriterV2 on the versions the claim specifies, but check your runtime's Spark/delta-spark version against these before relying on them.
Spark 4.2, paired with a matching delta-spark 4.2 build, adds withSchemaEvolution() on DataFrameWriterV2 as a typed replacement for the older option("mergeSchema", "true") pattern — but it only applies to append, conditional overwrite, and overwritePartitions, and throws if used with create or replace [S9]. Separately, Spark 4.0 introduces a DataFrame-native mergeInto() that returns a MergeIntoWriter (not a DataFrameWriterV2), giving a cross-provider way to express MERGE logic without importing Delta-specific DeltaTable.merge() [S9].
Choosing a concurrency pattern for high-volume jobs#
Running many small Spark jobs — one per table, one per partition, one per customer — surfaces a cluster-sizing problem that doesn't show up in a single big job: how do you avoid either starving jobs of compute or paying for a fleet of mostly-idle clusters?
What goes wrong: both obvious cluster patterns waste compute#
A single shared, high-concurrency cluster caps effective parallelism at its executor count — a four-worker cluster behaves like four execution slots no matter how many jobs are queued behind it — and funnels every job's submission and coordination through one driver node, which becomes a bottleneck and can even cause initialization failures when many jobs start at once [S6]. The opposite pattern, one dedicated cluster per job, removes that contention and lets each cluster be sized independently, but runs into per-region VM quotas and makes aggregate cost much harder to predict [S6]. Both patterns share the same underlying flaw: compute sitting idle in a lightly loaded job's cluster can't be borrowed by a different job that's running hot at the same moment [S6].
Do this: multithread within a shared Spark session#
Rule: For high-volume orchestration of many small jobs, use Python's concurrent.futures.ThreadPoolExecutor to fan work out within one or a few job clusters, rather than either one shared cluster or one cluster per job.
Why: Multithreading lets tasks share memory and processing power within a single Spark session, and is recommended as the most cost-effective pattern for high-volume lakehouse orchestration [S6]. ThreadPoolExecutor is preferred over the lower-level threading module for its simpler submission API, though its cancel() only affects threads still queued, not ones already running [S6]. In one author's benchmark on a single-node, four-core cluster running 48 jobs (each reading a million rows from a Delta table and writing to a new table), multithreading with 8 concurrent workers finished in about 9.5% of the total execution time that one dedicated job cluster per job required [S6].
from concurrent.futures import ThreadPoolExecutor, as_completed
def run_job(table_name: str) -> str:
df = spark.read.table(f"bronze.{table_name}")
df.writeTo(f"silver.{table_name}").createOrReplace()
return table_name
tables = [f"orders_{i:03d}" for i in range(48)]
with ThreadPoolExecutor(max_workers=8) as pool:
futures = {pool.submit(run_job, t): t for t in tables}
for future in as_completed(futures):
print(f"finished: {future.result()}")
Don't reach for Python's multiprocessing module here — its isolated per-process resource boundaries conflict with Spark's own distributed execution framework. Threads, coordinated by the GIL, are the concurrency model that's actually compatible with issuing many Spark actions from one session [S6]. Also budget for status visibility yourself: at the time of the author's writing, the Rich.Progress library for terminal progress bars was found to be incompatible with Fabric notebooks, so long-running multithreaded ELT jobs need their own explicit logging rather than relying on that kind of tool [S6].
A note on database queries: don't default to Spark#
A related but distinct decision shows up whenever a notebook needs to query an external database rather than a lakehouse table. Plain Pandas connects via PyODBC/SQLAlchemy over ODBC Driver 18, while both the Spark DataFrame API and Pandas-on-Spark connect over JDBC instead [S7]. For singleton lookups or low-volume queries, plain Pandas is the recommended approach; the Spark DataFrame API is a reasonable starting point mainly for developers newer to Pandas who expect to scale up later [S7]. Distributed tools aren't universally faster — coordination overhead can outweigh their benefit on small result sets [S7]. One author's test querying a single record from Azure SQL Database found Pandas completing in about 1.5 seconds versus roughly 5 seconds for Pandas-on-Spark and over 8 seconds for the Spark DataFrame API on first execution; even on repeated queries within the same warm session, Pandas averaged about 300 ms against roughly 900 ms for both Spark-based paths [S7]. That gap is architectural, not incidental: plain Pandas executes entirely at the notebook driver, while the Spark-based approaches distribute execution across executors, which pays off on large workloads but adds fixed coordination cost on small ones [S7].
Packaging production jobs: Spark Job Definitions#
Notebooks are built for interactive, cell-by-cell exploration. A Spark Job Definition (SJD) is Fabric's equivalent of spark-submit: a packaged, parameter-driven application runner for production use, distinct from that interactive model [S10].
Configuring one requires five components: a single entry-point script (.py, .scala, or .r), optional reference files for importable modules, optional command-line arguments, a lakehouse reference that sets the default metastore context, and an environment reference supplying libraries and Spark pool configuration [S10]. Unlike a notebook, an SJD does not auto-inject a SparkSession or pre-load common imports — the entry point has to instantiate its own session and import whatever notebookutils-equivalent helpers it needs explicitly [S10].
Rule: Structure the entry point around a main(argv) function guarded by if __name__ == "__main__", and separate structured configuration from runtime control flow.
Why: That guard pattern lets the same code be imported into unit tests without triggering execution — something notebooks don't support natively [S10]. Splitting concerns further, structured/versioned configuration (zone or load-group settings) is best kept in YAML read from OneLake via ABFSS paths, while runtime control flow belongs in argparse arguments with real type enforcement and validation [S10].
# entrypoint.py — a testable Spark Job Definition
import argparse
from pyspark.sql import SparkSession
def main(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument("--run-date", required=True, type=str)
parser.add_argument("--zone-config", required=True, type=str) # ABFSS path to YAML
args = parser.parse_args(argv)
spark = SparkSession.builder.getOrCreate()
zone_cfg = load_yaml(args.zone_config) # helper reading via OneLake ABFSS path
run_pipeline(spark, args.run_date, zone_cfg)
if __name__ == "__main__":
main()
The recommended path from prototype to production follows a staged sequence: build a proof of concept in a notebook, formalize it into a packaged and unit-tested library, write and test an entry-point script, attach the package to a Fabric environment, then create and run the SJD with its references and arguments [S10]. Because SJDs drop interactive cell output, production observability shifts to the Spark UI and to structured logging on stdout/stderr — making deliberate logging configuration matter more than it does in notebook development [S10].
Operations and observability#
Once a job is running in production, the operational question changes from "how do I write this" to "why did this fail, and how do I find out fast." Two preview capabilities target that directly.
Diagnosing a failed job without leaving your notebook#
The Fabric Spark Operations Skill is a preview, read-only, AI-assisted diagnostic tool: it resolves workspace and item identifiers from a plain-English description and queries Fabric's own Spark monitoring APIs to produce a severity-ranked diagnostic report [S11]. It's open source, published as part of the skills-for-fabric toolkit on GitHub, and designed to be invoked from AI coding assistants — including GitHub Copilot, VS Code, and Claude — after authenticating with az login [S11].
Its scenario coverage is broad: workspace-wide Spark health review, root-cause analysis of a failed notebook run, tracing a pipeline failure down to the underlying Spark session, run-history pattern analysis, Livy session triage, and diagnosing lakehouse table maintenance or load failures [S11]. Concretely, it can detect stuck, zombie, or otherwise unhealthy Livy sessions quietly consuming cluster capacity, and for performance problems it surfaces data skew, memory pressure, and under- or over-provisioned compute with recommended fixes — rather than requiring you to hand-inspect stage metrics yourself [S11].
For deeper analysis the skill can copy Spark event logs to OneLake and launch a local Spark History Server container, giving full Spark UI features — DAG visualization, SQL query plans, task-level metrics — without needing Fabric portal access at all [S11]. Underlying session monitoring data is retained for up to 30 days; if a run has aged out, the skill falls back gracefully and still surfaces what data remains, including the failed notebook's own run snapshot [S11].
The History Server no longer chokes on large jobs#
The Fabric Spark History Server previously had to fully parse and render an application's entire event log before its UI would even load, which made it a genuine bottleneck for large workloads [S12]. For enterprise-scale applications with hundreds of thousands of tasks, event logs over 10 GB, or long-running Streaming jobs, load time could stretch to roughly an hour — and sometimes failed to load at all [S12].
A preview snapshot-based loading approach changes that: the History Server can now load execution metrics and logs incrementally, surfacing an initial batch of data quickly and pulling further detail on demand instead of processing the full log up front [S12]. Applications that previously took up to an hour to open can instead render in seconds to minutes [S12]. The same release also lifts a prior restriction that had blocked enabling spark.eventLog.compress and spark.eventLog.rolling.enabled — previously, turning either on broke History Server rendering entirely, and both can now be enabled for Streaming and other long-running applications while the UI continues to render correctly [S12].
Runtime release channels: a validation window before updates land#
Fabric Runtime Release Channels exist to answer a specific complaint: runtime updates — library upgrades, dependency changes, security patches, OS updates — could previously reach production with no warning and disrupt a workload mid-flight [S13]. Each Spark runtime now ships two channels: a Default channel that's the current production-grade runtime running automatically for everyone, and an Early access channel, also production-grade, carrying the next round of changes for customers to validate before they're promoted [S13].
The lifecycle is cyclical: Microsoft publishes upcoming changes to early access, customers get a window to test their workloads against it, early access is then promoted to become the new default, and a fresh early-access channel opens with the next batch [S13]. Release notes are published per channel in a dedicated Spark Runtime Releases and Updates repository, so you can diff default against early-access notes to see exactly what's coming before it's promoted [S13].
Opting a session into early access takes two Spark configuration properties on an Environment item — spark.fabric.pools.skipStarterPools=true and spark.computeConf.runtime.releaseChannel=earlyAccess — attached to a notebook or Spark Job Definition; removing or resetting the setting reverts to default with no permanent change [S13].
# On the Environment item's Spark configuration (not inline in a notebook cell)
spark.fabric.pools.skipStarterPools=true
spark.computeConf.runtime.releaseChannel=earlyAccess
Inference: if you need to confirm exactly which build a session actually landed on beyond just "default" or "early access," the underlying claim states that calling spark.conf.get("spark.synapse.vhd.id", "") in that session returns the precise VHD build identifier [S13] — useful for filing a support ticket that needs an exact version, not just a channel name.
Migrating from Synapse: the command-line skill#
For teams moving off Azure Synapse Analytics, a preview Synapse Migration skill automates the move: a command-line, AI-assisted tool that migrates Spark pools, lake databases, notebooks, and Spark Job Definitions from a source Synapse workspace URL into their Fabric equivalents in a target workspace [S14].
It runs a fixed phase sequence — Spark pools become Fabric environments (upgrading the runtime to Spark 3.5 / Runtime 1.3), lake databases become Lakehouses, storage paths are scanned to create OneLake shortcuts for abfss:// references, notebooks migrate with automatic code refactoring, and Spark Job Definitions transfer with configuration preserved — ending in a cross-validation and summary report [S14]. Migration is idempotent by design, so an interrupted run can be safely re-run, and the tool automatically detects blockers with no Fabric equivalent, such as C#/.NET notebooks and GPU pools, flagging them for manual planning rather than silently failing [S14].
You choose between two strategies. Lift-and-shift auto-resolves all decisions with sensible defaults, migrates notebooks and job definitions verbatim with no code changes, and consolidates lake databases into a single Lakehouse. Migrate-and-modernize instead prompts guided decisions at each phase and applies Fabric-native refactoring — rewriting Synapse-specific APIs to Fabric equivalents (mssparkutils to notebookutils, TokenLibrary to notebookutils.credentials), plus path rewrites and removal of linked-service references — where lift-and-shift leaves that code untouched for refactoring after cutover [S14].
Internals#
Architecture & design#
Open up a Fabric Spark pool and you'll find one node designated as the head, with the rest acting as workers. Four processes live on that head node specifically: Livy, YARN's Resource Manager, ZooKeeper, and the Spark driver itself [S1]. Every single node, head or worker, additionally runs a Node Agent alongside YARN's per-node manager process, and each worker on top of that runs the process that hosts a Spark Executor [S1]. That layout is fixed rather than configurable: Fabric always pairs exactly one executor to each non-head node, reserving the head node purely for the driver — the sole exception is a single-node pool, where driver and executor share that one node's resources roughly in half, intended for small, high-availability-friendly workloads [S1].
Two related mechanisms handle elasticity. Autoscale watches activity and grows or shrinks the pool within the minimum/maximum node bounds you've configured; by default Fabric also has spark.yarn.executor.decommission.enabled turned on, which lets idle nodes shut themselves down automatically, and flipping that setting off makes the pool more reluctant to scale back down [S1]. Separately, dynamic allocation governs executor count within whatever nodes are available: when a job submits, Fabric starts it off with a number of executors tied to the pool's floor setting for node count, scales that number up as queued tasks outpace what's currently running, and hands executors back as work wraps up or the session sits idle — so you're not manually tuning executor counts stage by stage [S1].
A newer piece of that elasticity story is Efficient Scaledown, a preview capability that changes what scale-down actually costs. By default, Spark shuffle data lives on each executor's local disk, which is exactly what prevents that executor from being released until every downstream stage has finished reading its shuffle blocks [S15]. When an executor holding shuffle data is lost — to a crash, spot reclamation, or ordinary scale-down — Spark raises a FetchFailedException and has to re-execute the affected stages, which is a major source of unpredictable job runtime and wasted compute [S15]. Efficient Scaledown decouples shuffle data from executor lifetime by routing shuffle output to Azure Blob Storage, or migrating shuffle blocks on demand, instead of stranding them on local disk [S15]. It requires the Native Execution Engine and is available starting with Fabric Spark Runtime 3.5 and later [S15].
How it works internally#
The most significant architectural change to Fabric Spark recently is the Native Execution Engine (NEE), an opt-in path that swaps part of query execution from the JVM over to compiled C++ code. It's built from two existing open-source pieces rather than something Microsoft wrote from scratch: Velox, a vectorized C++ execution library that Meta built and open-sourced, handles the actual computation, while Apache Gluten — an Intel-led incubating project — sits in between and hands work from JVM-based SQL engines off to native engines like Velox [S2]. Mechanically, Gluten takes Spark's already-optimized physical query plan, translates the parts it can handle into a Substrait plan, and Velox executes that as columnar, SIMD-vectorized C++ instead of running the JVM's own operator code [S2].
The diagrams below cover this from two angles: the scaling/billing decision logic Fabric applies day to day, and the query-level execution path showing exactly where NEE and JVM fallback diverge.
NEE is built to fail safe at the operator level: if it hits something it can't handle — an unsupported operator, expression, data type, whatever — that specific piece of the query quietly drops back to the ordinary JVM Spark path instead of aborting the job, so the query still finishes correctly, just without acceleration on that segment [S2]. It also doesn't disturb the query planning Fabric Spark already does — adaptive execution, cost-based plan rewrites, column pruning, predicate pushdown all still apply, regardless of whether a given operator ends up running natively or falling back to the JVM [S2]. You can actually see which path an operator took: df.explain() or the Spark UI show native operators tagged with suffixes like *Transformer, *NativeFileScan, or *VeloxColumnarToRowExec, and there's a dedicated "Gluten SQL / DataFrame" tab with a visual execution graph where green nodes mark native execution and light blue marks JVM fallback [S2]. Fabric Spark Advisor now builds on that same visibility: it surfaces fallback behavior in real time during notebook execution, posting an alert directly in cell output whenever a query segment falls back, so you can spot unsupported operators without leaving the notebook [S16].
Lifecycle for the lifecycle of a Spark SQL statement itself — how a %%sql cell or spark.sql() call moves through Catalyst planning, adaptive query execution, and then either the native or JVM execution path down to Delta reads/writes on OneLake — is visualized in the diagram below.
Falling back safely is not the same guarantee as producing identical numbers on the operators that do run natively — NEE is a separate implementation of Spark's semantics, not a line-for-line port, and a handful of documented gaps mean it can quietly compute a different answer. collect_list() and collect_set() accumulate results internally as ARRAY under NEE, where vanilla Spark uses BINARY — a mismatch that can trip up compatibility in workloads that assume Spark's usual internal type [S2]. Set spark.sql.mapKeyDedupPolicy to EXCEPTION and vanilla Spark will throw on duplicate map keys as expected, but NEE currently doesn't perform that check at all — it silently keeps the last value it saw instead of raising an error [S2]. Rounding can diverge too: Velox implements NEE's round() by calling straight into the C++ standard library's rounding routine instead of reproducing Spark's own rounding logic, so the two engines can land on different answers for the same input [S2]. Casting DECIMAL to FLOAT is another gap — Spark's own cast preserves precision by converting through a string first, while Velox casts straight from its internal int128_t representation, which can round differently for the same value [S2]. collect_list() element order can also differ when a query relies on DISTRIBUTE BY plus SORT BY to fix ordering, because the two engines shuffle differently [S16]; separately, an unrecognized Spark session timezone string fails the job outright under NEE where vanilla Spark's JVM tolerates it, and pairing NEE with a managed private endpoint requires configuring that endpoint for both the Blob and DFS sub-resources of the storage account, since one endpoint alone doesn't cover both [S16]. None of these throw an error by default; they just quietly change the number. If you're running precision-sensitive math — anything financial or regulated — test that logic against vanilla Spark's output before trusting NEE's.
Coverage is still filling in. Right now, structured streaming, ANSI SQL mode, and the JSON and XML source formats get no native acceleration at all and always execute on plain JVM Spark [S2] [S16]. Other gaps have closed, though — Python and Scala UDFs and nested types such as arrays, maps, and structs now run on the native path with no code changes required, existing notebooks and jobs benefit automatically once NEE is enabled on the environment, and CSV parsing, once a fallback case, is now handled by a proper vectorized native reader [S2] [S17]. The recommended way to confirm NEE is actually helping a given workload is mechanical: enable it on an Environment item's Compute/Spark settings, attach that environment, run the workload, compare execution time against a run without NEE, and confirm the absence of native-engine fallback alerts [S17].
UDFs specifically get a structural speedup, not just a marginal one. Standard Spark UDF execution requires serializing rows out of Spark's internal format, shipping them to a separate Python worker process, executing the UDF, and serializing results back to the JVM — for every batch — which adds CPU overhead and breaks vectorized execution entirely [S17]. Under NEE, that round trip shrinks because data stays in columnar format longer, with vectorized UDFs seeing the largest gains and standard, non-vectorized UDFs improving as well [S17]. Complex data types get a related benefit: arrays, maps, and structs previously forced Spark's engine off its optimized columnar path into row-based processing for operations like explode, map access, and struct field extraction; under NEE those are processed natively in the columnar engine without that fallback, which the source frames as directly useful for advanced lakehouse patterns like Z-order optimization, Liquid Clustering, semi-structured analytics, and event-driven architectures — letting teams keep complex schemas without restructuring pipelines purely for performance [S17].
NEE also speeds up loading Delta table snapshots in parallel, and gets extra benefit reading tables laid out with Z-ordering or Liquid Clustering, stacking data-layout gains on top of the raw vectorization speedup [S2].
Networking posture changes the acquisition path outright, not just the timing. Turn on Tenant Private Links or a Managed VNet and starter pools stop being an option entirely — Fabric has no choice but to build the cluster on demand, which adds two to five minutes before you even factor in library installation time [S1]. A Managed VNet specifically pushes cold starts into the three-to-five-minute range and, once assigned to a workspace, also blocks that workspace from being migrated to a different region [S18].
Efficient Scaledown's internals are worth a closer look than "shuffle goes to Blob Storage." The capability is actually four cooperating parts: a Remote Shuffle Manager that reads and writes shuffle data to remote Blob Storage, shuffle migration that proactively moves blocks off an executor before decommission rather than dropping them, a per-stage decision layer that keeps small shuffles local while routing large shuffles remotely, and an AQE shuffle-write optimization that adjusts shuffle partitions at runtime [S15]. That per-stage decision layer matters for performance: routing only large shuffles remotely while keeping small shuffles on local disk is reported to deliver up to 57% better runtime than routing all shuffles remotely, while preserving the same scale-down benefit [S15]. A separate configuration property, spark.dynamicAllocation.excludeDeltaSnapshotCache, exists specifically to stop the Delta snapshot cache from blocking cluster scale-down on its own [S15]. Enabling the feature itself is a matter of Spark configuration — spark.remote.shuffle.enabled, spark.sql.rsm.decisionlayer.enabled.level, spark.sql.adaptive.shuffleWrite.enabled, and shuffle-block decommission settings — settable at notebook, environment, workspace, or Spark Job Definition scope [S15].
# Enabling Efficient Scaledown on an Environment item's Spark configuration
spark.remote.shuffle.enabled=true
spark.sql.rsm.decisionlayer.enabled.level=auto
spark.sql.adaptive.shuffleWrite.enabled=true
spark.dynamicAllocation.excludeDeltaSnapshotCache=true
Inference: the property names and their purposes are grounded directly in the claim [S15]; the specific values shown (auto, true) are standard defaults consistent with an "enable this feature" configuration, not a literal value quoted by the source.
Performance characteristics#
Microsoft has withdrawn its earlier headline NEE benchmark (a 4x TPC-DS / 6x end-to-end figure) from current guidance, and its documentation now describes NEE's gains as varying by workload rather than citing a single multiplier — so this article does not repeat that retired number. The verified benchmark figures currently available are narrower and more specific: Microsoft's internal testing reports vectorized Python/Scala UDFs running up to 5.76x faster under NEE, complex non-vectorized UDFs 1.08x to 2.5x faster depending on complexity, and TPC-DS end-to-end workloads that include complex-type queries up to 2.35x faster — all run on production-scale datasets with enterprise-typical cluster configurations, with the explicit caveat that actual gains vary by workload and cluster configuration [S17]. NEE ships as part of standard Spark capacity billing with no separate charge [S2].
Efficient Scaledown has its own, separately verified number: in Microsoft's TPC-DS benchmark on Spark 4.1, running all queries in a single session with a 3-minute pause between queries, total compute dropped from 3,724 to 2,121 VM-minutes — a reported 43% cost reduction attributed to the feature [S15].
Coming soon — this depth isn't in the knowledge base yet for baseline (non-NEE, non-UDF) Spark executor scaling or shuffle I/O against OneLake. It needs an L4/L5 source such as a Microsoft engineering blog benchmarking Fabric Spark executor scaling or shuffle I/O against OneLake at scale, independent of NEE-specific gains. Tracked in content/queue.md.
Declarative vs. procedural: choosing the right authoring model#
Fabric engineers increasingly have a real choice between writing declarative transformations — Materialized Lake Views, dbt, SQLMesh, or Spark's own new declarative pipelines — and sticking with procedural PySpark notebooks. That declarative option got a lot more mainstream recently: Databricks contributed Spark Declarative Pipelines upstream to Apache Spark in June 2025, and it landed as a flagship feature of the Spark 4.1.0 release in December 2025, meaning declarative execution is now native to open-source Spark itself rather than a vendor-only layer bolted on top [S6a].
That doesn't make notebooks obsolete, though. Procedural PySpark still earns its place for logic that's genuinely intricate — pulling data from APIs, handling streaming state, running ML inference, iterating algorithmically, or working with deeply nested structures — cases where forcing the logic into declarative SQL would be clumsy or outright impossible; the frameworks themselves aren't trying to take over that territory [S6a].
Declarative frameworks deliberately do not try to replace notebooks in these domains.
Rule: Reach for a declarative framework by default when the transformation logic is stable and tabular; keep procedural notebooks for anything stateful, iterative, or deeply nested.
Why: There's a benefit here beyond readability that matters more as AI-generated code enters pipelines: declarative frameworks check that a pipeline's declarations are internally consistent before anything actually runs, and features like dbt's dry-run mode or Spark SDP's --dry-run flag let you catch a generated mistake before it ever touches production data [S6a].
# Procedural PySpark — appropriate for stateful streaming logic (do NOT rewrite this declaratively)
def process_batch(batch_df, batch_id):
enriched = batch_df.join(broadcast(lookup_df), "device_id")
enriched.write.format("delta").mode("append").saveAsTable("iot_events")
stream.writeStream.foreachBatch(process_batch).start()
-- Declarative equivalent for stable, SQL-expressible business logic (dry-run first)
-- spark-pipelines run --dry-run
CREATE MATERIALIZED VIEW daily_device_summary AS
SELECT device_type, event_date, COUNT(*) AS event_count
FROM STREAM(iot_events)
GROUP BY device_type, event_date;
Worked example: a partitioned streaming pipeline, configured correctly#
Take a streaming pipeline that's ingesting IoT telemetry into a Delta table partitioned by device_type and event_date, running in a workspace without private networking, packaged as a Spark Job Definition for production.
Session and pool choice. For exploratory development, a starter pool gets you the 5-to-10-second attach time [S1]. The production job itself is a better fit for a custom live pool sized and scheduled to stay warm during its run window, since a long-running streaming job cares less about interactive attach latency and more about consistent, predictable starts [S1].
Write configuration, tuned for a Runtime 1.3+ table that's both highly partitioned and streaming — exactly the shape Optimized Write is meant for [S4] [S5] — written with DataFrameWriterV2 for explicit intent [S8]:
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true")
spark.conf.set("spark.databricks.delta.optimizeWrite.binSize", "256") # general-workload target
def process_batch(batch_df, batch_id):
(batch_df.writeTo("iot_events")
.append())
stream.writeStream.foreachBatch(process_batch).start()
Auditing compaction after rollout. Check that consolidation is genuinely coming from Auto Compaction and not an unexpected manual OPTIMIZE:
from delta.tables import DeltaTable
dt = DeltaTable.forPath(spark, "abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse/Tables/iot_events")
dt.history().filter("operationParameters.auto = 'true'").show(truncate=False)
Inference: the auto flag itself is grounded in the verified claim about Delta log auditing [S5]; the specific PySpark Delta API call above is standard Delta Lake usage consistent with that claim, but the exact syntax goes beyond what the claim states literally.
Verifying NEE acceleration, if enabled. Before trusting NEE's output on this pipeline's aggregations, check df.explain() for the native operator suffixes, or watch for a Fabric Spark Advisor fallback alert in cell output [S2] [S16], and specifically compare any collect_list/collect_set/round/DECIMAL→FLOAT logic against vanilla Spark's output, since those are the documented points where the two engines can disagree [S2].
If something breaks in production. Rather than manually digging through the Spark UI, point the Fabric Spark Operations Skill at the failed run in plain English — it resolves the workspace/item identifiers itself and returns a severity-ranked report, including copying event logs to a local History Server container if you need full DAG and query-plan detail [S11]. If the job ran long enough to produce a large event log, the snapshot-based History Server loading means you're not stuck waiting up to an hour just to see what happened [S12].
Networking check. If this workspace later turns on a Managed VNet or Tenant Private Link, expect starter pools to disappear and cold starts to move into the multi-minute range [S1] [S18] — factor that into any SLA for the interactive pool, not just the production job.