What Spark is doing for you in Fabric#
When you write a PySpark or Scala notebook in Fabric's Data Engineering or Data Science experience, that code runs on a fully managed Apache Spark platform — Fabric provisions and operates the Spark clusters for you [S1]. You don't install Spark, patch it, or manage a cluster manager; you pick a pool, run your code, and Fabric handles the compute underneath.
Fabric offers two styles of Spark compute, and the difference matters the moment you open your first notebook. Starter pools are preconfigured and kept ready so sessions begin almost instantly. Custom Spark pools are pools a workspace admin configures explicitly — node size, autoscaling behavior, and other compute settings — for workloads with specific needs [S1].
Why starter pools feel instant#
Starter pool sessions typically come up in 5 to 10 seconds, because Fabric keeps Medium-node clusters pre-provisioned and running in the background, ready to be handed to your session the moment you attach a notebook [S1]. The instant you change anything — pick a different node size, or point at a custom pool — Fabric switches you to on-demand provisioning, where it acquires fresh nodes from Azure. That takes roughly 2 to 5 minutes, and a plain custom pool startup lands around three minutes for the same reason: nodes have to be requested and allocated before Spark can start on them [S1].
What you're billed for#
Fabric only meters Spark usage for the time a session is actively running code. Idle time on a pre-warmed cluster, the time Fabric spends acquiring a cluster, Spark context initialization, and the deallocation that happens after your job finishes are all unbilled, for both starter and custom pools [S1]. Sessions that sit unused don't run forever, either: by default a session expires after 20 minutes of inactivity (an admin can change this), and an unused pool is torn down 2 minutes after that expiry [S1].
Sizing custom pools#
If you do move to a custom pool, node sizes run from Small (4 vCores, 32 GB memory) up through Medium (8 vCores, 64 GB), Large (16/128), X-Large (32/256), to XX-Large (64 vCores, 512 GB) — with the two largest sizes reserved for non-trial Fabric SKUs [S1]. One practical wrinkle: changing the node size on a pool after it's created requires restarting any sessions currently attached to it, so resizing isn't something you can do underneath a live job [S1].
A worked example#
Imagine you open a new notebook to explore a lakehouse table for the first time. You attach it to the workspace's default starter pool, and within about 10 seconds you have a live Spark session — no cluster request, no wait [S1]. You run a few df.show() and groupBy() calls, then step away for lunch. Twenty minutes of inactivity later, Fabric expires the session, and two minutes after that, the pool itself deallocates since nothing else is using it [S1]. None of that idle time appeared on your Spark bill — only the minutes your code was actually executing did [S1]. Later, a teammate needs an X-Large custom pool for a heavier job; because it's a non-default configuration, their session takes a few minutes to spin up rather than seconds, since Fabric has to acquire those nodes from Azure fresh [S1].
A few things beyond the pool basics#
As you write and tune notebooks, you'll run into some engine-level details that are useful to recognize even at this stage, even though the mechanics behind them go deeper than this lesson covers:
- Fabric has an opt-in Native Execution Engine that transparently accelerates supported Spark SQL operators — you can confirm whether a given query actually ran natively by checking the Spark UI or
df.explain()output for suffixes like*Transformeror*NativeFileScan, or by opening the dedicated "Gluten SQL / DataFrame" tab, where green nodes mean native execution and light blue means it fell back to the standard JVM path [S2]. - A setting called Optimized Write helps control how many files your writes produce, and its configuration key isn't stable across runtimes: Fabric Runtime 1.2 used
spark.microsoft.delta.optimizeWrite.enabled, while Runtime 1.3 and later renamed it tospark.databricks.delta.optimizeWrite.enabled[S3]. If you copy a setting from an older notebook, check which runtime you're actually on. - Runtime 1.3 also removed the older
spark.sql.parquet.vorder.enablesetting entirely; that data-layout optimization is now applied during Delta table compaction (OPTIMIZE) instead [S4]. - Apache Spark itself is evolving toward more declarative pipeline authoring — Spark Declarative Pipelines shipped as a headline feature of open-source Spark 4.1.0 in December 2025, after being contributed by Databricks in mid-2025 — so declarative, dependency-managed pipeline styles are becoming a first-class part of the Spark runtime, not just a platform-specific wrapper [S5].
What goes wrong#
- Assuming every session starts in seconds. Only starter pools (default node size, default config) get the pre-warmed fast path; any custom pool or non-default node size falls back to on-demand provisioning that takes minutes, not seconds [S1].
- Panicking about idle-time cost. Time spent waiting for a cluster, initializing the Spark context, or sitting pre-warmed and unused isn't billed — only actual running time is [S1].
- Resizing a pool without expecting a restart. Changing node size on an existing custom pool forces active sessions on it to restart; it's not a live, in-place change [S1].
- Copying Optimized Write or V-Order settings across runtimes unchanged. The config key names and mechanisms shifted between Runtime 1.2 and 1.3+, so a setting that worked in an older notebook may silently do nothing in a newer runtime [S3] [S4].
AI-generated deep dive (beyond the verified knowledge base)#
The section below is AI-generated from model knowledge, not from verified Fabric Codex claims. It is believed factual; verify specifics against current documentation before relying on them.
Notebooks and Spark job definitions: the two ways you run Spark code#
Everything above described the compute underneath your code; it's just as useful to know the two front doors Fabric gives you for running that code. Notebooks are the interactive path: a cell-based editor where you write code, run it against a live Spark session, and see tables and charts inline. They're built for exploration, iterative development, and narrative documentation of your logic. Spark job definitions are the batch path: you point Fabric at a script file (such as a Python file or a compiled Scala/Java artifact), give it arguments and a default lakehouse, and run it non-interactively — typically on a schedule or as a step in a data pipeline. A good rule of thumb: prototype in a notebook, and when the logic stabilizes into a repeatable production job, either promote it to a Spark job definition or schedule the notebook itself from a pipeline so it runs unattended.
Languages you can use#
Fabric notebooks support several languages on the same engine: PySpark (Python) is the most widely used, and you can also write Spark SQL, Scala, and R. You can mix languages inside one notebook using cell-level magic commands, and pass data between them by registering a dataframe as a temporary view — the view becomes queryable from a SQL cell, and vice versa. Whichever language you pick, the work compiles down to the same Spark execution plans, so language choice is mostly about team skills and library ecosystems rather than performance.
The basic dataframe workflow#
Most beginner Spark work in Fabric follows one loop: read a lakehouse table into a dataframe, transform it, and write the result back. Reading is typically spark.read.table("my_table") for a lakehouse table; transformations are chained methods like select, filter, withColumn, groupBy, and join; writing back is usually df.write.mode("overwrite").saveAsTable("my_result"). Because Fabric lakehouse tables are Delta tables, whatever you save this way shows up in the lakehouse's Tables area and is immediately readable by the SQL analytics endpoint and Power BI — you don't do anything extra to "publish" it.
One behavior that surprises every newcomer: Spark is lazily evaluated. Transformations like filter and groupBy don't run anything — they just build up a plan. Only an action (such as show(), count(), or a write) triggers actual execution. That's why a long chain of transformations appears to complete instantly and the final write seems to "do all the work": it really is doing all the work, all at once. Understanding this makes job timings, and the Spark UI you'll eventually graduate to, far less mysterious.