Microsoft Fabric Runtime 2.0 Internals

1. Chronological Timeline of Fabric Runtimes

Understanding the component evolution across Fabric runtimes is essential when upgrading existing Lakehouse pipelines or debugging subtle behavioral discrepancies.

2. Architectural Blueprint: Runtime 1.3 to 2.0 Changes

3. The ANSI × Native Execution Engine (NEE) Conflict

The single most impactful behavioral difference in Fabric Runtime 2.0 is the interaction between ANSI SQL mode and the Native Execution Engine (Gluten/Velox).

How to Mitigate the ANSI vs NEE Conflict

# Strategy 1: Use try_cast() in SQL - Preserves native SIMD acceleration under ANSI mode
df_optimized = spark.sql("""
  SELECT 
    order_id,
    try_cast(amount_raw AS DOUBLE) AS amount,
    try_cast(tax_raw AS DOUBLE) AS tax
  FROM bronze_orders
""")
# Strategy 2: For pure batch ETL pipelines where performance supersedes ANSI strictness
spark.conf.set("spark.sql.ansi.enabled", "false")

4. Interactive Migration Diagnostic Simulator

Select a code pattern to inspect the behavioral transition from Runtime 1.3 to Runtime 2.0:

5. V-Order (spark.sql.parquet.vorder.default) — Workload Decision Matrix

Enabling V-Order globally across all workloads is a major anti-pattern in Microsoft Fabric. V-Order imposes a 10% to 15% write-time CPU & memory overhead during writes in exchange for ~20% to 40% faster read scans in Power BI Direct Lake and the Polaris SQL Endpoint.

# Bronze / Streaming Pipeline: Disable V-Order to minimize write latency and save CUs
spark.conf.set("spark.sql.parquet.vorder.default", "false")
# Gold Aggregation Pipeline: Enable V-Order for Direct Lake sub-second dashboard queries
spark.conf.set("spark.sql.parquet.vorder.default", "true")

6. Fact-Checked Configuration Reference: When is %%configure Actually Needed?

In Microsoft Fabric, %%configure -f forces a complete kernel restart and allocates a new session. It should only be used when configuring session-locked infrastructure properties:

7. Validated Workload Defaults & Configuration Scopes

Important Validation Finding: AQE settings such as spark.sql.adaptive.advisoryPartitionSizeInBytes and parallelismFirst are runtime-mutable SQL configurations. Putting them in %%configure -f is an anti-pattern that forces an unnecessary session restart.

In Microsoft Fabric, Starter Pools require ZERO %%configure boilerplate. All tuning should happen dynamically per-cell via spark.conf.set() or at table creation via TBLPROPERTIES.

# Bronze / Ingestion: Maximize write throughput (Skip V-Order, tune small shuffle)
spark.conf.set("spark.sql.parquet.vorder.default", "false")
spark.conf.set("spark.sql.shuffle.partitions", "128")

-- Standard Delta Table (Optimized Write is active by default in Fabric)
CREATE TABLE bronze_telemetry (
device_id BIGINT,
timestamp TIMESTAMP,
payload STRING
) USING DELTA;

# Silver Merge / CDC: 128 MB bins minimize rewrite footprint during MERGE
spark.conf.set("spark.microsoft.delta.optimizeWrite.binSize", "134217728")
spark.conf.set("spark.sql.parquet.vorder.default", "false")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5")

ALTER TABLE silver_orders SET TBLPROPERTIES (
'delta.enableDeletionVectors' = 'true',
'delta.enableChangeDataFeed' = 'true'
);

# Gold Consumption: Enable V-Order for Direct Lake sub-second queries
spark.conf.set("spark.sql.parquet.vorder.default", "true")
spark.conf.set("spark.microsoft.delta.optimizeWrite.binSize", "536870912") # 512 MB bins for Direct Lake

CREATE TABLE gold_sales_mart (
sale_id BIGINT,
customer_id INT,
store_id INT,
total_amount DOUBLE
) USING DELTA
CLUSTER BY (customer_id, store_id)
TBLPROPERTIES ('delta.enableDeletionVectors' = 'true');
-- Scheduled Maintenance
OPTIMIZE gold_sales_mart;

# Large-Scale Shuffle (> 1 TB): Scale shuffle partitions and broadcast thresholds dynamically
spark.conf.set("spark.sql.shuffle.partitions", "8192")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "268435456") # 256 MB broadcast ceiling
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "134217728")