Fabric Spark Toolkit

48-section interactive internals reference, unit-tested utilities, and executed notebooks for Microsoft Fabric Spark — Runtime 1.3 (Spark 3.5 / Delta 3.2) and Runtime 2.0 (Spark 4.x / Delta 4.2), configuration-driven throughout. All links are relative: keep the files in one folder (as unpacked from fabric_spark_toolkit.zip) and everything below just works. Notebooks open as rendered previews (browsers cannot display raw .ipynb files) with a download link alongside.
Start here: open the interactive internals reference and set the runtime selector (top right) to your workspace's runtime — the Feature Explorer, table advisor, config reference and callouts adapt. Then jump straight to what you need: the live cluster console · the end-to-end map · recommended defaults · ingestion patterns · CU accounting · setup runbook · expert path · reading Spark logs · CDF incremental in SQL · settings matrix · Runtime 2.0 deep dive · dbt, SQL DB & functions deep dives · capacity & CU planning · troubleshooting & Livy/logs · Delta optimization · reading query plans · NEE fallback · Spark SQL patterns · migration framework · Warehouse / Polaris · pain-point diagrams · standards gallery.

Reference documents

spark_internals.html INTERACTIVE
48 sections: engine internals (Catalyst, AQE, NEE, memory, shuffle, RSM), Fabric platform (runtimes, capacity, monitoring), best practices, config reference + advisors, annotated plan explorer, E2E architecture map, NEE fallback simulator, Spark SQL gallery, Warehouse/Polaris internals, step-through pain-point diagrams, interactive standards gallery.
runtime_2_0_guide.html RUNTIME 2.0 GA
Dedicated deep dive into Fabric Runtime 2.0 (Spark 4.1, Delta 4.2, Python 3.13, Java 21, Azure Linux 3.0). The ANSI vs. Gluten SIMD trade-off, Spark Connect, Python UDAFs, Liquid Clustering GA, and the interactive code migration simulator.
onelake_polaris_deepdives.html ONELAKE & POLARIS
OneLake unified storage architecture, shortcut security & delegated identity resolution, zero-copy Iceberg REST Catalog sharing, Polaris Two-Phase Cascades query compilation, Direct Lake lazy transcoding & framing, and SQL DB translytical replication.
fabric_deepdives.html DEEP DIVES
Seven deep dives: Fabric Data Agents & governed NL2Ontology, Fabric IQ enterprise knowledge graphs, Spark views & functions, NEE do's and don'ts, Fabric SQL Database, User Data Functions, and dbt on Lakehouse + Warehouse.
efficient_scaledown.html SCALEDOWN & RSM
Complete standalone internals reference: Spark shuffle mechanics, executor decommissioning, Remote Shuffle Manager (RSM), Decision Layer, AQE Shuffle Write, constraints, and TPC-DS results.
Coding Standards RENDERED
Worked before/after examples across PySpark, Spark SQL, Delta, NEE, metadata and engine choice — every example tagged 3.5+, 4.x only or differs.
All documents VIEWER
Standards, build plan & gap analysis, analyzer sample output and the file index — rendered with syntax highlighting, readable offline.

Utilities — Python modules (unit-tested)

fabric_workload_advisor.py MODULE
CU/capacity math, admission checks (430 vs queue), engine choice (Python vs Spark), spill-risk estimation, Delta table properties, runtime profiles + %%configure generation.
spark_eventlog_analyzer.py MODULE
Fabric URL normaliser (handles every "bad URL" shape) + a genuine Spark event log parser: critical path, skew, spill, GC, executor lifecycle, effective config — all derived from the raw event stream.
spark_plan_analyzer.py MODULE
Notebook linter (L-codes) + dependency-tree walker (%run / notebook.run / runMultiple) + EXPLAIN-based plan generation and issue detection (P-codes) → one markdown review.
spark_autoconfig_core.py + spark_autoconfig.py MODULE
Spark tuning heuristics (partitions, executor shape, broadcast, memory) with basis tags, plus the session wrapper honouring the runtime/session-start scope split.

Tests

tests/ UNIT TESTED
Unit and end-to-end tests for the utilities, plus probe_mutability.py — run it inside Fabric to empirically confirm which Spark configs are runtime-mutable vs session-start on your exact runtime.
tests/README.md

Notebooks — all executed end-to-end on the Runtime 1.3 stack

fabric_best_practices.ipynb PATTERNS
Runtime-portable session config, Polars/DuckDB/delta-rs on Delta, DV/CDF/liquid clustering executed, metadata loop, runMultiple DAG, Fabric-only paste cells.
Open preview →Download .ipynb
spark_auto_config_utility.ipynb TUNING
Analyze data + cluster → recommend, apply (runtime scope) and emit %%configure (session-start scope).
Open preview →Download .ipynb
nb_lakehouse_health_audit.ipynb OPS
Inventory every Delta table: size, files, DV/CDF, pending deletion vectors, last OPTIMIZE → flagged health report.
Open preview →Download .ipynb
nb_lakehouse_maintenance.ipynb OPS
Acts on the health report: OPTIMIZE / REORG PURGE / VACUUM. Dry-run by default, fully logged, post-checked.
Open preview →Download .ipynb
nb_data_quality.ipynb OPS
Declarative rules table, generic runner, severity gates: error stops the pipeline, warn quarantines and continues.
Open preview →Download .ipynb
nb_ingestion_generic.ipynb OPS
One parameterized worker for N entities: watermark-gated incremental MERGE, run-logged, self-proving idempotency.
Open preview →Download .ipynb
nb_nee_fallback_analyzer.ipynb MEASURE
Runs a query set with NEE on vs off, computes native coverage per query from the plans, and quantifies the ANSI×NEE trade-off on Runtime 2.0.
Open preview →Download .ipynb
nb_metadata_sqldb_prototype.ipynb METADATA
Fabric SQL Database metadata framework: full DDL, pyodbc+Entra connectivity, forward-only watermarks, batched run logs, and the 11 gotchas that save days.
Open preview →Download .ipynb
nb_cdf_incremental_pattern.ipynb INCREMENTAL
CDF-driven deltas without MLV: SQL Database watermark table, version-bounded reads with pushdown asserted from the plan, net-change collapsing, idempotency proof and the retention guard.
Open preview →Download .ipynb
nb_api_zip_ingestion.ipynb INGESTION
Rate-limited API crawl (token bucket, Retry-After, resumable cursor) and streamed zip-over-HTTPS, landing into datetime partitions as gzipped NDJSON, with CU-estimate logging.
Open preview →Download .ipynb
nb_fabric_log_diagnostics.ipynb DIAGNOSTICS
Fabric Spark Monitoring REST API client (jobs/stages/tasks/advisor/resourceUsage) that turns "this cell is slow" into a ranked diagnosis - skew, spill, Advisor findings. Demo-mode tested; live-API paths documented.
Open preview →Download .ipynb
nb_eventlog_analysis.ipynb EVENT LOGS
Paste any Fabric URL shape, download the Spark event log, and get a real diagnosis: critical path, skew ratios, spill, GC, findings. Proven against a real event log generated in-session.
Open preview →Download .ipynb
nb_workspace_monitoring.ipynb OPS
KQL over the Monitoring Eventhouse (ItemJobEvents etc.), alerting guidance, and the Spark-logs route: emitter → Event Hubs → Eventstream → Eventhouse.
Open preview →Download .ipynb
Design choice: one adaptive internals document with a runtime selector rather than per-runtime sub-documents — duplicated HTML drifts out of sync the first week. Modularity lives here instead: this hub links every asset, and the big document deep-links per section. Heuristic thresholds across the toolkit are tagged as such; validate against your own Spark UI and Capacity Metrics data. Notebooks were executed on Spark 3.5 + Delta 3.2 locally; Fabric-only cells are guarded and labelled.