Every claim cites one of these. Tier 1 = Microsoft Learn, 6 = unknown.
A Fabric Updates Blog preview announcement for two command-line, AI-assisted migration skills -- a Synapse Migration skill for Spark pools, lake databases, notebooks, and job definitions, and a companion Pipeline Migration skill for Synapse Data Factory pipelines -- both operating over REST APIs with support for lift-and-shift or migrate-and-modernize strategies.
Vendor best-practice guidance for managing Fabric content through its full lifecycle using Git integration and deployment pipelines, organized around four stages: content preparation, development, test, and production.
Microsoft explains the billing model behind Planning in Microsoft Fabric (Preview), an enterprise-planning experience built on Fabric IQ, describing why it uses a hybrid role/session-based pricing structure for people plus job-based pricing for automation rather than pure consumption or seat licensing, and how that usage surfaces in the Capacity Metrics app.
Fabric Jumpstart is Microsoft's new open-source catalog of pre-built, self-installing Fabric solutions — accelerators, demos, and tutorials that bundle datasets, notebooks, pipelines, and reports so a full end-to-end scenario can be deployed into a workspace with one command. This source combines the live catalog/filter taxonomy at jumpstart.fabric.microsoft.com with the official Fabric blog launch announcement and the backing microsoft/fabric-jumpstart GitHub repo (the PyPI-published Python library that actually performs the install) to capture both what the catalog offers and how the deploy mechanism works underneath. The catalog is a client-rendered (Next.js) app; catalog item listings render via client-side JavaScript, so this extraction draws on the filter/sort chrome and site metadata reachable without executing that client app, not a diff of individual catalog entries.
An Azure Architecture Center guide that lays out the four-level Fabric deployment hierarchy (tenant, capacity, workspace, item) and evaluates four deployment patterns for structuring capacities and workspaces, from a single shared workspace up to fully separate Fabric tenants, mapped against Well-Architected Framework pillars.
A Microsoft Learn how-to tutorial that walks through standing up a local Node.js Model Context Protocol (MCP) server in front of a Fabric GraphQL endpoint, so AI agents such as GitHub Copilot or Cursor can discover a Fabric schema through introspection and translate natural-language questions into GraphQL queries — plus the service-principal auth setup, the two MCP tools the server exposes, and production security guidance.
Microsoft Learn's how-to for building a Fabric data agent end to end: creating the item, attaching data sources, configuring agent-level and data-source-level instructions plus few-shot example queries, testing the chat experience, and publishing/sharing a draft-vs-published version with colleagues.
Microsoft Learn's concept article enumerating the four configuration surfaces available when curating a Fabric data agent — agent-level instructions, per-data-source instructions, data source descriptions, and example queries — including a recommended template structure for each and how the agent actually consumes example queries at answer time.
Microsoft Learn's reference for authoring and running Fabric notebooks, covering the multi-language cell model that underpins Spark SQL usage: the %%sql and other language magic commands, IntelliSense support per language, %run for referencing notebooks and .sql script files, and the %%configure session magic (including a Spark-SQL-specific row-limit setting). A supporting fact on how the pinned default lakehouse determines which Hive metastore Spark SQL resolves table names against is drawn from Microsoft's companion lakehouse-notebook-explore page.
Microsoft's entry point into a dimensional-modeling series for Fabric Warehouse, framing star schema (fact + dimension tables) as the recommended design approach for a Fabric data warehouse and as a prerequisite for well-performing enterprise Power BI semantic models.
Microsoft Learn's how-to for the (preview) Fabric Data Agent Python SDK evaluation workflow: installing the SDK, loading a ground-truth question/expected-answer dataset, running evaluate_data_agent, and inspecting summary and row-level results, plus how to swap in a custom LLM-as-judge critic prompt and use the Diagnostics export for troubleshooting.
Microsoft announces general availability of a direct integration that lets a Fabric Activator rule invoke a Copy job as its action, so data movement can be triggered by a detected event (e.g. a file landing or a table changing) instead of running on a fixed schedule, without needing a pipeline in between.
A Fabric Updates Blog post announcing general availability of a public management-plane API for Fabric data agents, letting the existing data agent SDK run outside a Fabric notebook — from a developer's own machine, CI/CD pipeline, container, or backend service — for creating, configuring, updating, and publishing agents programmatically.
Microsoft Learn's conceptual reference for the Fabric data agent: a generally-available, configurable conversational Q&A artifact that lets users ask plain-language questions over governed Fabric data and get structured answers back. It walks through the question-to-query pipeline, the governance/precedence model that constrains agent behavior, how it differs from Fabric Copilot, and a list of current limitations.
A Fabric Updates Blog announcement introducing Fabric Runtime Release Channels, a mechanism that splits every Spark runtime into a default and an early-access channel so customers can validate upcoming library, dependency, security, and OS updates against their own workloads before those changes become the production default.
Microsoft Learn's architecture reference for Hyperscale, the distributed engine design that underlies both Azure SQL Database Hyperscale and SQL database in Microsoft Fabric (the doc's moniker range explicitly includes fabricsql). It breaks the engine into four cooperating component types — compute nodes, page servers, a log service, and Azure Storage — and explains how they separate query processing from durable storage so each can scale independently.
Microsoft Learn's architecture reference for Hyperscale, the distributed engine design that underlies both Azure SQL Database Hyperscale and SQL database in Microsoft Fabric (the doc's moniker range explicitly includes fabricsql). It breaks the engine into four cooperating component types — compute nodes, page servers, a log service, and Azure Storage — and explains how they separate query processing from durable storage so each can scale independently.
Microsoft Learn's guide to mapping the bronze/silver/gold medallion pattern onto Real-Time Intelligence, using Eventstream ingestion, KQL update policies, and materialized views inside an Eventhouse/KQL database to keep each layer continuously current without extra orchestration infrastructure.
Microsoft's implementation guide for medallion (bronze/silver/gold) lakehouse architecture on OneLake, covering layer responsibilities, deployment patterns (all-lakehouse vs. lakehouse+warehouse gold layer), Delta Lake storage mechanics, materialized lake views for declarative transformations, and file-size/partitioning/retention guidance per layer.
A Fabric Updates Blog post walking through why Python/Scala UDFs and nested data types (arrays, maps, structs) used to defeat Spark's columnar execution, how the Native Execution Engine (NEE) now keeps both in the native path, and the internal benchmark multipliers Microsoft is citing for the improvement.
A vendor overview of Fabric's CI/CD platform, framing it as a layered stack: the Fabric REST API as the programmatic foundation, an integration/delivery layer (Git integration, deployment pipelines, Variable library), source-control automation in GitHub or Azure DevOps, and developer tooling (Fabric CLI, Terraform, fabric-cicd) for infrastructure as code.
A Fabric Updates Blog post announcing General Availability of Item Recovery, a tenant-configurable safety net that moves deleted Fabric items into a soft-deleted, restorable state instead of removing them immediately. It covers how the Workspace Recycle bin and REST API restore items, how the retention window is configured, and how admins should roll it into runbooks and governance tooling.
A Fabric Updates Blog post introducing sp_get_table_health_metrics, a built-in stored procedure exposed through the SQL analytics endpoint that lets users diagnose the physical health of a Lakehouse Delta table in T-SQL, without opening a Spark notebook. It explains what the anomaly and storage metrics mean, and shows a check-then-act pipeline pattern that runs OPTIMIZE only when the procedure flags a real problem.
Microsoft's guidance for loading dimension and fact tables in a Fabric Warehouse dimensional model: staging strategy, ETL orchestration order, transformation types, surrogate key handling, slowly changing dimension (SCD) processing logic, and fact table incremental load and integrity patterns.
Microsoft's official GA announcement for Materialized Lake Views (MLV), the declarative Spark SQL/PySpark layer for building medallion-architecture Lakehouse pipelines in Fabric. It covers what changed between the Build 2025 preview and March 2026 GA: wider incremental-refresh coverage, PySpark authoring, multiple independent refresh schedules, in-place definition updates, and expanded data-quality enforcement.
Microsoft Learn's conceptual overview of Fabric Eventstreams: a no-code capability to ingest events from a broad connector catalog (Event Hubs, IoT Hub, CDC sources, Kafka, cloud pub/sub services, Fabric/Azure native events), transform them in a drag-and-drop processor, and route them to destinations like Eventhouse, Lakehouse, or Activator.
Vendor architectural overview of Fabric's security model aimed at admins: how the SaaS platform separates metadata and compute, how authentication and authorization flow through Microsoft Entra ID and signed tokens, how data residency and multi-geo work, and how data is protected at rest and in transit.
Walks through the operational surfaces for observing Fabric Data Factory pipeline runs: the run-history flyout and monitoring hub, filtering and keyword search over activity runs, CSV export, rerun options, the Gantt view for comparing run durations, and workspace-level log monitoring via an eventhouse KQL database plus the Capacity Metrics app for capacity-level utilization.
Microsoft demonstrates an AI-assisted authoring workflow for Fabric Real-Time Intelligence: an Eventstream AI Skill takes a single natural-language prompt and generates, deploys, and validates a multi-source real-time weather-monitoring pipeline (11 cities, a humidity filter, and an Eventhouse KQL destination) in minutes instead of the many manual portal steps normally required per source.
Microsoft explains Efficient Scaledown, a preview Fabric Spark capability that decouples shuffle data from executor lifetime by routing shuffle output to remote Blob Storage, aiming to let autoscale reclaim idle executors faster and make jobs more resilient to executor loss.
A Fabric Updates Blog post positioning Fabric Data Factory as the orchestration hub for organizations running data across Snowflake, Databricks, Google BigQuery, and Salesforce. It walks through the native connectors available for each platform, the orchestration primitives that let a single pipeline span multiple clouds, and how OneLake features (shortcuts, mirroring, Purview lineage, Fabric IQ) extend those cross-cloud pipelines into governed AI scenarios.
Microsoft Learn's reference for Fabric's Spark Native Execution Engine (NEE), an opt-in vectorized C++ execution path that transparently accelerates supported Spark operators without code changes, falling back to the JVM engine for anything it can't handle. Cross-referenced with the Fabric product blog's GA announcement for adoption context and benchmark figures.
Microsoft Learn's reference for Fabric's Spark Native Execution Engine (NEE), an opt-in vectorized C++ execution path that transparently accelerates supported Spark operators without code changes, falling back to the JVM engine for anything it can't handle. This revision drops the page's own GA benchmark figures (the current text now says results 'vary by workload') and adds several newly documented edge cases and a Spark-native advisor integration.
A Fabric Data Warehouse team post introducing preview approximate string-matching functions (edit distance and Jaro-Winkler based) alongside modern string-composition additions (concatenation operators and Unicode string construction), aimed at helping T-SQL developers detect near-duplicate or inconsistently entered text values directly in queries.
Microsoft's release notes for on-premises data gateway version 3000.322 (June 2026), covering a new Windows Web Account Manager authentication option, an Apache Log4j security update, a consent-driven diagnostics upload capability surfaced in Dataflow Gen2, and alignment with the June 2026 Power BI Desktop release.
Explains the parameter and expression system that Fabric Data Factory pipelines use to avoid hard-coded values: how pipeline parameters are declared and referenced, how the @ / @{} expression syntax and string interpolation behave, how a parameter reference resolves to a connection GUID, and the built-in system variables and function families (string, collection, logical, conversion, math, date) available inside expressions.
Introduces the pipeline as the orchestration unit in Fabric Data Factory: a logical grouping of activities that is deployed, scheduled, and monitored as a set rather than individually. Covers the three activity categories, the three ways a pipeline run can be started, the role of parameters versus variables, and the monitoring surfaces available once a pipeline is running.
The landing page for Microsoft's four-part Fabric capacity planning series, aimed at administrators and Center of Excellence leads. It frames the series arc from a first proof-of-concept deployment through self-service scaling, enterprise/centralized capacity strategy, and ongoing growth governance, and points to the Fabric SKU estimator for initial sizing.
A Microsoft product-team recap of a Reddit AMA on Rayfin, the developer toolkit for building and deploying Fabric Apps, separating what the tool does today (combining semantic models and Fabric SQL databases in one app, inheriting Fabric row-level security, its own policy-based authorization layer, a recommended backend-secrets pattern) from a long list of features the team describes only as planned or upcoming.
Microsoft's how-to for evolving a Delta table's schema in Fabric without rewriting existing data: merge schema for additive changes, overwrite schema for full schema replacement, column mapping for rename/drop, type widening for compatible type promotions, ALTER TABLE for explicit changes, MERGE INTO WITH SCHEMA EVOLUTION, and the downstream impact of each change type.
Microsoft explains how Fabric Events and Azure Events (the Real-Time hub eventing layer that feeds Activator alerts and Eventstream sources across workspaces) now participate in Workspace Outbound Access Protection, so cross-workspace event consumption is governed by the same default-deny outbound network policy already available to other Fabric workloads.
Microsoft explains how Fabric Events and Azure Events (the Real-Time hub eventing layer that feeds Activator alerts and Eventstream sources across workspaces) now participate in Workspace Outbound Access Protection, so cross-workspace event consumption is governed by the same default-deny outbound network policy already available to other Fabric workloads.
Microsoft describes a strategic shift to bring Power Query's data connectors fully in-house, lays out a four-stage connector lifecycle (Preview, GA, Transparent Migration, Retirement) meant to make upgrades predictable, and lists several V2 connectors (Snowflake, Google BigQuery, Vertica, IBM Netezza, Amazon Redshift) that reached general availability under this approach.
Microsoft describes a strategic shift to bring Power Query's data connectors fully in-house, lays out a four-stage connector lifecycle (Preview, GA, Transparent Migration, Retirement) meant to make upgrades predictable, and lists several V2 connectors (Snowflake, Google BigQuery, Vertica, IBM Netezza, Amazon Redshift) that reached general availability under this approach.
Fabric Updates Blog post giving architectural design guidance for choosing between pass-through and delegated OneLake shortcuts when distributing data at scale. Overlaps heavily with the shortcuts and security sources already in this KB; the material captured here is limited to the intersection security rule for delegated shortcuts, the design-choice framing, and the external-shortcut identity clarification not already recorded.
Fabric Updates Blog post announcing a ServiceNow Workflow Data Fabric connector that reads OneLake data directly through Iceberg-compatible table APIs. The angle for Codex readers is what the integration says about OneLake's open-standards query surface, not the ServiceNow product pitch itself.
A Microsoft Learn tutorial walking through installing the community-maintained dbt-fabric adapter, wiring a profiles.yml connection to a Fabric Warehouse, and running the standard dbt debug/seed/run/test cycle against it using the jaffle_shop sample project.
Microsoft introduces the Fabric Spark Operations Skill, an open-source, AI-assisted diagnostic tool that queries Fabric's Spark monitoring APIs to turn a failed notebook, pipeline, or session into a plain-English, severity-ranked root-cause report.
Fabric Updates Blog post introducing Delegated OneLake Shortcuts as a preview auth option layered on top of the shortcut model already documented in this KB. The new material is the formal delegated-identity connection method for OneLake-to-OneLake and cross-tenant shortcuts, and how it differs from External Data Sharing - not the base mechanics of shortcuts, which onelake-shortcuts.json already covers.
Microsoft describes a rework of the Fabric Spark History Server's log-loading path, moving from an all-at-once event-log parse to an incremental, snapshot-based read so large Spark UIs open far faster.
Microsoft rounds up a set of recent Eventstream connector updates spanning private-network connectivity, Apache Kafka and Azure Service Bus source connectors, certificate-based authentication, IoT Hub metadata handling, Oracle CDC, and HTTP pagination, with each feature's GA-versus-preview status called out.
Microsoft's practitioner guidance on star schema design specifically for Power BI semantic models, covering normalization vs. denormalization, measures, surrogate keys, snowflake dimensions, slowly changing dimensions (Type 1/Type 2), role-playing dimensions, junk dimensions, degenerate dimensions, and factless fact tables.
A Fabric Updates Blog post from the Power BI team framing what it means for an organization's data estate to be safely exposed to Copilot and agentic AI, then mapping that framing onto the Purview-integrated protection controls already built into Fabric: sensitivity labels, protection policies, DLP, DLP restrict access, and Purview DSPM for Fabric.
A Fabric Updates Blog post proposing a design pattern where an AI skill or agent reads a Fabric item's existing Purview sensitivity label not just to enforce access control, but as a context signal that changes how the agent responds — for example summarizing rather than detailing Confidential content, or declining to touch Highly Confidential material even for users who technically have access.
A Fabric Data Warehouse team post explaining when to use the preview Bulk Copy (BCP) API for client-side ingestion instead of the standard server-side COPY INTO path, with concrete usage patterns for application developers (SqlBulkCopy/SQLServerBulkCopy), script-driven operators (bcp.exe), and data-integration tool users (SSIS, Azure Data Factory, Informatica).
Microsoft Learn's conceptual overview of Fabric Activator, the no-code rules engine that watches streaming and Fabric-native event sources for pattern/threshold conditions and fires downstream actions (notebooks, pipelines, Power Automate, Teams, email) without a human in the loop.
Microsoft Learn's conceptual introduction to Fabric IQ, the new Fabric workload that turns raw OneLake data into shared business meaning (via semantic models and an ontology item) so that people and AI agents can reason about customers, shipments, and other business concepts consistently. It also positions Fabric IQ inside the broader 'Microsoft IQ' enterprise-intelligence layer alongside Work IQ, Foundry IQ, and Web IQ, and inventories the Fabric items that make up the IQ (preview) workload.
Microsoft Learn's conceptual overview of Fabric's API for GraphQL: a no-code data access layer that auto-generates a GraphQL schema and endpoint over Fabric data sources so applications can query exactly the fields they need in one round trip, instead of teams hand-building REST services or wiring SQL drivers into every app.
Microsoft Learn's architectural deep dive on the ontology (preview) item that anchors Fabric IQ: how enterprise vocabulary is modeled as entity types, entity instances, properties, and relationships, how that model is bound to live data in OneLake/lakehouses/eventhouses/semantic models to form a queryable instance graph, and how natural-language and structured queries traverse that graph.
Microsoft's mid-2026 roundup of SQL-family updates spans SQL Server, Azure SQL, and SQL database in Fabric, tagging each item Preview or Generally Available. This source is scoped to only the SQL database in Fabric-relevant items in that roundup, since the rest of the post covers Azure SQL Database and SQL Server items outside the Fabric platform.
Microsoft rounds up Workspace Outbound Access Protection (preview) support across the full Real-Time Intelligence surface — Eventstream, Eventhouse, KQL Queryset, Real-Time Dashboard, and Activator — detailing per-item which outbound scenarios remain supported under a default-deny policy and which are currently blocked, including a recurring Copilot-related restriction.
Microsoft rounds up Workspace Outbound Access Protection (preview) support across the full Real-Time Intelligence surface — Eventstream, Eventhouse, KQL Queryset, Real-Time Dashboard, and Activator — detailing per-item which outbound scenarios remain supported under a default-deny policy and which are currently blocked, including a recurring Copilot-related restriction.
A Build 2026 roundup post from the Fabric Data Warehouse team bundling a large wave of announcements: GPU-based query acceleration, configurable data-history retention, a rebuilt web/query-editing experience, several T-SQL language additions, transactional schema changes, faster SQL Analytics Endpoint sync, migration tooling updates, and a forward-looking list of near-term features including cache cooldown control and a lakehouse table health check.
A Microsoft Research team (Interlandi, Bruno, Haynes, Curino, Sen, and collaborators) describes CoddSpeed, a multi-year effort to bring hardware-accelerated query execution into Microsoft Fabric's Data Warehouse engine. It builds on the team's earlier Tensor Query Processor (TQP) research, which recasts relational operators as tensor-runtime operations, and packages that idea behind two production abstraction layers so the warehouse can target GPUs today and other accelerators (FPGA/ASIC) later without rewriting the query surface. The paper won Best Paper at SIGMOD 2026's Industrial Track.
A Microsoft Research team (Interlandi, Bruno, Haynes, Curino, Sen, and collaborators) describes CoddSpeed, a multi-year effort to bring hardware-accelerated query execution into Microsoft Fabric's Data Warehouse engine. It builds on the team's earlier Tensor Query Processor (TQP) research, which recasts relational operators as tensor-runtime operations, and packages that idea behind two production abstraction layers so the warehouse can target GPUs today and other accelerators (FPGA/ASIC) later without rewriting the query surface. The paper won Best Paper at SIGMOD 2026's Industrial Track.
A community deep dive into Optimized Write behavior for Fabric and Delta Lake workloads. The extracted claims focus on how executor shuffling reduces small files, where optimized writes help most, how they interact with partitioning and deletion vectors, and which table-maintenance patterns still matter.
A community architecture analysis of tenant-level and workspace-level private link options in Fabric. The extracted claims focus on scope, feature tradeoffs, deployment friction, unsupported scenarios, and the operational impact of private networking choices across Fabric workloads.
A community walkthrough introducing Fabric Spark Job Definitions (SJDs) as the production execution mechanism corresponding to spark-submit, contrasted with interactive notebooks. The extracted claims cover the required configuration components, structural differences from notebooks (explicit SparkSession, argparse-based parameters, testability), and a recommended path from notebook prototype to production SJD.
A first-party coding-standards document for Fabric notebooks and Spark code, organized around the principle that a notebook is a job in a platform, not a script on a laptop. Nine sections of paired BAD/GOOD examples, each tagged with a Fabric Runtime compatibility marker (works on both 1.3 and 2.0, requires 2.0 only, or differs by version) and cross-referenced to analyzer finding codes.
A community-curated aggregation hub for Microsoft Fabric Data Agent content, run by Microsoft MVP Sandeep Pawar and open-sourced on GitHub. It indexes dozens of third-party articles, videos, tools, learning paths, and events from Fabric practitioners and independent MVPs, functioning as a discovery index rather than a primary knowledge source in its own right. The hub is a client-rendered site; this extraction is grounded in the site's crawlable metadata and previously-observed content categories rather than a live diff of every listed item, since most content loads dynamically.
A first-party companion reference to the Spark internals guide, covering five practitioner topics: which Spark view and function types actually exist and behave as expected (with several claims executed against a live Spark 3.5 session and marked VERIFIED or FALSIFIED rather than assumed); a do's-and-don'ts practice sheet for the Native Execution Engine; the three distinct ways a Fabric notebook can talk to a Fabric SQL Database and the gotchas of each; where Fabric User Data Functions items fit relative to Spark UDFs; and a deep dive on Microsoft's two separate dbt adapters for Fabric (Lakehouse/Spark vs. Warehouse/T-SQL), including a dbt-scheduled Materialized Lake View pattern.
A first-party deep-dive on Microsoft Fabric's Efficient Scaledown feature (Remote Shuffle Manager, Shuffle Migration, the per-stage Decision Layer, and AQE Shuffle Write), written to trace each Fabric-specific mechanism back to the stock Apache Spark subsystem it configures or extends, rather than restating the product page. Covers shuffle internals from first principles, the Spark 3.1+ executor decommissioning framework, the ShuffleDataIO plugin interface and comparable remote-shuffle implementations on other platforms, a full configuration reference with scope taxonomy (session-startup vs runtime-mutable), storage-account and Private Link constraints, and architectural guidance for governed/regulated Fabric estates.
A first-party deep dive on Microsoft Fabric Runtime 2.0, framed around one central trade-off: ANSI SQL mode is on by default in Runtime 2.0's Spark 4.1, and because the Native Execution Engine's Velox kernels implement non-ANSI null semantics, ANSI expressions force a fallback to JVM execution. The document walks the full Runtime 1.1 through 2.0 component lineage, gives a layer-by-layer V-Order enable/disable decision matrix with an explicit CPU-overhead-versus-read-speed trade-off, a reference table for when %%configure session restarts are actually required versus when a setting is runtime-mutable, and four named workload archetypes (bronze streaming ingestion, silver CDC/merge, gold Direct Lake serving, heavy ML/wide-shuffle) each with a concrete starting configuration.
The second quarter of a large first-party interactive Spark internals toolkit, spanning the configuration-reference and capacity sections through the Spark Logs REST API. Covers a searchable Spark-setting reference and an interactive config advisor with medallion presets, Fabric node-sizing and capacity-unit tables, a Spark 3.5-vs-4.x runtime comparison with a migration checklist, an engine-compatibility feature explorer, the mechanics of Fabric capacity job admission and CU chargeback, and a deep walk through the five overlapping Spark troubleshooting surfaces culminating in the Spark Logs REST API. Several tier-1 Microsoft Learn sources already in the knowledge base cover the generic capacity throttling ladder, node-size/vCore tables, and Spark History Server snapshot-loading performance in detail, so this extraction is deliberately narrow: only Spark-specific mechanics, chargeback patterns, and diagnostic-tooling detail not already captured.
The third quarter of a large first-party interactive Spark internals toolkit, spanning five sections: Delta table optimization mechanics (Materialized Lake View refresh eligibility, the CDF-vs-MLV control-flow decision, and Spark 4.x SQL session variables replacing parameter markers); a masterclass on the three-tier lakehouse statistics stack culminating in the 32-column data-skipping ceiling in the Delta transaction log; a conditional 'check-then-act' table maintenance framework built on a SQL Analytics Endpoint stored procedure; a side-by-side Data Vault 2.0 (silver) and Kimball dimensional (gold) modeling pattern for Fabric Lakehouse, including the BIGINT-vs-string surrogate key memory argument for Direct Lake; and Polaris/Warehouse/SQL-endpoint internals and diagnostics. Several tier-1/tier-3/tier-4 sources already in the knowledge base cover V-Order overhead percentages, Delta maintenance file-size targets, CDF antipatterns, deletion vector soft-delete mechanics and benchmarks, and Polaris's Cascades-compiler/task-DAG/cell-abstraction internals in much greater depth, so this extraction is deliberately narrow: only genuinely new mechanics, decision logic, and Fabric-specific application detail not already captured.
The first quarter of a large first-party interactive Spark internals toolkit, covering the engine's core execution model end to end: the driver/executor/cluster-manager roles and Fabric's 1:1 node-to-executor topology, the job/stage/task hierarchy with narrow-vs-wide transformation stage cuts, the DAGScheduler/TaskScheduler split and its locality-preference ladder, the four-phase Catalyst compilation pipeline including the cost-based optimizer's two off-by-default gates, Tungsten's binary UnsafeRow layout and whole-stage code generation, Adaptive Query Execution's five runtime rewrite features and its documented hard limits, the Native Execution Engine's Gluten/Velox fallback mechanics, unified memory management's fixed reservation and executor-memory-overhead formula, and the write/read internals of Spark's SortShuffleManager. Six other already-ingested sources (five first-party toolkit companions plus several tier-1/tier-3 Apache Spark and Microsoft Learn docs) cover much of the same ground in depth, so this extraction is deliberately narrow: only claims not already captured elsewhere in the knowledge base.
A 29-cell, first-party executable notebook that walks Fabric's full working-practice surface — runtime detection and portable session configuration, engine choice between a Python notebook and Spark, executed Delta best practices (deletion vectors, Change Data Feed, MERGE hygiene, liquid clustering, maintenance), a metadata-driven ingestion loop, and orchestration guidance — as genuinely executed code against a real local Spark 3.5 plus Delta 3.2 session, with Fabric-only pieces (notebookutils, MLVs, V-Order) cleanly guarded so the identical notebook runs unchanged inside Fabric. Its most distinctive real, executed content is a three-way single-node engine interoperability proof: the same real Delta table on disk is queried with Polars' lazy scan_delta, then with DuckDB's delta_scan SQL extension, and a separate table is written entirely with delta-rs (no Spark session at all) and then read back correctly by Spark — genuinely demonstrating, rather than asserting, that these engines interoperate on the same Delta table format. The knowledge base already holds this toolkit's config five-layer precedence, metadata-driven orchestration philosophy, engine-choice GB threshold, and deletion-vector/CDF mechanics in depth from prior sources, so this extraction is deliberately narrow: the runtime-portability technique itself (one profile dict driving both Runtime 1.3 and 2.0 without code branches), the ANSI-strategy-as-migration-testing-parameter idea, the real cross-engine interop demonstration, and the notebook's own measured Delta-operation proof points (deletion-vector metrics, CDF change counts, MERGE row counts) that go beyond what prior sources documented conceptually.
An 18-cell, first-party executable notebook that stands up a real local HTTP server to demonstrate two single-node, network-bound ingestion patterns end to end: a token-bucket-rate-limited, cursor-paginated API crawl that genuinely triggers and recovers from a 429 response with a Retry-After header, and a streamed zip-over-HTTPS download that extracts member JSON files one at a time without ever materializing the whole archive in memory. Both patterns land gzipped NDJSON into datetime-partitioned directories driven by a small `etl_api_config` metadata table, and both runs log an estimated CU cost so ingestion cost is attributable per entity. The knowledge base already holds this toolkit's generic-worker metadata schema and engine-choice threshold from prior sources, so this extraction captures only what is specific to this notebook: the token-bucket-plus-Retry-After backoff mechanics, the streamed zip-extraction technique, and the notebook's own measured CU comparison between the Python-notebook path actually used and an equivalent idle Spark node.
A 27-cell, first-party executable PySpark/Delta notebook demonstrating an incremental silver-to-gold pattern built on Change Data Feed plus a watermark held in an external SQL metadata store (Fabric SQL Database in production, SQLite locally) rather than in a Materialized Lake View or a Delta control table, chosen explicitly because MLV's Optimal Refresh requires append-only sources between refreshes and this pattern's source table receives real UPDATE and DELETE traffic. The knowledge base already holds this toolkit's general CDF mechanics — version-bounds-as-source-options versus post-read filtering, net-change collapse via window functions, CDF's VACUUM-retention exposure, and pure appends emitting no CDF rows — from prior sources, so this extraction captures only what is genuinely new: the external-SQL-store watermark variant and its explicit rationale relative to MLV's append-only constraint, the forward-only watermark upsert's concrete SQL guard, and the real executed proof (physical-plan pushdown verification, a zero-row idempotent second run, a zero-row proof that OPTIMIZE emits no CDF churn, and a triggered retention-trap exception) that grounds those already-known mechanics in genuine run output rather than restating the theory.
A six-cell, first-party executable PySpark/Delta notebook implementing data quality as a declarative rules table (rule_id, table_path, kind, JSON params, severity) evaluated by one generic runner that never changes when a rule is added, with two severity behaviors: error-severity failures raise and stop the pipeline, warn-severity failures quarantine only the offending rows to a sibling Delta table and let the pipeline continue. This rules-as-data architecture, its severity-gating behavior, and its quarantine mechanism are not covered by any prior-ingested toolkit source, so the full pattern is extracted here, grounded in the notebook's real saved execution: five rules of five distinct kinds ran against a 100,000-row table, four passed, one (an accepted_values check) failed as a warning and quarantined exactly 14,286 rows, and the pipeline completed successfully rather than halting.
A 15-cell, first-party executable notebook that downloads or reads a Spark event log — the raw SparkListener JSON-lines stream the Spark UI itself is rendered from — and parses it directly into stage-level metrics and findings, rather than going through the Fabric REST API. The knowledge base already holds this toolkit's REST-API surface (event log download path, History-Server-compatible endpoints, five common URL-parsing failure causes) from a prior source, so this extraction captures only the genuinely new material: the URL-shape normalization the notebook performs across six different ways a Fabric application URL can be pasted, and the stage-ranking and skew-detection algorithm itself, proved against a real Spark session run live in this notebook with a deliberately skewed join, with the actual measured skew ratio and timings the run produced.
A 14-cell, first-party executable notebook implementing a client for the Fabric Spark Monitoring REST API and, more specifically, the seven-step ranking and synthesis algorithm that turns five separate API calls (jobs, stages, task list, advisor advice, resource usage) into a single ranked diagnosis. The knowledge base already holds this toolkit's REST API endpoint paths, the job-group-id troubleshooting pattern, and the Advisor API from a prior source, so this extraction captures only what's new: the specific ranking logic (skew outranks spill), the corroboration step that cross-checks a spill finding against measured peak memory utilization before presenting it as confirmed, and the explicit 'no findings' branch that concludes data volume rather than an anti-pattern is the cause. All logic is proven end to end against schema-accurate simulated API responses (the notebook is explicit that the live HTTP calls themselves are untested here); the demonstrated ranking and synthesis logic is real, executed code, not just documented behavior.
A six-cell, first-party executable PySpark/Delta notebook implementing one generic ingestion worker that takes only an ENTITY_ID parameter and derives everything else — source path, target path, load type, merge keys, watermark column, column renames, and target table properties — from a per-entity JSON configuration row, intended to sit behind a runMultiple DAG or pipeline coordinator that fans out many entities to the same notebook. The knowledge base already holds this toolkit's metadata-driven orchestration philosophy in depth (querying metadata once per run rather than per row, forward-only watermark MERGE guarded by a greater-than comparison, and the runMultiple DAG pattern itself) from a prior source, so this extraction captures only the one element not previously documented: the concrete JSON configuration schema that makes a single worker notebook generic across entities, grounded in the notebook's real saved execution proving that schema drives a correct initial load, then a correct idempotent no-op on immediate re-run.
A six-cell, first-party executable PySpark/Delta notebook that walks every Delta table under a Lakehouse root using only Delta's own metadata APIs (DESCRIBE DETAIL, SHOW TBLPROPERTIES, DESCRIBE HISTORY) and writes the results as a versioned, append-only health report table rather than acting on anything itself. The notebook's saved execution shows it running end to end against four real Delta tables and computing a per-table flag verdict from concrete numeric thresholds. The knowledge base already holds this toolkit's broader maintenance-gating philosophy (conditional maintenance costs nothing when tables are healthy, table-property vs session-config, the 128/256/400MB layer targets) from prior sources, so this extraction is narrow: the specific single-table health-scoring flag taxonomy and its threshold logic, and the deletion-vector-pending counting technique, which were not previously captured at this level of implementation detail.
A five-cell, first-party executable PySpark/Delta notebook that reads the latest run of the health-audit notebook's report table and executes only the flagged actions, dispatching each of the audit's four flag codes to a specific Delta maintenance SQL statement, with a DRY_RUN parameter that prints the planned SQL instead of running it. Every action is logged to a separate Delta log table with a status field (PLANNED/OK/FAILED) and linked back to the audit run that triggered it. The knowledge base already holds this toolkit's broader conditional-maintenance philosophy and the underlying VACUUM/Direct-Lake and table-property-vs-session-config guidance from prior sources; the health-audit sibling source in this batch already captures the flag taxonomy and thresholds, so this extraction is narrow: the specific audit-report-to-action dispatch architecture, its dry-run-by-default safety convention, and the post-action self-verification step, none of which were previously captured.
A 14-cell, first-party executable notebook implementing a complete metadata-driven ingestion framework schema — etl_source, etl_entity, etl_watermark, etl_run_log, dq_rules, and a vw_active_entities view — as real, executed DDL and DML against a local SQLite stand-in that mirrors the intended Fabric SQL Database T-SQL exactly, so every statement in the notebook genuinely runs and is checked with assertions rather than only described. The knowledge base already holds this toolkit's abstract touch-metadata-twice orchestration principle and forward-only watermark guard from prior sources, so this extraction is narrow: the concrete five-table schema itself (with its specific column purposes), the pyodbc-plus-Entra-token connection mechanics for the real Fabric SQL Database path, and the notebook's own executed proof that a disabled entity is correctly excluded from the cached active-entity set and that an attempted backwards watermark write is silently rejected by the upsert guard.
A 17-cell, first-party executable notebook that measures — rather than assumes — whether the Native Execution Engine is actually accelerating a given query set, via an A/B wall-clock harness across nine real queries (five generic-shape, four investment-data-domain-shaped) plus a plan-text native-coverage scorer. The knowledge base already holds this toolkit's NEE fallback taxonomy and the ANSI-versus-native mechanism from prior sources, so this extraction captures only the genuinely new material: the actual measurement methodology (two-Environment A/B comparison, not a runtime toggle, because the toggle is explicitly documented as unreliable), the plan-text regex technique used to compute a 'native coverage percentage' from operator names, and the notebook's own honest real-run result — a fully JVM local run producing a negative delta on every query, which the notebook itself explains validates the measurement mechanism rather than demonstrating acceleration, since no native execution was actually available locally.
A 15-cell, first-party executable notebook demonstrating Fabric Workspace Monitoring — a read-only Eventhouse KQL database provisioned per-workspace that centrally collects logs and metrics from Fabric items — via a parameterized KQL query library (queries stored as data, run through one generic runner) covering job failure triage, duration-regression trend analysis, and Eventhouse query/ingestion health, plus the explicit gap that Spark application logs are not yet a native workspace-monitoring table and the documented diagnostic-emitter-to-Eventstream-to-Eventhouse route that closes it. This is genuinely new material not covered by prior toolkit sources, which document the Spark diagnostic emitter's Log-Analytics configuration and the Spark History Server/Advisor REST APIs but not the separate Workspace Monitoring Eventhouse feature, its table set, or its capacity-throttling-immunity property.
The final quarter of a large first-party interactive Spark internals toolkit. Structurally this batch is mostly reconciliation and restatement: a 'Settings Matrix' section explicitly states its own purpose is resolving prior sections' %%configure blocks into one baseline, a 'Recommended Defaults' section restates the same baseline with rationale, a coding-standards gallery and a maintenance runbook both point back to prior sections' anti-patterns and targets, and a glossary and expert-learning-path close out the document as pure reference/curriculum material rather than new claims. Three areas nonetheless contain genuinely new mechanics not captured in the three prior batches or the eight other already-ingested toolkit sources: a chained-Materialized-Lake-View gotcha (any node in an MLV dependency chain lacking CDF or full-refreshing propagates that status downstream), Spark 4.x's native VARIANT column type for semi-structured data with an explicit Spark-only interop warning, and a rate-limited-API/archive ingestion pattern (token-bucket limiting, Retry-After handling, splittable-file landing layout, and the CU cost of running network-bound crawls on Spark). A fourth area, 'Beyond Spark' governance/CI-CD/security, turned out to be almost entirely already covered by existing tier-1 OneLake-security and deployment-pipeline sources already in the knowledge base; only a narrow interoperability gotcha about combining shortcuts, SQL-endpoint RLS and OneLake security across workspaces, and a specific pre-migration ANSI-testing technique, survived that overlap check.
A 22-cell, first-party executable notebook that builds and runs SparkAutoConfigurator, a small library that profiles a real Delta/Parquet table and a cluster shape and produces a complete, basis-tagged, ready-to-apply set of Spark configuration recommendations — shuffle partitions, executor shape, memory fractions, broadcast threshold, and Fabric Efficient Scaledown wiring — rather than leaving those formulas to be copy-pasted from memory or a blog post. Every recommendation is tagged with a basis (SPARK_DEFAULT, FABRIC_DOC, or HEURISTIC, so the tool never dresses a rule of thumb up as a documented fact) and a scope (runtime, safe to apply via spark.conf.set on the current session, versus session_start, which requires a %%configure block before the session exists). The scope split is not asserted from memory: the notebook states it was discovered by empirically breaking a live local Spark 3.5.1 session (calling spark.conf.set on a session-start key raises AnalysisException[CANNOT_MODIFY_CONFIG]) and built the fix — an apply() method that only ever calls spark.conf.set on runtime-scoped keys, routing session-start keys to a separate session_start_snippet() renderer instead — directly into the tool. The knowledge base already holds this toolkit's engine-choice threshold, config five-layer precedence, and Runtime 2.0 workload-archetype recipes from prior sources, so this extraction focuses on what is genuinely new: the specific heuristic formulas themselves (shuffle-partition sizing, broadcast-threshold headroom, YARN-style executor-core sizing, autoscale bounds), the empirically-discovered and now-encoded config mutability boundary, the DESCRIBE DETAIL-first data-profiling strategy, and the tool's own real executed report and applied-configuration proof against a live session and a real 4-million-row dataset.
A community perspective on how Fabric data engineering is moving from imperative notebook-style pipelines toward declarative transformation patterns. The extracted claims focus on where declarative approaches help, where procedural PySpark remains appropriate, and how Fabric Materialized Lake Views, dbt, SQLMesh, and Spark declarative pipelines fit into that shift.
A community deep dive into Spark's DataFrameWriterV2 API (df.writeTo), framed as a shift from file-oriented to table-oriented write semantics. The extracted claims cover how V2's explicit method names remove the ambiguity of V1's mode()-based API, how table properties and per-write options are kept in separate maps, and which V2 capabilities (clustering, schema evolution, DataFrame-based MERGE) depend on newer Spark and Delta versions.
A community explanation of table compaction patterns for Spark and Delta Lake. The extracted claims cover auto compaction, optimized writes, manual compaction, small-file impacts, and maintenance tradeoffs for lakehouse workloads.
A community-authored introductory post on the Fabric Community's Data Engineering blog board, framing Materialized Lake Views (MLV) as a simplification over hand-rolled notebook/pipeline orchestration for medallion-architecture Lakehouse workloads, aimed at readers new to the feature.
A practical community review of Fabric Mirroring tradeoffs for operational replication into OneLake. The extracted claims emphasize current-state replication behavior, reseeding implications, operational limits, security and governance constraints, and when mirroring is a good fit versus when event-oriented or custom ingestion patterns are safer.
A first-party architectural guide to Microsoft Fabric's storage and distributed query tier, covering four areas: how OneLake resolves shortcut access through three identity-resolution paths behind a security broker; a summarized walkthrough of the Polaris distributed SQL engine's compilation-to-result pipeline, including the client-facing result-delivery protocol; a diagnostic T-SQL query pattern for checking Delta table file fragmentation before publishing a Direct Lake semantic model; and a PySpark pattern for querying a Fabric SQL Database's automatic OneLake replica alongside lakehouse data for HTAP-style analytics.
A community benchmark comparing three ways to query a database from a Fabric/Spark notebook: plain Pandas over ODBC, the Spark DataFrame API over JDBC, and Pandas-on-Spark over JDBC. The extracted claims cover the timing results for a small, single-record query and the underlying reason distributed tooling can lose to simple Pandas for low-volume lookups.
A technical community walkthrough of Delta Lake deletion vectors and their performance tradeoffs. The extracted claims explain merge-on-read mutation behavior, deletion-vector metadata, benchmark results, read overhead, OPTIMIZE requirements, and compatibility concerns for Fabric workloads.
A community comparison of three ways to run large numbers of concurrent Spark jobs in a lakehouse: one shared high-concurrency cluster, one dedicated job cluster per job, and multithreading many tasks inside a single job cluster. The extracted claims cover the tradeoffs of each pattern, why Python multiprocessing is a poor fit for Spark, and a benchmark comparing multithreading against per-job clusters.
A 2026 comparison of Apache Iceberg and Delta Lake table formats, covering their differing metadata architectures, the two formats' recent convergence efforts, and how OneLake exposes cross-format interoperability for Fabric lakehouses.
An independent practitioner walkthrough of Fabric Mirroring's snapshot-then-CDC mechanics, schema-change behavior, and setup pitfalls, aimed at engineers configuring and operating mirrored sources rather than evaluating the feature.
A practitioner-style walkthrough of Fabric Data Factory pipeline activities, trigger types, and production orchestration patterns, written and last verified in June 2026 by an independent tutorial site (not a Microsoft-affiliated publisher).
A tutorial-style community post walking through building a Fabric Warehouse Gold layer with CTAS and MERGE patterns, migrating off a Synapse dedicated SQL pool, and modeling conventions for star-schema fact/dimension tables.
A community troubleshooting note on Dataflow Gen2 error code 20302, a generic 'user configuration issue' message that the post argues actually masks either a stale owner identity or a CI/CD deployment defaulting the dataflow's type property to a legacy value.
An independent tutorial-site walkthrough of Dataflow Gen2 performance features (Fast Copy, partitioned compute) with a worked cost-optimization example, last verified June 2026.
An independent practitioner tutorial walking through Fabric Eventstream's pipeline model, transformation operators, windowing types, and destination routing, with an emphasis on operational tuning and delivery-semantics gotchas rather than feature enumeration.
A community blog post walking through when to choose a Fabric Lakehouse versus a Fabric Warehouse, framed as a decision tree (dev-tooling fit, transactional needs, data shape) rather than a size-based choice, plus notes on how the two engines interoperate inside one workspace.
A community/blog walkthrough of the Real-Time Intelligence Operations agent: what data it can monitor, how its detection-to-action loop is timed and gated by human approval, its concrete data-source and regional limits, and how it differs from a Fabric data agent. The post links directly to Microsoft Learn's operations-agent docs (overview, limitations, transparency note, billing) as its primary sources.
An unvetted third-party blog explains the 'Prep data for AI' workflow in Power BI: configuring an AI data schema, writing AI instructions, and setting verified answers so that Copilot gives more grounded responses, plus a testing/validation loop for confirming those changes actually take effect.
A practitioner-oriented walkthrough of Fabric capacity economics: how Capacity Units are consumed, how throttling escalates, and a sequence of cost-optimization moves the author recommends before resizing a capacity, built around Microsoft's own throttling/optimization documentation.
A community/blog step-by-step walkthrough of building and publishing a Fabric data agent, largely restating Microsoft Learn's setup guide but adding specifics on how a published agent is consumed outside the Fabric chat UI — its callable API endpoint shape, the named Microsoft Foundry integration tool, Copilot Studio/Teams consumption paths, and available client SDK languages.
An independent tutorial-site walkthrough of Fabric's governance surface — the OneLake catalog's Explore/Govern/Secure tabs, a medallion-architecture governance-gate pattern, and a routine governance operating cadence — last verified June 2026.
A community blog post collecting practitioner-level optimization tactics for Fabric Warehouse: partitioning and file-compaction habits, when Direct Lake silently falls back to DirectQuery, capacity-throttling stages and their symptoms, and a set of recurring error codes with fixes.
A community/blog roundup of Fabric IQ as it stood after Microsoft Build 2026, mostly restating Microsoft Learn concepts but adding a same-page inventory of which IQ-family items reached GA at Build 2026 (Operations agent, Graph in Fabric, Planning) versus which stayed in preview (Ontology), plus a list of the June 2026 wave of Fabric Data Agent feature previews and where Fabric IQ shows up across the wider Microsoft agent ecosystem.
An independent practitioner tutorial covering Lakehouse provisioning, the medallion-architecture layering pattern, SQL-endpoint visibility rules, Direct Lake fallback diagnosis, and table-maintenance guidance, oriented toward engineers building and operating Lakehouses rather than evaluating the concept.
An independent tutorial-series overview walking through Fabric's workload lineup, OneLake storage model, compute engines, and a snapshot of platform news as of mid-2026, aimed at readers getting oriented on the whole suite rather than one workload.
A review-style piece assessing how ready Fabric is for production enterprise analytics as of mid-2026, covering governance tooling changes, mirroring/eventing capabilities, and operational patterns (CI/CD gates, retry/idempotency, monitoring cadence) that the author says separate stable deployments from fragile ones.
An independent comparison blog positioning Microsoft Fabric as the SaaS successor to Azure Synapse Analytics' PaaS model, itemizing feature gaps (notably around Spark and dedicated SQL pools) and outlining migration paths between the two platforms.
An independent comparison blog contrasting Microsoft Fabric's SaaS analytics platform with Databricks' lakehouse platform across architecture, security, BI integration, compute engines, and hybrid deployment patterns.
An independent explainer on Rayfin, an open-source SDK/CLI announced at Microsoft Build 2026 that provisions a governed, code-first application backend (SQL database, auth, GraphQL API, static hosting) directly onto Microsoft Fabric capacity, underpinning the broader Fabric Apps initiative.
A community troubleshooting walkthrough for a 403 'OneLake shortcut authorization failed' error that appears when a lakehouse shortcut spans two workspaces. It maps the error to three distinct root causes and gives a matching fix for each.
An unvetted third-party blog walks through using Power BI's Copilot-based DAX assistant across three surfaces (Desktop query view, model view, and web modeling), with worked prompt examples for common measure patterns and a rundown of documented request/character limits and known blind spots.
A community explainer on PBIR, the newer internal storage format for Power BI report definitions inside a Power BI Project (.pbip), contrasted with the legacy single-file report.json format, plus its stated rollout status and how to enable or opt out of it.
A practitioner walkthrough of Fabric notebooks for data transformation, covering runtime specifics, capacity/cold-start behavior, production patterns (schema enforcement, idempotent merges, incremental ingestion), and a decision guide for when to pick notebooks over Dataflow Gen2 or the SQL warehouse.