Integrating Microsoft Fabric with an existing multi-stack estate

Goal#

Bring Microsoft Fabric alongside an existing non-Fabric estate — Databricks, Snowflake, ADLS Gen2, on-prem SQL, and AWS/GCP object storage — without a wholesale migration. For each source, decide deliberately whether to virtualize it in place, replicate it, or copy it, keep the data in one open format that every engine can read, expose Fabric data back outward, and govern the whole thing consistently.

Architecture#

The hub is OneLake: Fabric's single tenant-wide logical data lake, built on Azure Data Lake Storage Gen2, and provisioned exactly once per tenant with no storage to manage ^[S1]. Every Fabric data item — lakehouse, warehouse, SQL database, mirrored database — persists its tabular data in OneLake as open Delta Parquet, regardless of which engine wrote it ^[S1], and Delta Lake is the standard table format across all Fabric workloads, which is what lets different engines read each other's output without conversion ^[S11]. That one-copy design is the backbone of multi-stack integration: it means you integrate into a shared open store, not into a proprietary silo.

The integrate-in-place vs move decision#

Three mechanisms connect an external estate to OneLake. Choosing among them is the central design decision.

1. Virtualize — OneLake shortcuts (zero copy)#

A OneLake shortcut is an embedded reference that points from a shortcut path to a target path in another storage location, behaving like a symbolic link: deleting the shortcut leaves the target untouched, while moving, renaming, or deleting the target can break it ^[S3]. Shortcuts mount existing external storage — Azure Data Lake Storage, Amazon S3, and Google Cloud Storage — as zero-copy references, so cross-cloud data can be analysed in Fabric without ETL or migration ^[S2]. The documented external targets are ADLS, Amazon S3, and Dataverse, and shortcuts make those remote files and folders appear as if stored locally, enabling virtual data products without duplication ^[S1].

Shortcuts are transparent to consumers: through the OneLake API they look like ordinary folders, so Spark, SQL, Real-Time Intelligence, Analysis Services, and non-Fabric ADLS-compatible applications all query shortcut data without shortcut-specific logic ^[S3]. In a lakehouse's Tables folder, when a shortcut target holds Delta Parquet data the lakehouse automatically syncs the metadata and registers the folder as a table ^[S3]. ADLS and S3 shortcuts delegate authorization to a cloud connection bound at shortcut creation, and that binding is a privileged operation ^[S3].

Use it when the source stays authoritative and you want no second copy and no pipeline. Inference: shortcuts are the default first choice for object storage precisely because they add no copy, no schedule, and no failure surface of their own.

2. Replicate — Fabric Mirroring (managed near-real-time copy)#

Fabric Mirroring is a fully managed, serverless replication service that continuously brings data from external operational and analytical databases into OneLake, converting it to Parquet in Delta Lake format, without users building or maintaining ETL pipelines ^[S5]. Fabric supports three distinct mirroring approaches: database mirroring (physically replicates table data), metadata mirroring (synchronizes only catalog structure via OneLake shortcuts, leaving data in place), and open mirroring (any developer writes change data to a mirrored database item via public APIs and a landing zone URL) ^[S5].

The supported database mirroring sources (GA or preview as of June 2026) include Azure Cosmos DB, Azure SQL Database, Azure SQL Managed Instance, Azure Database for PostgreSQL, Azure Database for MySQL (preview), SQL Server, Snowflake, Oracle, SAP, Google BigQuery (preview), Fabric SQL database, and open mirrored databases; Azure Databricks is supported via metadata mirroring, and Dremio via metadata mirroring (preview) ^[S5]. This distinction matters for a Databricks/Snowflake estate: Snowflake can be database-mirrored into OneLake, while Databricks is integrated via metadata mirroring, which leaves the data in place and points at it with shortcuts ^[S5].

For database and open mirroring, the replicator polls a Fabric landing zone at high frequency and merges incremental Delta files immediately, so changes can propagate into OneLake in as little as 15 seconds under optimal conditions ^[S5]. Mirroring storage is free up to one terabyte per capacity unit — an F64 includes 64 TB of free mirroring storage — and the background replication compute itself is free and does not consume capacity units; standard capacity consumption applies only to downstream queries ^[S5]. Because mirrored data lands as Delta Lake in OneLake, Power BI semantic models can connect in Direct Lake mode, and cross-database T-SQL queries can span mirrored databases, warehouses, and lakehouse SQL endpoints in one statement ^[S5].

Use it when you need a governed, near-real-time local replica of a supported database. Watch: mirroring delivers a current-state replica, not a change stream or an immutable store — stopping and restarting causes a full reseed from the present source state ^[S6], and it does not constitute a disaster-recovery backup, since deletion or corruption at the source eventually replicates into the mirror ^[S6].

3. Copy — Data Factory pipelines#

Mirroring is surfaced from Data Factory as one data-movement method, keeping a near-real-time replica of an operational database inside OneLake ^[S15]; alongside it, data can enter a lakehouse through Spark notebooks, pipeline copy activities, Spark job definitions, Dataflows Gen2, or OneLake shortcuts that reference external data in place ^[S4].

Use it when you must transform, curate, or reshape data on the way in, or when the source is neither a shortcut target nor a supported mirroring source. Inference: copy is the fallback that trades duplication and orchestration ownership for full control over shape and schedule; prefer shortcut or mirror first to avoid a second copy.

Open Delta / Parquet interop with external engines#

OneLake is layered on ADLS Gen2 and exposes the same ADLS Gen2 APIs and SDKs, so existing ADLS-aware applications such as Azure Databricks can read and write OneLake data without code changes ^[S1]. The one-copy design lets multiple engines (T-SQL, Spark, Analysis Services, and others) operate over the same physical Delta Parquet data, so teams do not duplicate datasets per engine or get locked into whichever engine loaded the data first ^[S1]. Warehouse tables are stored as Delta and automatically published to OneLake Files in Delta format, making the same tables readable by external engines without export jobs ^[S7].

Fabric's V-Order write-time Parquet optimization does not compromise this openness: V-Ordered files remain fully compliant with the open-source Parquet specification, so any standard Parquet reader can consume them — non-Fabric engines simply do not get the read acceleration ^[S12]. Inference: Databricks or Snowflake reading OneLake see ordinary open Delta/Parquet; the V-Order benefit is Fabric-side only and costs external readers nothing.

A tenant admin setting controls whether applications running outside Fabric (for example custom ADLS API clients or OneLake file explorer) can reach OneLake data; switching it off still leaves internal workloads like Spark and Data Warehouse able to access the data ^[S9] — the lever that gates external-engine reach.

Exposing Fabric data outward#

• SQL analytics endpoints. Every lakehouse provisions a read-only T-SQL endpoint at creation, exposing its Delta tables ^[S14]; warehouses, Fabric SQL databases, and mirrored databases likewise expose SQL analytics endpoints on the same warehouse engine ^[S7], and cross-database T-SQL can span mirrored databases, warehouses, and lakehouse endpoints via three-part naming ^[S5]. Note the endpoint is strictly read-only over its autogenerated tables ^[S14].
• Fabric API for GraphQL. Fabric auto-generates a GraphQL schema, resolvers, and a live endpoint with no custom backend code ^[S8]. Its supported sources include Fabric Warehouse, Fabric SQL database, Lakehouse (via its SQL analytics endpoint), Azure SQL Database, and Fabric mirrored databases — including mirrored Azure Databricks and Snowflake — accessed through their SQL analytics endpoints ^[S8]. Microsoft also documents connecting AI agents to Fabric data through the GraphQL endpoint via the Model Context Protocol ^[S8].
• Cross-tenant data sharing. Fabric lets an organization expose live governed datasets to external Microsoft Entra tenants without copying; recipients read the data in place and source-side governance stays in force ^[S2]. Metadata mirroring supports this via OneLake external data sharing, creating a read-only shortcut in the consumer tenant so both organizations query the same live source data ^[S5].
• Purview / Atlas APIs. Metadata scanning exposes admin scanner REST APIs that external cataloging tools call to retrieve item-level metadata ^[S10], and Purview Data Map can scan Fabric items to extract metadata and lineage, exposing it via Atlas APIs for external services ^[S10].

Performance#

• Cross-cloud shortcut caching. Shortcut caching stores files read through external shortcuts in a per-workspace cache to cut cross-cloud egress: retention is configurable from 1 to 28 days and resets on each access, a newer remote version triggers a refresh, unused files are purged after the window, and individual files over 1 GB are never cached; the cache currently covers Google Cloud Storage, S3, S3-compatible, and on-premises data gateway shortcuts ^[S3]. Separately, cross-cloud shortcut reads benefit from intelligent caching that keeps frequently used data closer to Fabric compute, improving locality and reducing egress cost ^[S2].
• Warehouse / SQL endpoint caching is fully transparent and always on: queries cache whatever data they touch regardless of origin, including OneLake shortcuts and even shortcuts pointing at non-Azure storage ^[S7].
• Mirroring freshness. Changes can propagate into OneLake in as little as 15 seconds ^[S5], but the SQL analytics endpoint adds roughly 30–60 seconds of synchronisation delay on top of the underlying Delta replication latency — reading directly from the Delta layer via Spark avoids that extra lag when the freshest data is required ^[S6].

Governance & security#

Governance in Fabric is centralized in the OneLake Catalog and powered by built-in Purview: permissions, sensitivity labels, and auditing are applied automatically and inherited across Fabric items, and these controls remain enforced even when data is shared across tenant boundaries ^[S10]. Governance in OneLake is hierarchical — tenant-level policies automatically cover any data landing in OneLake, while workspaces distribute ownership and access ^[S1].

For policy enforcement across stacks, the authorized engine model lets external query engines register with OneLake, pull security policy definitions and precomputed effective-access decisions from OneLake APIs, and enforce table, row-, and column-level security at query time — keeping OneLake the single source of truth for policy across Fabric and third-party engines ^[S9]. OneLake authenticates exclusively with Microsoft Entra ID for users and service principals ^[S9], and data is encrypted at rest by default with the option of customer-managed keys ^[S9].

Critical cross-stack governance caveat. Source-defined row-level security, column-level security, and dynamic data masking are not carried forward into a mirrored copy — teams must rebuild all security policies at the Fabric layer before exposing mirrored data to end users ^[S6]. Inference: replication moves data but not its access controls, so a mirror is not governed until you re-establish policy in Fabric.

Cost & capacity#

Mirroring storage is free up to 1 TB per capacity unit (F64 = 64 TB of free replica storage) and mirroring's replication compute is free, so the metered cost of a mirror is the downstream queries — SQL, Power BI, or Spark — that read it ^[S5]. Shortcuts add no storage cost at all (zero copy) and their main variable cost is cross-cloud egress, which the shortcut cache is designed to reduce ^{[S3] [S2]}. Inference: on cost alone the ordering is shortcut (no copy) before mirror (free storage/compute within limits) before copy (a full second dataset plus pipeline runtime).

Risks & anti-patterns#

• Network isolation can invalidate a mirroring plan. When Block Public Internet Access is enabled at the tenant, the majority of native database mirroring connectors pause and stop replicating — only Open Mirroring, Cosmos DB, Azure SQL Managed Instance, and SQL Server 2025 continue ^[S6]. On-premises data gateways cannot register at all under that setting; the only replacement is a VNet data gateway requiring ExpressRoute or a site-to-site VPN ^[S13]. Check network isolation before committing to mirroring.
• Type-handling gaps in mirroring. PostgreSQL mirroring silently drops unsupported column types (geometric, network address, range, JSON/JSONB, XML, INTERVAL) rather than blocking the table, so downstream consumers may receive incomplete rows without any error ^[S6]; an Oracle NUMBER column without explicit precision/scale blocks the whole table with InvalidDecimalPrecision, remedied by upgrading the gateway ODP driver, not the source schema ^[S6].
• Mirror ownership is non-transferable. A mirrored database is permanently associated with its creator's Fabric identity; if that person leaves, the item must be deleted and recreated — use a service principal from the outset ^[S6].
• Standing up separate physical lakes per business group is an anti-pattern in Fabric, recreating the duplication and overhead the single tenant-wide OneLake with workspaces and shortcuts is designed to eliminate ^[S1].
• A space in a table-shortcut name breaks Delta recognition — OneLake will not see such a shortcut as a Delta table ^[S3].
• Do not rely on OneLake security roles to constrain workspace Admins/Members/Contributors — those roles bypass OneLake security and can read/write all item data; granular roles only constrain Viewers ^[S9].

Assumptions#

• The estate includes at least some object storage (ADLS/S3/GCS) reachable by network, some operational/analytical databases on the supported mirroring source list, and a desire to avoid wholesale migration — matching the shortcut/mirror/copy split above.
• Downstream consumption is a mix of Fabric-native engines and external engines (Databricks/Snowflake) reading OneLake over open Delta.
• Standard Fabric capacity (F SKU) is available; specific SKU sizing is out of scope here.

Open questions#

• Exact per-connector mirroring latency and change-volume behaviour beyond the documented ~15s best case and the polling/backoff model ^[S5] is not specified in the verified claims for each source; validate against the specific Snowflake/Oracle/SAP source in your estate.
• Whether a given non-Fabric engine can write to OneLake is confirmed only for ADLS Gen2-compatible engines such as Databricks ^[S1]; Snowflake's write path to OneLake is not covered by the verified claims — treat as an open question for the knowledge-curator rather than assuming.
• Detailed private-endpoint / managed-VNet topologies for cross-cloud shortcut traffic are beyond the claims cited here; if network-level isolation is mandated, revisit the mirroring/gateway constraints ^{[S6] [S13]}.

Source legend#

Internals#

Architecture & design#

OneLake virtualizes the whole tenant as one giant ADLS storage account: each workspace surfaces as a container and each data item as a folder inside its workspace container ^[S1] — the addressing model that makes shortcuts "look like ordinary folders" to Spark, SQL, and non-Fabric ADLS-compatible apps ^[S3]. Shortcuts have documented structural limits: an item can hold up to 100,000 shortcuts, a single OneLake path supports at most 10 shortcuts, direct shortcut-to-shortcut chains max out at 5, and names/paths cannot contain '%', '+', or non-Latin characters ^[S3]. Internal OneLake shortcuts can target lakehouses, warehouses, KQL databases, SQL databases, mirrored databases, and semantic models across items and workspaces, authorized with the calling user's own identity ^[S3].

How it works internally#

Shortcut caching is the mechanism that makes cross-cloud virtualization performant: files read through external shortcuts are stored in a per-workspace cache with configurable 1–28 day retention that resets on each access; a newer remote version triggers a refresh from source, unused files are purged after the retention window, and individual files over 1 GB are never cached — the cache currently covers Google Cloud Storage, S3, S3-compatible, and on-premises data gateway shortcuts ^[S3]. Identity passthrough has a subtlety: when shortcuts are reached via Power BI semantic models in DirectLake-over-SQL mode or via T-SQL delegated-identity mode, the identity passed to the shortcut target is the calling item owner's, not the end user's; DirectLake-over-OneLake or T-SQL user-identity mode restores per-user passthrough ^[S3]. On the replication side, the internal replicator engine polls a Fabric landing zone at high frequency and immediately merges incremental Delta files into the target Delta table, with backoff logic that scales polling down when source activity is low and back up as change volume rises ^[S5]; metadata mirroring moves no data, synchronizing only catalog names, schemas, and definitions and using shortcuts to point at source data in place, so its latency is governed by source access time and shortcut performance rather than a copy pipeline ^[S5].

Performance characteristics#

Mirroring propagates changes into OneLake in as little as 15 seconds under optimal conditions ^[S5], while the SQL analytics endpoint adds roughly 30–60 seconds of synchronisation delay on top of the underlying Delta replication latency — reading directly from the Delta layer via Spark avoids that extra lag ^[S6]. Warehouse and SQL-endpoint caching is transparent and always on, caching whatever data a query touches including shortcuts to non-Azure storage ^[S7]. Mirroring automatically runs vacuum on its Delta tables to remove files no longer referenced in the Delta log, with a default retention of one day for databases created after mid-June 2025 (seven days for older ones), adjustable via portal or REST API to balance storage cost against time-travel range ^[S5]. Deeper OneLake Delta storage internals — optimistic-concurrency commit protocol, checkpoint compaction, and min/max data-skipping statistics — sit below this design's scope and are covered in the OneLake-specific material; Coming soon for a multi-stack-specific treatment of how those internals affect concurrent external-engine writes.