Multi-cloud data architecture with Fabric Data Factory

Goal#

Bring data that physically lives across several clouds — BigQuery, Snowflake, Databricks, Amazon S3, Google Cloud Storage, and Azure — into one governed Microsoft Fabric analytics estate, without re-platforming every system, and let each source pick the integration mechanism that fits its freshness, cost, and copy-tolerance profile. Fabric Data Factory is the single orchestrator and OneLake is the single landing zone.

Why Fabric Data Factory for multi-cloud#

Data Factory is Fabric's data integration workload: it connects to data sources, moves data, transforms it, and orchestrates movement and transformation from a single place, with connectivity to more than 170 data sources spanning cloud, multicloud, and on-premises systems reached through gateways ^[S2]. It is framed as a cloud-native, serverless orchestration environment that auto-scales its underlying compute and includes built-in CI/CD and deployment capabilities for multi-cloud pipeline solutions ^[S1].

A concrete multi-cloud orchestration pattern is a single Fabric pipeline that extracts from Google BigQuery, processes the data in a Databricks notebook, and merges the result into a Snowflake table — all under one pipeline and scheduler ^[S1]. Logic-heavy cross-system workflows are supported through If-Condition, Until, and ForEach iterators ^[S1], and pipelines can start automatically from event-based triggers such as a new file landing in a Lakehouse or a Blob storage container rather than only on a schedule ^[S1].

Inference: This means Fabric can act as the control plane over other clouds without those clouds giving up their own engines — the pipeline reaches into BigQuery/Snowflake/Databricks and coordinates them, rather than forcing a migration off them.

Recommended architecture#

1. Sources (other clouds)#

Operational and analytical data stays where it is: BigQuery, Snowflake, Databricks, Amazon S3, Google Cloud Storage, and Azure stores. Data Factory's 170+ connectors reach these across cloud and multicloud boundaries ^[S2].

2. Ingestion choice, per source#

Data Factory offers three distinct data-movement options: Copy job for simplified movement with built-in delivery styles (bulk, incremental, and change-data-capture replication), Copy activity for fully customizable movement with manual control of parallelism, and Mirroring for near-real-time database replication into OneLake ^[S2]. On top of these, OneLake shortcuts let you analyze data that physically resides in Snowflake, Amazon S3, or Google Cloud Storage without copying it into OneLake, reducing both latency and storage cost ^[S1]. The decision guidance below (and the decision-tree diagram) maps each source to one of these mechanisms.

• Shortcut (zero-copy, query in place). OneLake shortcuts are references to data held in other locations, including stores like ADLS, Amazon S3, or Dataverse, and make remote files and folders appear as if stored locally, enabling virtual data products without duplication ^[S6]. Shortcuts are transparent to consumers — through the OneLake API they look like ordinary folders, so Spark, SQL, Real-Time Intelligence, Analysis Services, and non-Fabric ADLS-compatible applications can all query shortcut data without shortcut-specific logic ^[S7]. ADLS and S3 shortcuts delegate authorization to a cloud connection bound at shortcut creation, and binding is a privileged operation ^[S7].

• Mirroring (near-real-time CDC replication). Fabric Mirroring is a fully managed, serverless replication service that continuously brings data from external operational and analytical databases into OneLake, converting it to Parquet files in Delta Lake format, without requiring users to build or maintain ETL pipelines ^[S5]. Mirroring is positioned as the mechanism for replicating change data from Snowflake and BigQuery into Fabric to support near-real-time AI and BI scenarios, distinct from scheduled Copy jobs used for broader cross-platform batch movement ^[S1]. Supported database mirroring sources (GA or preview as of June 2026) include Snowflake and Google BigQuery (preview), among others; Azure Databricks is supported via metadata mirroring ^[S5].

• Copy Job / Copy activity (batch and event-driven). Copy job is described as Fabric's simplest and most scalable mechanism for moving data across multiple clouds and tenants ^[S4]. Data Factory pipelines with Copy activity are Microsoft's recommended mechanism for large-scale one-time migrations and for high-volume recurring ingestion, on the grounds that the engine is serverless, performant at scale, and cost-effective ^[S3]. Copy Job execution can be event-driven: Activator is a real-time engine that monitors data and reacts to events, in contrast to Data Factory's traditional fixed-schedule execution model ^[S4].

3. OneLake landing#

Every Fabric tenant is provisioned with exactly one OneLake, and no separate storage resources need to be provisioned or managed ^[S6]. All Fabric data items, including lakehouses and warehouses, automatically persist their tabular data in OneLake using the open Delta Parquet format, regardless of which engine wrote the data ^[S6]. The one-copy design lets multiple Fabric engines (T-SQL, Apache Spark, Analysis Services, and others) operate over the same physical Delta Parquet data, so teams do not duplicate datasets per engine ^[S6]. Mirroring lands its replicas in the same Delta Lake format ^[S5], and shortcuts surface external data in place ^{[S6] [S7]} — so all three ingestion paths converge on one logical lake.

4. Downstream serving#

Because mirrored data lands in Delta Lake format in OneLake, Power BI semantic models can connect to it using Direct Lake mode, enabling near-real-time reporting on operational data without importing or duplicating the data into the model ^[S5]. Cross-database T-SQL queries can span mirrored databases, Fabric warehouses, and lakehouse SQL analytics endpoints in a single statement using standard three-part naming, because all of these item types expose data through OneLake ^[S5]. Once multi-cloud data lands in OneLake it can also be made available to Fabric IQ, extending Fabric's AI-oriented capabilities across models and agents built on that unified data ^[S1].

Decision guidance: shortcut vs mirroring vs copy#

Microsoft publishes a data-movement decision guide to choose between Copy job, Copy activity, and Mirroring for a given scenario ^[S2]. Layering the shortcut option on top, the practical rules from verified claims are:

• Prefer a shortcut when the data can be queried in place and you want to avoid a copy: shortcuts reduce both latency and storage cost by not copying data into OneLake ^[S1], and cover Snowflake, Amazon S3, and Google Cloud Storage among others ^{[S1] [S6]}.
• Choose Mirroring when you need continuous change data from an operational or analytical database for near-real-time BI/AI — the case Mirroring is explicitly positioned for against Snowflake and BigQuery ^{[S1] [S5]}.
• Choose a Copy job for broader cross-platform batch movement ^[S1], and specifically as the simplest, most scalable way to move data across multiple clouds and tenants ^[S4]; make it event-driven with Activator when a schedule is too coarse ^[S4].
• Choose the Copy activity when you need fully customizable movement with manual control of parallelism ^[S2] — for example large one-time migrations or high-volume recurring ingestion ^[S3].

Inference: Freshness and copy-tolerance are the two axes that separate these. Shortcuts win when zero-copy is acceptable and the source engine can serve the query; Mirroring wins when you need a continuously fresh local Delta copy; Copy jobs/activities win when movement is periodic or transformational and a materialized copy is the point.

Governance & security#

Governance in OneLake is hierarchical: tenant-level security, compliance, and data-management policies automatically cover any data landing in OneLake, while workspaces distribute ownership and access policies ^[S6]. OneLake security operates on two planes — a control plane (what users can do) and a data plane (what data users can read) ^[S10] — and a OneLake security role can carve out specific rows or columns, enabling table, folder, row-level, and column-level security on lake data ^[S10].

For cross-cloud and cross-tenant sharing, external shortcuts to non-Fabric stores such as AWS S3 and Google Cloud Storage are categorized as a form of delegated shortcut, where the delegated identity is typically an account key or system identity native to the external system ^[S9]. Delegated OneLake Shortcuts add a second authentication mode where access flows through one configured connection identity — an organizational account, service principal, or workspace identity — instead of each signed-in user's identity ^[S8]. For delegated shortcuts, access is computed as the intersection of what the delegated identity is permitted to see and what OneLake security grants the end user, so an end user can never be given more access than the delegated identity itself was authorized for ^[S9].

Design guidance from the verified claims: choose pass-through shortcuts when the goal is letting multiple teams reuse a shared dataset while preserving strict source-managed per-person authorization; choose delegated shortcuts when the goal is broader curated distribution where centralizing per-user provisioning on the source does not scale ^[S9]. The recommended pattern for delegating access to thousands of downstream users is to represent each consuming team or region with its own service principal, scope it to only its slice of the data, and let the team's owners manage individual end-user permissions behind that service principal ^[S9].

Because Fabric integrates with Microsoft Purview, data lineage can be tracked end-to-end from an external source such as Snowflake through to the AI model that ultimately consumes the data ^[S1].

Inference: In a multi-cloud estate the identity boundary is the hard part — pass-through preserves the source's own per-user rules, while delegated centralizes trust in one connection identity and then re-derives per-user access via OneLake security intersection. Pick per source based on who owns authorization.

Cost & capacity#

Mirroring storage for replicated data is free up to one terabyte per capacity unit purchased — an F64 capacity therefore includes 64 TB of free mirroring storage exclusively for replicas — with charges applying if that limit is exceeded or when the capacity is paused ^[S5]. Background compute used by the mirroring replication process itself is free and does not consume Fabric capacity units; standard capacity consumption applies to downstream analytical workloads (SQL, Power BI, Spark) that query the mirrored data ^[S5]. Shortcuts avoid a copy entirely, reducing storage cost ^[S1], and shortcut caching stores files read through external shortcuts in a per-workspace cache to cut cross-cloud egress, with retention configurable from 1 to 28 days ^[S7].

Inference: The cheapest-to-store option is a shortcut (no copy), the cheapest-to-replicate is Mirroring (free replication compute, generous free storage), and Copy jobs/activities cost capacity for the movement itself — so the ingestion choice is also a cost lever, not only a freshness one.

Risks & anti-patterns#

• Tuning copy throughput before measuring the bottleneck. Raising intelligent throughput optimization or parallelism cannot push throughput past an external limit such as the source store's IOPS or the network path ^[S3]; end-to-end throughput is bounded by the minimum of source serve rate, destination absorb rate, and network bandwidth ^[S3].
• Switching a shortcut's authentication mode in place. Moving a shortcut between pass-through and delegated is not an in-place setting change — the documented procedure is to delete and recreate it ^[S8].
• Relying on OneLake security roles to restrict workspace Admins/Members/Contributors. Those roles bypass OneLake security entirely; granular roles only constrain Viewers and users holding item Read permission ^[S10].
• Naming a table shortcut with a space. Delta does not allow spaces in table names, so OneLake will not recognize such a shortcut as a Delta table ^[S7].
• Standing up separate physical data lakes per business group. In Fabric this recreates the resource-management overhead and duplication that one tenant-wide OneLake with workspaces and shortcuts is designed to eliminate ^[S6].

Assumptions#

• The organization already holds (or will provision) a Fabric capacity sized for downstream query workloads; SKU sizing for those workloads is out of scope here.
• Sources listed (BigQuery, Snowflake, Databricks, S3, GCS, Azure) match the verified connectivity/mirroring/shortcut claims; other sources should be checked against the current connector, mirroring-source, and shortcut-target lists.
• Preview features (e.g. BigQuery database mirroring, delegated shortcuts) are acceptable for the target rollout timeline ^{[S5] [S8]}.

Open questions#

• Cross-cloud shortcut query latency versus a mirrored local copy is a known knowledge gap (already tracked in the ingestion queue) — no verified benchmark exists yet, so the Performance sub-section below is a placeholder.
• Whether a given BigQuery/Snowflake workload should be shortcut-in-place or mirrored depends on query latency tolerance that the current claims do not quantify.

Alternatives#

1. Zero-copy-first (shortcut-led). Default every source to a OneLake shortcut and only materialize (Mirror or Copy) where in-place query latency or source load is unacceptable. Minimizes storage and duplication ^{[S1] [S6]}; leans on source engines to serve queries.
2. Replicate-first (Mirroring/Copy-led). Land continuously changing databases via Mirroring and bulk/periodic data via Copy jobs, using shortcuts only for large immutable file stores in S3/GCS. Maximizes local query performance and isolation from source availability at the cost of stored copies ^{[S1] [S5] [S3]}.

Source legend#

Internals#

Architecture & design#

The Copy activity runs on a serverless architecture that exposes parallelism at three levels: control flow can launch many copy activities concurrently (for example inside a ForEach loop), each activity can scale its own compute via intelligent throughput optimization (ITO), and within one activity multiple threads read and write in parallel ^[S3]. To scale ingestion beyond one Copy activity, fan out at the orchestration layer — drive multiple Copy activities in parallel from a ForEach loop, partitioning the workload by table or file range ^[S3]. OneLake itself virtualizes the whole tenant as one giant ADLS Gen2 storage account: each workspace surfaces as a container and each data item as a folder inside it ^[S6], which is why external shortcuts and mirrored replicas can share one addressing model.

How it works internally#

For database and open mirroring, the internal replicator engine polls a Fabric landing zone at high frequency and immediately merges incremental Delta files into the target Delta table; changes can propagate in as little as 15 seconds under optimal conditions ^[S5]. The engine employs backoff logic that scales down polling frequency when source activity is low and adjusts back up as change volume rises ^[S5]. Metadata mirroring, by contrast, does not move data — it synchronizes catalog names, schemas, and table definitions using OneLake shortcuts to point at source data in place, so its latency is governed by source access time and shortcut performance, not a copy pipeline ^[S5]. For delegated shortcuts, effective access is computed as the intersection of the delegated identity's permissions and the end user's OneLake grants ^[S9], and external query engines can register with OneLake to pull precomputed effective-access decisions and enforce table/row/column security at query time ^[S10]. ITO and degree of parallelism are orthogonal: ITO sizes the run's compute, while parallelism caps concurrent reader/writer threads ^[S3].

Performance characteristics#

Coming soon — the cross-cloud shortcut query-latency gap (shortcut-in-place vs a mirrored local Delta copy for BigQuery/Snowflake sources) is a known knowledge gap already tracked in the ingestion queue; no verified benchmark claim exists yet. What is grounded today: end-to-end copy throughput can be estimated as the minimum of source serve rate, destination absorb rate, and network bandwidth ^[S3]; shortcut caching (retention 1–28 days, per-workspace, files over 1 GB never cached) cuts cross-cloud egress on repeat reads ^[S7]; and mirroring can reach ~15-second propagation under optimal conditions ^[S5].