OneLake: The Unified Data Lake at the Heart of Microsoft Fabric

What Is OneLake and Why Does It Matter?#

Every Microsoft Fabric tenant is provisioned with exactly one OneLake — no second instance can be created, and the one you have cannot be deleted ^[S1]. It is Fabric's single logical data lake, built on Azure Data Lake Storage Gen2, and every compute engine in Fabric is preconfigured to use it as its native store from the moment the tenant exists ^[S2]. Microsoft's own comparison is to how Office apps quietly save into a user's OneDrive: the storage layer is simply present and unified, and nobody has to provision or babysit a separate resource for it ^{[S1] [S2]}.

For practitioners coming from a world of per-team or per-business-unit data lakes, this changes the starting point. Standing up an independent physical lake for every group used to be normal practice; doing that inside Fabric now just reintroduces the duplication and operational overhead OneLake exists to remove — it has shifted from default approach to antipattern ^[S1].

Any data item in Fabric — a lakehouse, a warehouse, and others — writes its tabular data into OneLake as open Delta-Parquet files automatically, no matter which engine produced it ^[S1]. There's no manual "push to storage" step. As soon as a lakehouse or warehouse exists, its data is already sitting in OneLake in an open, engine-neutral format, which is also why newer capabilities like Fabric IQ can sit on top of OneLake and treat shortcuts, mirrored data, and the OneLake catalog as one governed surface for semantic models and agents to reason over ^[S3].

That same open-format foundation is now extending outward. Microsoft has been shipping a run of capabilities — a ServiceNow zero-copy connector, delegated shortcuts, and an Iceberg-compatible table surface — that let data outside Fabric read OneLake in place, and let OneLake reach curated data across organizational boundaries, without copying bytes either direction. This article covers the core architecture and shortcut system first, then the zero-copy sharing story in depth, before going underneath the API to the Delta transaction log that makes all of it hold up under concurrent, multi-engine access.

Core Architecture: One Namespace, Every Engine#

A Single File-System Namespace#

A Fabric tenant's OneLake is one continuous file-system namespace stretching across every user, region, and cloud the tenant touches ^[S2]. The tenant itself sits at the root of that namespace, workspaces hang off it the way folders would, and lakehouses and other items live inside those workspace folders ^[S2]. Under the hood, OneLake's addressing model virtualizes the entire tenant as a single giant ADLS storage account: each workspace surfaces as a container in that account, and each data item surfaces as a folder inside its workspace container ^[S1]. Every path in OneLake is therefore unique across the whole tenant — a Spark job and a T-SQL query referencing the same item point at the exact same bytes, with nothing to reconcile.

Because OneLake speaks the standard ADLS Gen2 API and SDK surface, tools already built for ADLS — Azure Databricks, Azure Data Factory, custom code on the Azure Storage SDK — can read and write OneLake data with zero modification ^[S1]. This is not a thin compatibility shim in front of something else; it is the genuine wire protocol.

One Copy, Multiple Engines#

Several Fabric engines — T-SQL, Apache Spark, Analysis Services, and others — operate over the exact same physical Delta Parquet data, so teams never duplicate a dataset per engine or get locked into whichever engine first loaded it ^[S1].

That single architectural choice removes the recurring problem where a reporting copy drifts from the serving copy, which in turn drifts from the raw copy.

Governance and Resilience#

Governance in OneLake cascades down a hierarchy. Security, compliance, and data-management policy set at the tenant level apply automatically to anything landing in OneLake, and workspaces — each tied to a capacity in a given region and billed on its own — handle ownership and access decisions beneath that tenant-wide layer ^[S1]. On the resilience side, OneLake keeps data on zone-redundant storage wherever the region supports it, and capacities can additionally enable business continuity and disaster recovery (BCDR) to replicate data into a secondary geographic region ^[S1]. Deletion is not instant either: files removed from OneLake sit in a seven-day soft-delete window, recoverable with Azure Storage Explorer or PowerShell ^[S1].

Shortcuts: Connecting Data Without Moving It#

What a Shortcut Is#

A OneLake shortcut is an object that points from a shortcut path to a target path in another storage location ^[S4]. The symbolic-link analogy holds well: deleting the shortcut leaves the target untouched, but moving, renaming, or deleting the target can break the shortcut ^[S4]. The practical effect is that remote files and folders show up as though stored natively, which is what makes virtual data products possible without duplicating a byte ^[S1]. Any Fabric engine reading through the OneLake API treats a shortcut exactly like an ordinary folder — Spark, T-SQL, Real-Time Intelligence, Analysis Services, and even non-Fabric ADLS-compatible tools query through it with no shortcut-specific logic at all ^[S4].

Shortcuts can be created inside lakehouses and KQL databases, either interactively in the Fabric portal or programmatically through the OneLake shortcuts REST API ^[S4]. The target can be another spot inside OneLake itself — a different item, a different workspace, or even another tenant through cross-tenant data sharing — or it can reach outside Fabric altogether. Verified external targets include Azure Data Lake Storage Gen2, Amazon S3, Dataverse, Google Cloud Storage, and — per a Fabric Data Factory multi-cloud patterns post — Snowflake-hosted data addressed the same zero-copy way ^{[S1] [S2] [S4] [S5] [S6]}.

Internal vs. External Shortcuts#

An internal shortcut points at data in another Fabric item — a lakehouse, warehouse, KQL database, SQL database, mirrored database, or semantic model — anywhere across the tenant's items and workspaces, and access through it is authorized with the calling user's own identity, which must independently hold read permission at the target ^[S4]. External shortcuts (ADLS Gen2 or S3, for example) work differently: ADLS and S3 shortcuts delegate authorization to a cloud connection bound at shortcut creation, and binding that connection is a privileged operation — a user without permission on the connection cannot create a new shortcut that uses it ^[S4]. A dedicated post on securing zero-copy distribution patterns generalizes this: external shortcuts to non-Fabric stores like S3 and Google Cloud Storage are themselves a form of delegated shortcut, where the delegated identity is typically an account key or system identity native to the external system rather than a Fabric organizational account or service principal ^[S7].

Shortcut Caching, Deletes, and Limits#

Shortcut caching stores files read through external shortcuts in a per-workspace cache to cut cross-cloud egress. Retention is configurable from 1 to 28 days and resets on each access; a newer remote version triggers a refresh from the source; unused files are purged after the retention window; and individual files over 1 GB are never cached. Coverage today spans Google Cloud Storage, S3, S3-compatible, and on-premises data-gateway shortcuts ^[S4]. Writes bypass the cache entirely and go straight to the source — it is strictly a read-side optimization.

Shortcut deletion does not cascade, but deletes made through a shortcut do write through: removing the shortcut object leaves target data intact, whereas deleting a file or folder beneath the shortcut path removes it from the target store if the caller has write permission there ^[S4]. Fabric also publishes hard shortcut ceilings: an item can hold up to 100,000 shortcuts, a single OneLake path supports at most 10 shortcuts, chains of direct shortcut-to-shortcut links max out at 5, and shortcut names or paths cannot contain %, +, or non-Latin characters ^[S4].

Zero-Copy Enterprise Data Sharing#

OneLake's shortcut model was already zero-copy by design. What's new is a set of capabilities purpose-built for sharing outward — to other applications, other business units, and other tenants — without handing over a static export or building a bespoke integration.

Delegated Shortcuts: A Second Authentication Mode#

Every shortcut historically authorized access as whoever was signed in — pass-through identity, all the way to the target. Delegated OneLake Shortcuts, in preview at the time of the source post, add a second mode: access to the target flows through one configured connection identity instead of each individual signed-in user's identity ^[S9]. That connection identity can be an organizational account, a service principal, or a workspace identity, and delegation is optional per shortcut — shortcuts still default to pass-through authentication unless delegation is explicitly selected at creation time ^[S9].

Creating one in the Fabric portal follows a specific path: Get data > New table shortcut, choosing Microsoft OneLake as the source, then selecting Delegated identity as the connection method and either reusing or creating a connection with an organizational-account or service-principal authentication kind ^[S9].

Delegated shortcuts also work cross-tenant: a Fabric tenant can create a shortcut to OneLake data living in a different organization's Fabric tenant, authenticating with an identity from that other tenant, so downstream users reach it through the configured delegated identity rather than each needing their own cross-tenant permission ^[S9]. A named use case is sharing curated data between an organization's own test and production tenants, or between a parent company and a subsidiary, using the same zero-copy, intersection-based security model that applies within a single tenant ^[S9]. This is explicitly distinguished from Fabric's separate External Data Sharing feature: External Data Sharing targets consumers who hold no identity in the producer's tenant at all (an outside partner, an ISV's customers), while cross-tenant delegated shortcuts assume the consumer already holds an identity — an organizational account or service principal — in the producer's tenant ^[S9].

Securing the Zero-Copy Pattern#

Delegating identity raises an obvious question: what stops the delegated identity from becoming a backdoor around per-user security? The answer is an intersection model. For delegated OneLake shortcuts, access is computed as the intersection of what the delegated identity itself is permitted to see and what OneLake security additionally grants the specific end user — so an end user can never be given more access through the shortcut than the delegated identity was authorized for in the first place ^[S7]. A central data owner narrows what the delegated identity can see using table- and column-level OneLake security, and the consuming business unit then separately manages its own users' access on top of that already-constrained view ^[S9].

That two-layer model is what turns a shortcut into what one Fabric blog post frames as a governed publishing architecture: the central team keeps ownership of the source, consuming teams avoid copying the data at all, and downstream users get a managed access experience rather than raw-path access to the source item ^[S7].

For scaling this to many downstream consumers, the recommended design pattern is per-consumer service principals: represent each consuming team or region with its own service principal, scope that service principal to only its slice of the data, and let the team's own owners manage individual end-user permissions behind that service principal — rather than trying to centrally provision every end user against the source directly ^[S7].

# Zero-copy distribution: central data + delegated per-consumer identity

lakehouse-central-sales/                    <- source of truth, owned by central data team
  Tables/
    sales_curated/                          <- OneLake security: column-masked, region-scoped

lakehouse-region-emea/
  Tables/
    sales_shortcut/  <- delegated shortcut, connection identity = spn-emea-reader
                        (spn-emea-reader is scoped to EMEA rows only via OneLake security
                        on the source; EMEA team's own Viewers/Members see the
                        INTERSECTION of spn-emea-reader's access and their own OneLake role)

lakehouse-region-apac/
  Tables/
    sales_shortcut/  <- delegated shortcut, connection identity = spn-apac-reader
                        (same pattern, separate service principal, separate scope)

Design guidance from the same source draws a clear line between the two shortcut modes: choose pass-through shortcuts when the goal is letting multiple teams reuse a shared dataset while preserving strict source-managed, per-person authorization; choose delegated shortcuts when the goal is broader curated distribution and centralizing per-user provisioning on the source no longer scales ^[S7]. Pass-through has a hard constraint worth internalizing here: OneLake security cannot be additionally configured on the shortcut path itself for a pass-through shortcut — access is evaluated only at the target, so the data owner retains full, exclusive control regardless of which shortcut path is used to reach the data ^[S7].

ServiceNow: Zero-Copy from an Operational System#

ServiceNow Workflow Data Fabric ships a zero-copy connector that lets ServiceNow read data stored in OneLake in place, without moving or duplicating it into ServiceNow ^[S2]. Microsoft positions this as one example of a broader direction rather than a one-off integration: enabling any partner or application to build on OneLake through open standards — specifically the Iceberg REST Catalog — instead of requiring a proprietary, Fabric-specific integration layer ^[S10]. Mechanically, the connector is built on OneLake Table APIs that implement the Apache Iceberg REST Catalog standard, which is what lets an external engine discover and query OneLake tables through an open interface rather than custom code ^[S10]. Because that REST Catalog interface is open, the same door works for engines beyond ServiceNow's connector — Microsoft names Trino, DuckDB, and Apache Spark as examples that can query OneLake data through the same Table APIs with no engine-specific integration work ^[S10].

The intended shape of this pattern is a single governed dataset serving both operational and analytical consumers: the same underlying data can back incident enrichment, field-service context, supply-chain visibility, or AI workflow assistants in ServiceNow, without a separate copy per use case ^[S2].

Iceberg Interoperability#

Delta Lake has been OneLake's native table format from the start, but OneLake is extending toward genuine bidirectional interoperability with Apache Iceberg.

That source describes OneLake gaining native support for both the Delta Lake and Apache Iceberg table formats, enabling bidirectional access with Iceberg-managed tables originating on other platforms ^[S11]. Two distinct mechanisms are described for reaching Iceberg data from OneLake: OneLake Shortcuts can point a Fabric lakehouse at externally hosted Iceberg tables stored in S3 or ADLS Gen2 ^[S11], and separately, OneLake exposes an Iceberg REST Catalog API so external query engines can address Fabric-resident tables using standard Iceberg catalog connection strings ^[S11] — this is the same open-catalog surface the ServiceNow connector is built on ^[S10]. The source also describes table-format virtualization: OneLake presenting a Delta table's metadata as Iceberg-readable, and vice versa, without physically rewriting the underlying Parquet data ^[S11].

# Two paths into/out of Iceberg from OneLake (per a tier-6 comparison source — verify against
# Microsoft Learn before relying on this for a production design)

# 1. Pull an external Iceberg table INTO a Fabric lakehouse (no data movement)
lakehouse-analytics/
  Tables/
    external_iceberg_orders/   <- OneLake shortcut -> S3 or ADLS Gen2 Iceberg table

# 2. Let an external Iceberg-speaking engine read a NATIVE OneLake Delta table
#    via the Iceberg REST Catalog API — no shortcut needed, no format conversion of the
#    underlying Parquet, just metadata presented in Iceberg's catalog shape
trino> SELECT * FROM iceberg.onelake_catalog.sales_curated;

Security Model#

Two Planes: Control and Data#

OneLake security operates on two planes. Control-plane permissions decide what users can do in the environment — create, manage, share items — while data-plane permissions, the OneLake security model proper, decide what data users can actually read; control-plane grants often confer data access by default ^[S12]. Among the four workspace roles, Admin, Member, and Contributor can read and write OneLake data outright, while Viewer cannot read OneLake data unless explicitly granted access through a OneLake security role ^[S12].

OneLake authenticates exclusively with Microsoft Entra ID for both users and service principals, mapping the extracted identity onto Fabric-defined permissions; service-principal use requires a tenant administrator to enable SPNs tenant-wide or for specific security groups ^[S12].

Hierarchical Permissions and Sharing Scopes#

Permissions in OneLake follow the storage hierarchy of workspace, item, and folder (such as Tables/ and Files/), with lower levels inheriting from the parent by default ^[S12]. Item-level sharing has distinct data scopes: Write grants both SQL and OneLake data access, Read grants only item metadata, ReadData exposes data through SQL endpoints only (including delegated mode), and ReadAll exposes OneLake data only via the built-in DefaultReader security role ^[S12].

Every lakehouse ships with a DefaultReader OneLake security role that gives any user holding ReadAll access to all lakehouse data; tightening that requires editing or deleting the default role ^[S12]. A OneLake security role itself has four parts: the data (tables or folders) in scope, the permission granted, the member users, and optional constraints that carve out specific rows or columns — enabling table-, folder-, row-, and column-level security on lake data ^[S12].

Encryption, Transport, and Tenant Controls#

OneLake data is encrypted at rest by default with rotated Microsoft-managed keys in a FIPS 140-2 compliant manner, and workspaces can layer customer-managed keys on top ^[S12]. Inbound traffic enforces at least TLS 1.2 and negotiates TLS 1.3 when possible, but outbound Fabric connections to customer-owned infrastructure may fall back to older protocols — including TLS 1.0 — when the destination does not support newer versions ^[S12]. A tenant-admin setting also controls whether applications running outside Fabric (custom ADLS API clients, OneLake file explorer) can reach OneLake data at all; switching it off still leaves internal workloads like Spark and Data Warehouse able to access the data ^[S12].

A separate tier-6 source on Fabric access control reports that these mechanisms compose across four layers end to end: workspace-level RBAC via Entra ID groups, per-item permissions (ReadAll/ReadData/Build), OneLake security roles at folder/table/row/column granularity, and SQL-endpoint-level row-level security and column masking ^[S13]. A second tier-6 source separately claims that table, row, and column-level controls apply consistently regardless of whether the access path is the SQL analytics endpoint, Spark, or Power BI ^[S14]. Treat both as single-source characterizations rather than documented guarantees until corroborated by Microsoft Learn.

Migrating Existing Data In: Shortcuts as an Alternative to Copying#

Zero-copy access isn't only for steady-state sharing — it also shows up as a migration lever. As part of AI-assisted Synapse Spark and pipeline migration to Fabric (itself a preview capability), a dedicated migration phase scans notebook and job code for abfss:// storage paths pointing at external ADLS containers and creates OneLake shortcuts for them, as an alternative to physically copying that data into Fabric ^[S15].

# Before: Synapse notebook reads directly from an external ADLS path
df = spark.read.format("delta").load(
    "abfss://raw@synapsestorage.dfs.core.windows.net/sales/"
)

# After migration: the same external container is shortcut into the Fabric lakehouse,
# and the notebook reads through the OneLake path instead — no data was copied
df = spark.read.format("delta").load(
    "abfss://Lakehouse@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Tables/sales_shortcut/"
)

Inference: the exact abfss://onelake.dfs.fabric.microsoft.com path shape follows the general OneLake addressing model described earlier in this article; the migration source verifies that the migration tool creates shortcuts as its strategy, not the literal before/after code shown.

Best Practices#

Let the single-copy model work for you. The one-copy design is the core value proposition — the same physical Delta Parquet files can be loaded once with Spark, queried with T-SQL, and served in Power BI, because every engine reads the identical bytes on disk ^[S1]. Resist staging intermediate copies of the same data across different items; that recreates the silo problem OneLake exists to eliminate.

# Anti-pattern: per-engine copies
lakehouse/Tables/sales_raw/          <- Spark writes here
warehouse/Tables/sales_reporting/    <- ETL job copies + reshapes for T-SQL
semantic_model/import_cache/         <- Power BI Import mode copies again

# One-copy pattern: shortcut, don't copy
lakehouse/Tables/sales/              <- single Delta table, written once by Spark
# T-SQL analytics endpoint reads it directly; Power BI uses Direct Lake
# against the same Delta files — zero extra copies

Use shortcuts to unify data without migration. Rather than migrating everything into Fabric immediately, use shortcuts to bring existing ADLS Gen2, S3, Dataverse, GCS, or Snowflake storage into the namespace as live, read-only references, combinable with cross-tenant data sharing for governed external access ^{[S1] [S2] [S4] [S5] [S6]}. Pipelines and notebooks operate on the shortcut path as if the data were native, while the source data stays exactly where it is.

Choose pass-through vs. delegated shortcuts deliberately, not by default. Pass-through preserves strict source-managed, per-person authorization for teams reusing a shared dataset; delegated shortcuts trade that per-person granularity for distribution that scales past what centralized per-user provisioning on the source can handle ^[S7]. Because switching modes later means deleting and recreating the shortcut ^[S9], decide up front.

# Right: delegated shortcut chosen because the consumer count won't scale
# under per-user provisioning at the source
Shortcut: sales_shortcut (region: EMEA)
  Auth mode: Delegated
  Connection identity: spn-emea-reader (scoped via OneLake security at source)

# Wrong: defaulting to pass-through for a 40-team distribution fan-out, then
# discovering months later that per-user provisioning at the source doesn't scale,
# and having to delete + recreate every downstream shortcut to switch modes

Scope security roles precisely, and know who they can't restrict. The DefaultReader role is broad by design ^[S12]. Define custom OneLake security roles scoped to specific tables or folders wherever row- or column-level restriction matters, but relying on OneLake security roles to restrict workspace Admins, Members, or Contributors is an antipattern — those roles bypass OneLake security entirely and can read and write all item data, so granular roles only ever constrain Viewers and users holding item Read permission ^[S12].

# Wrong: trying to lock out a Contributor with a OneLake security role
Role: "finance-restricted"
  Members: [contributor-user@tenant.com]   <- has no effect; Contributor bypasses OneLake security
  Scope: Tables/gl_sensitive

# Right: put the sensitive table in a separate item with its own sharing
lakehouse-finance-restricted/
  Tables/gl_sensitive/    <- separate item; share only with Read/ReadData as needed

Never name a shortcut table with a space. The Delta format does not allow spaces in table names, so a shortcut created with one will not be recognized as a Delta table in the lakehouse at all ^[S4] — a common trap when shortcutting in tables from upstream systems with looser naming rules.

What Goes Wrong#

Siloed data lakes per business group. Standing up separate physical data lakes per business group, a common pre-OneLake practice, is an antipattern in Fabric because it recreates the resource-management overhead and data duplication that the single tenant-wide OneLake with workspaces and shortcuts is designed to eliminate ^[S1]. If the goal is organizational separation, cross-workspace shortcuts achieve it without standing up another lake.

Confusing shortcut deletion with delete-through. Removing the shortcut object itself only severs the link — target data is left alone. Deleting a file or folder through the shortcut path is a different operation: if the caller has write permission at the destination, that delete propagates all the way to the source ^[S4]. Treat these as two operations with opposite consequences, not interchangeable ways to clean up a shortcut.

Assuming DirectLake-over-SQL passes the end user's identity through a shortcut. Both DirectLake-over-SQL and T-SQL's delegated identity mode authorize shortcut reads as the item's owner, not as whoever is actually running the query ^[S4]. Any design that takes per-user passthrough for granted will quietly grant too much or too little access unless it explicitly uses DirectLake-over-OneLake or T-SQL user-identity mode instead.

Treating delegated-shortcut identity as a bypass for OneLake security. A delegated shortcut's connection identity is not a blank check to the end user behind it — access is the intersection of what the delegated identity can see and what OneLake security separately grants that user, so narrowing the delegated identity's own access is what actually caps downstream exposure ^[S7]. Designing a delegated shortcut and then relying only on the consuming workspace's own permissions, without also constraining the delegated identity at the source, defeats the model.

Internals#

Architecture & design#

Each Fabric tenant gets exactly one OneLake with a single file-system namespace spanning users, regions, and clouds; the tenant maps to the root of the lake, workspaces act like folders beneath it, and lakehouses and other items live inside workspaces ^[S2]. OneLake is layered on Azure Data Lake Storage Gen2 and exposes the same ADLS Gen2 APIs and SDKs, so existing ADLS-aware applications such as Azure Databricks can read and write OneLake data without code changes ^[S1]. Structurally, the addressing model virtualizes the whole tenant as one giant ADLS storage account: each workspace surfaces as a container, and each data item surfaces as a folder inside its workspace container ^[S1].

Security is architected as two cooperating planes rather than one flat permission set — a control plane governing what users can do, and a data plane (OneLake security) governing what data they can read — with third-party engines able to plug into the same policy layer through the authorized engine model: an external engine registers with OneLake, pulls security policy definitions and precomputed effective-access decisions from OneLake APIs, and enforces table, row-level, and column-level security locally at query time, keeping OneLake the single source of truth for policy across both Fabric-native and third-party engines ^[S12].

The zero-copy sharing capabilities layer onto this same architecture rather than replacing it. Delegated shortcuts add a second authorization mode alongside pass-through identity passthrough, resolved through an intersection of the delegated identity's own access and the end user's OneLake security grant ^{[S7] [S9]}. The Iceberg REST Catalog surface is likewise architected as an additional presentation of the same underlying storage — OneLake Table APIs implement the open Iceberg REST Catalog standard as another way to address data that is still, physically, Delta-Parquet-on-ADLS-Gen2 ^[S10]; table-format virtualization goes further still, presenting a Delta table's metadata as Iceberg-readable and vice versa without physically rewriting the underlying Parquet objects ^[S11].

How it works internally#

The mechanics that make OneLake's "one copy, every engine" promise actually safe under concurrent access come from the Delta Lake storage format underneath it, described in the peer-reviewed VLDB paper on Delta Lake's design ^[S16]. Every Delta table in OneLake stores its data as immutable Parquet objects plus a write-ahead transaction log in a _delta_log subfolder; the log — not a directory listing — is the single source of truth for which Parquet objects currently belong to the table, which is what lets ADLS Gen2-style object storage offer ACID table semantics on top of a storage layer that was never built with transactions in mind ^[S16].

Each commit is a zero-padded, sequentially numbered JSON file in _delta_log (for example 000003.json) holding an array of actions: add and remove record which Parquet objects entered or left the table, metaData and protocol record schema and format-version changes, commitInfo carries audit and provenance data, and an application-defined txn action lets writers such as streaming engines track exactly-once progress ^[S16].

Write serializability comes from optimistic concurrency rather than locking: a writer stages new Parquet objects independently, then atomically creates the single next-numbered log JSON file. Only one writer can win that specific commit number, so a losing writer detects the collision immediately, rereads the latest table state, and retries rather than blocking behind a lock held by another writer ^[S16]. The atomic "only one writer wins this commit number" guarantee is implemented differently per storage backend: object stores with atomic put-if-absent use that directly; filesystem-style stores — including ADLS Gen2, which OneLake is built on — use an atomic rename that fails if the target name already exists; stores lacking either primitive need an external coordination service to serialize commit-number assignment ^[S16].

Readers get snapshot isolation for free from this same log design: a Delta table reader pins its read to whatever the newest fully-committed log record is at the moment it starts, reconstructs table state as of that version, and never observes files added or removed by commits that land afterward — so concurrent writers never produce a dirty or partial read for a query already in flight ^[S16]. That is the property underneath OneLake's one-copy, multi-engine model: Spark, T-SQL, and Power BI can all read the same physical files concurrently with a writer, and every reader still sees a single consistent snapshot. It is the same guarantee that makes delegated-shortcut reads safe when the source table is mid-write — a delegated identity querying through a shortcut still pins to one consistent committed snapshot ^{[S7] [S16]}.

Because the log is append-only and every add/remove action stays immutable and versioned, OneLake's Delta tables also support time travel: a client can reconstruct any earlier table state by replaying the log up to an older commit ID, which is the same mechanism that lets an accidental overwrite be corrected by merging a table against one of its own prior versions instead of restoring from a separate backup ^[S16].

To keep metadata reads fast as a table accumulates history, the Delta engine periodically compacts the log into a Parquet checkpoint — every 10 commits by default — that folds in every still-relevant action and drops superseded ones, so readers reconstruct current state from the latest checkpoint plus a handful of trailing JSON commits instead of replaying the log from commit zero ^[S16]. Checkpoint compaction is deliberately safe to skip or fail: checkpoint writing and the pointer file recording the newest checkpoint ID are both best-effort and separate from the atomic commit itself, so a client that crashes mid-checkpoint leaves the table's actual data and log entirely uncorrupted — other readers simply fall back to an older checkpoint or the raw commit files ^[S16].

On the query-planning side, writers record per-column min/max value statistics and null counts for each Parquet object directly in that object's add action in the log or checkpoint, so query engines perform data skipping: they consult the log's stored statistics to exclude whole Parquet objects whose value range cannot satisfy a filter predicate, before issuing any I/O against those objects ^[S16]. Skipping effectiveness depends heavily on physical data layout, not just the presence of statistics — sorting or partitioning by a single column makes skipping strong for predicates on that column but ineffective on others, since every object's min/max range collapses toward the whole table's range for unsorted columns, which is the underlying motivation for layout-optimization techniques like Z-ordering across multiple columns ^[S16].

On top of this same log, OneLake's authorized engine model lets external engines enforce security without a remote call on every row: an engine pulls policy and precomputed access decisions from OneLake once, then applies them locally during its own execution plan ^[S12]. The Iceberg REST Catalog connector follows the identical shape one layer up the stack — an external engine discovers table metadata through the open catalog API rather than a proprietary integration, while the actual row data underneath is unchanged Delta-Parquet governed by the same transaction log ^[S10].

Performance characteristics#

Because Delta engine metadata scans go through the transaction log and checkpoint rather than listing and re-reading Parquet footers directly from the object store, table discovery stays roughly two orders of magnitude faster at large partition counts than a raw Parquet-on-object-store layout: in a benchmark table with 1,000,000 partitions, non-Delta engines took over an hour to enumerate the table while the Delta-backed engine did it in 108 seconds (17 seconds with the log cached on SSD) ^[S16].

OneLake's Delta storage layer also avoids the metadata bottleneck that plain object-store data lakes hit at scale — S3-style LIST calls cap out around 1,000 keys per request and take tens to hundreds of milliseconds each — by parallelizing both file-listing and column-statistics collection across a Spark cluster rather than serializing them on a single driver, which is what keeps "find the files relevant to my query" fast even as object counts grow into the hundreds of millions ^[S16].

Worked Example: Multi-Team Analytics Without Data Duplication#

Inference: the following scenario synthesizes multiple verified OneLake claims into one plausible end-to-end setup; the scenario framing is original, but every architectural step cites the claim it relies on.

Consider a retailer with three teams — Supply Chain, Finance, and Marketing — each owning data in its own pre-Fabric ADLS Gen2 storage account, and a fourth, central Analytics team that needs to join across all three without copying anyone's data. A fifth party, a corporate parent company, also needs a curated slice of the results.

Step 1 — Provision lakehouses. Create a Fabric workspace per team; each workspace gets a lakehouse backed by OneLake automatically, with its tabular data persisted as Delta-Parquet regardless of which engine writes it ^[S1].

Step 2 — Create external shortcuts to legacy storage. In each team's lakehouse, create an ADLS Gen2 shortcut pointing at that team's existing storage container. The shortcuts surface as folders such as Tables/supply_chain_inventory, Tables/finance_gl, and Tables/marketing_events inside their respective lakehouses, with no data copied ^[S4]. Each shortcut's authorization is delegated to a cloud connection bound at creation time, so only users with permission on that connection can create or modify it ^[S4].

lakehouse-supplychain/
  Tables/
    supply_chain_inventory/   <- ADLS Gen2 shortcut (external, cloud-connection auth)
lakehouse-finance/
  Tables/
    finance_gl/                <- ADLS Gen2 shortcut (external, cloud-connection auth)

Step 3 — Build a cross-team view with internal shortcuts. In a shared Analytics workspace, create a new lakehouse and add internal shortcuts pointing at the supply-chain and finance Tables/ folders ^[S4]. A data engineer can now write a single Spark notebook or T-SQL query joining inventory against the general ledger, with OneLake transparently authorizing each internal shortcut read against the calling user's own identity rather than a shared credential ^[S4]. Because every reader pins to the newest committed Delta log entry when the query starts, this join sees one consistent snapshot of both tables even if either source lakehouse is mid-write ^[S16].

lakehouse-analytics/
  Tables/
    inventory_shortcut/   <- internal shortcut -> lakehouse-supplychain (caller identity)
    gl_shortcut/           <- internal shortcut -> lakehouse-finance (caller identity)

Step 4 — Secure the shared lakehouse. Grant most analysts only Read access, which exposes item metadata alone and no actual data ^[S12]. For analysts who need to query underlying tables through the SQL analytics endpoint without full file-level access, grant ReadData instead ^[S12]. For the two senior analysts who need direct file access, grant ReadAll and then narrow the default DefaultReader role with a custom OneLake security role scoped to the specific tables they need, rather than leaving the broad default in place ^[S12]. Do not rely on OneLake security roles to keep workspace Contributors out of anything — Contributor bypasses OneLake security entirely, so data that must stay restricted belongs in a separate item with its own sharing ^[S12].

Step 5 — Share a curated slice cross-tenant, zero-copy. For the corporate parent company, create a delegated shortcut from the analytics lakehouse's finished result table into the parent's own tenant, authenticated with a service principal scoped to only the columns and rows the parent is entitled to see via OneLake security at the source ^{[S7] [S9]}. Because access is computed as the intersection of what that service principal can see and what OneLake security grants each parent-side user, no parent-side user can ever see more than the service principal itself was authorized for — even if the parent's own workspace administrators are generous with their local roles ^[S7].

lakehouse-analytics/
  Tables/
    quarterly_summary/   <- source of truth; column-masked via OneLake security

# In the parent company's own tenant:
lakehouse-parent-view/
  Tables/
    quarterly_summary_shortcut/   <- delegated shortcut, cross-tenant
                                      connection identity: spn-parent-readonly
                                      (scoped at source to non-sensitive columns only)

Step 6 — Confirm soft-delete coverage and time travel. Before going to production, verify that an accidental deletion of a shortcut path is recoverable inside the seven-day soft-delete window using Azure Storage Explorer ^[S1], and separately confirm that an accidental bad write to the analytics lakehouse's own Delta tables can be corrected by merging against a prior committed version reconstructed from the transaction log, rather than needing a full restore ^[S16].