FABRIC SPARK TOOLKIT
ENTERPRISE STORAGE & QUERY ENGINE INTERNALS

OneLake Storage, Polaris Distributed SQL & Direct Lake

In-depth architectural guide to Microsoft Fabric's unified storage substrate and distributed relational query tier. Covers OneLake shortcut identity resolution, Iceberg REST Catalog zero-copy sharing, Polaris Cascades compilation, and Direct Lake lazy paging internals.

1. OneLake Storage, Shortcuts & Delegated Identity

OneLake provides a single, tenant-wide hierarchical namespace over Azure Data Lake Storage Gen2. Instead of copying files between services, Fabric uses lightweight metadata references called Shortcuts.

Architecture: OneLake Shortcut Identity & Security Resolution
Consumer Request Power BI / Spark / T-SQL Target: /Tables/dim_customer (Shortcut Pointer) OneLake Security Broker 1. Internal Shortcut: User Entra token pass-through 2. External Shortcut: Workspace stored SAS / Key 3. Delegated Shortcut: Consumer-specific App Role ⚠ DirectLake-over-SQL Trap: Forces delegation Physical Storage Target Target Lakehouse Parquet Data Zero data movement External ADLS / AWS S3 Bucket Zero-Copy REST Translation
CRITICAL DELETE SEMANTICS: Deleting a Shortcut object in the Fabric UI removes the metadata pointer only. However, running DELETE FROM my_shortcut_table or issuing an SDK file delete against the shortcut path permanently deletes the underlying physical data in the source container. Always educate engineering teams on path boundary safety.

2. Polaris: Inside Microsoft's Distributed SQL Query Engine

Polaris is the distributed, stateless cloud query engine powering the Fabric Data Warehouse and the automatic SQL Analytics Endpoint over Lakehouse Delta tables.

Architecture: Polaris Two-Phase Cascades Compilation & Distributed Scheduling
1. SQL Submission T-SQL Query Parser Cell Metadata Binding Delta Transaction State 2. Cascades Compiler Two-Phase CBO Optimization Task Fusion (Zero Shuffle) or Data-Move Enforcer (Hash Partition Exchange) 3. Workload Scheduler Cache-Affinitized Placement NVMe SSD Cell Cache Hit Stateless Compute Worker Hierarchical State Machine 4. Result Return Direct Stream Sub-second client TDS protocol

3. Direct Lake Residency & Lazy Column Transcoding

Direct Lake bridges the gap between massive data lakes and sub-second BI dashboards by allowing Power BI Analysis Services to load Parquet columns directly into VertiPaq memory pages without executing DAX-to-SQL query translation.

T-SQL FABRIC SQL ANALYTICS ENDPOINT / WAREHOUSE 
-- Check table file fragmentation and row counts before publishing Direct Lake semantic models
SELECT
t.name AS TableName,
COUNT(f.file_id) AS TotalParquetFiles,
SUM(f.size_in_bytes) / (1024.0 * 1024.0) AS SizeMB,
AVG(f.size_in_bytes) / (1024.0 * 1024.0) AS AvgFileMB
FROM sys.tables t
CROSS APPLY sys.dm_delta_files(t.object_id) f
GROUP BY t.name
ORDER BY TotalParquetFiles DESC;

4. Fabric SQL Database Translytical Path

Fabric SQL Database represents a cloud-native Hyperscale OLTP engine with autonomous, continuous physical Delta replication into OneLake, enabling true HTAP (Hybrid Transactional/Analytical Processing).

PYSPARK RUNNABLE IN FABRIC NOTEBOOK 
# Query the automatic OneLake shadow replica of an operational SQL Database without hitting the OLTP engine
df_replicated = spark.read.table("fabric_sqldb_name.dbo.customer_orders")
# Join with historical Lakehouse cold data seamlessly
df_joined = df_replicated.join(
spark.read.table("lakehouse_gold.dim_customer"),
"customer_id"
)
display(df_joined.groupBy("region").count())