Real-time / streaming analytics pattern on Microsoft Fabric

Goal#

Stand up an end-to-end streaming analytics solution on Microsoft Fabric that serves a seconds-fresh hot path for operational monitoring and alerting, and a cold path over OneLake for long-history batch BI and cross-engine analysis — from a single physical copy of the events, and with automated actions fired directly off the stream. Real-Time Intelligence in Fabric is a complete end-to-end solution covering ingestion, transformation, storage, modelling, analytics, visualization, and automated actions, all purpose-built for event-driven and streaming scenarios ^[S1].

Recommended architecture#

The pattern has five stages: sources cataloged in the Real-Time hub, an Eventstream that collects and shapes the feed, an Eventhouse/KQL hot store, a OneLake Delta cold path, and a serving/action layer (Real-Time Dashboards, Power BI, and Activator).

1. Sources and the Real-Time hub#

The Real-Time hub is the tenant-wide counterpart of OneLake for data in motion: it catalogs eventstreams, KQL database tables, Microsoft-managed source streams, and Fabric/Azure events so users can discover, ingest, and react to streaming data in one place ^[S2]. It surfaces active data streams, Microsoft-managed sources — including CDC feeds from Azure SQL DB, Cosmos DB, and PostgreSQL — and Fabric workspace item events, all accessible to every Fabric workload ^[S1].

Inference: Because the hub is tenant-wide and reachable from every workload, it is the natural discovery and governance entry point for this pattern — teams register a stream once and reuse it across dashboards, alerts, and downstream jobs rather than re-plumbing each source per consumer.

2. Ingestion and routing: Eventstream#

Eventstreams provide a no-code pipeline for collecting, transforming, and routing high-volume real-time data to multiple destinations simultaneously; supported sources include Apache Kafka, AWS Kinesis, Google Cloud Pub/Sub, and MQTT v3.1/v3.1.1 ^[S1]. They expose built-in stream-processing operators — filtering, cleansing, windowed aggregations, deduplication, and content-based routing — so data can be shaped before it lands at a destination, and derived eventstreams can be produced as reusable, shareable outputs ^[S1].

Inference: Content-based routing plus multiple simultaneous destinations is what lets one Eventstream fan a single feed out to both the hot store and the cold path — you route by latency need at ingestion time instead of copying data later.

3. Hot store: Eventhouse / KQL database#

An Eventhouse is a workspace-level container that can host multiple KQL databases, letting them share underlying capacity and resources so cost and performance are jointly optimized across all contained databases ^[S3]. Real-Time Intelligence storage is hierarchical: a workspace can contain multiple eventhouses, an eventhouse multiple KQL databases, and each KQL database multiple tables ^[S4]. Eventhouses automatically organize ingested records by arrival time, enabling fast, indexed queries over structured, semi-structured, and unstructured data at any scale ^[S3].

KQL database storage is split into two independently billed tiers: a premium cache tier (comparable to ADLS premium) that holds the hot dataset governed by the caching policy, and a standard tier (comparable to ADLS hot) that holds the full retention window governed by the retention policy; configuring a shorter cache window reduces premium-tier spend without affecting total queryable history ^[S3]. Eventhouse data is queryable through native KQL and through a managed, governed T-SQL analytics endpoint, so BI tools and analysts can connect without learning KQL ^[S1].

Three hot-path features round out the store: Eventhouse includes native anomaly detection that evaluates live time-series and event datasets in place, without copies or exports, reducing detection latency and supporting direct investigation from dashboards ^[S1]; and monitoring is surfaced at the container level, where a single system-overview page tracks storage, compute, ingestion rates, and top-queried databases across every KQL database in the eventhouse ^[S3].

Inference: Update policies and materialized views are the standard KQL mechanism for pre-aggregating hot rows into continuously-updated snapshots; the KB verifies materialized views exist as eventhouse child items ^[S3] but does not carry claims on their execution semantics, so treat the pre-aggregation design as an inference to be confirmed against a source (see Open questions).

4. Cold path: OneLake availability and offload#

Data stored in an Eventhouse is also mirrored to OneLake as a single logical copy for use by other Fabric experiences ^[S3]. Enabling OneLake availability exposes KQL table data in Delta Lake format, making it directly queryable by other Fabric engines — including Power BI Direct Lake mode, Warehouse, Lakehouse, and Spark notebooks — without any data movement, and with no additional storage charge beyond the standard OneLake tier already billed ^[S3]. OneLake availability can be enabled at the database level (all current and future tables, with optional backfill) or per table, and the KQL retention policy governs the OneLake copy too: data aged out of the KQL database is concurrently soft-deleted from OneLake ^[S3].

A Fabric lakehouse merges data-lake-scale storage with warehouse-style querying, with structured and unstructured data in one place analysable by both Spark and T-SQL and no data movement between systems ^[S5]. This makes the lakehouse the natural home for the cold path when the streaming data must be joined against curated batch datasets. A shortcut created in a KQL database lands in its Shortcuts folder and is treated as an external table, queried with the Kusto external_table() function rather than as a native table ^[S6] — the route for referencing lakehouse data from KQL in place.

Anti-pattern (cited): While OneLake availability is enabled on a table, certain DDL operations are blocked — renaming the table, changing a column's data type, applying row-level security, and deleting or purging rows; these require temporarily disabling availability, and re-enabling it triggers a full historic backfill ^[S3]. Plan schema changes before turning availability on.

5. Serving and action: Real-Time Dashboards, Power BI, Activator#

Real-Time Intelligence supports geospatial analytics through Map, a built-in visualization that layers bubbles, heatmaps, polygons, and 3D extrusions over Eventhouse or Lakehouse data with configurable KQL refresh intervals for live map updates ^[S1]. For business reporting, Power BI operates as a core Fabric workload supplying the platform's analytics and visualization surface ^[S8]; a Power BI report draws from exactly one semantic model ^[S7], and a Power BI dashboard is a single screen of tiles used to monitor key metrics at a glance ^[S7].

Activator monitors data flowing through the Real-Time hub, KQL queries, or Power BI reports and fires automated actions — sending notifications, launching Fabric pipeline, Spark job, or Dataflow executions, calling User Data Functions, or triggering Power Automate workflows ^[S1]. Fabric events and Azure Blob Storage events exposed in the Real-Time hub can trigger Activator rules to start Spark jobs or Dataflows, enabling fully event-driven orchestration that requires no polling schedule ^[S1]. Fabric notebooks can operate directly over Eventhouse data using KQL, T-SQL, Python, or Spark without data movement, and RTI integrates with Fabric data agents for conversational analytics over live and historical data, complementing Activator's threshold-based automation ^[S1].

Data flow#

1. Events arrive from Kafka, Kinesis, Pub/Sub, MQTT, or CDC feeds and are cataloged in the Real-Time hub ^{[S1] [S2]}.
2. An Eventstream collects the feed and applies filtering, cleansing, windowed aggregation, deduplication, and content-based routing, then routes to multiple destinations simultaneously ^[S1].
3. Hot path: records land in a KQL database, indexed by arrival time, resident in the premium cache tier for sub-second KQL queries and materialized-view reads ^[S3].
4. Cold path: with OneLake availability on, the same records are exposed as Delta in OneLake for cross-engine batch analytics, governed by the retention policy ^[S3].
5. Serve: Real-Time Dashboards and Map render live KQL results ^[S1]; Power BI reports read the T-SQL endpoint or Direct Lake over the OneLake copy ^{[S3] [S7]}.
6. Act: Activator watches the hub, KQL queries, or reports and fires notifications, jobs, functions, or flows ^[S1].

Component responsibilities#

Performance#

• Hot reads come from the premium cache tier; a shorter caching window lowers premium spend without shrinking total queryable history ^[S3].
• Cold-start latency: an Eventhouse compute node becomes available within 5 to 10 seconds of a cold start because the engine is fully managed and autoscales; design latency-sensitive pipelines with idle periods around this warm-up ^[S3].
• Cold-path freshness is tunable: when OneLake availability is active, an adaptive batching mechanism delays writes until enough data accumulates for optimally-sized Parquet files (typically 200–256 MB); the default maximum delay is up to 3 hours and can be lowered to as little as 5 minutes, though smaller delays risk many small files that degrade downstream query performance ^[S3].

Inference: The 5-minute floor on OneLake batching means the cold path is minutes-fresh at best — sub-minute freshness is a hot-path (KQL cache) responsibility, not something to force onto the Delta copy.

Governance & security#

• Sharing an Eventhouse link propagates the sender's permission level across all child items — KQL databases, tables, dashboards, functions, materialized views, and embedded querysets — whereas sharing an individual KQL database grants access only to that database and excludes Eventhouse-level navigation such as the system overview ^[S3].
• Workspace-level surge protection lets capacity administrators cap each workspace's share of total CU consumption as a percentage over a rolling 24-hour window; a workspace that breaches its threshold is automatically blocked from new operations, while mission-critical workspaces can be exempted ^[S3].
• Row-level security applied to a KQL table is one of the DDL operations blocked while OneLake availability is active, so plan the security model before enabling the cold copy ^[S3].
• The Power BI service implements security and oversight through row-level security (RLS), sensitivity labels, usage metrics, and audit logs ^[S8].

Cost & capacity#

• Eventhouse capacity consumption is metered through an 'UpTime' metric: CU seconds equal active seconds multiplied by the virtual-core count allocated by the autoscaler; a KQL database that is a child of an eventhouse rolls its UpTime up to the eventhouse item, with no separate CU line for the database subitem ^[S3].
• Storage cost is split across the two tiers, so the caching-policy window is the primary lever for premium-tier spend ^[S3].
• The OneLake Delta copy carries no additional storage charge beyond the standard OneLake tier already billed ^[S3].
• Under query pressure, an Eventhouse applies throttling in three escalating stages: proactive (queries slowed, ingestion unaffected), reactive (both paused, no data lost), and extreme-reactive (both paused with risk of data loss after a holding period) ^[S3].

Risks & anti-patterns#

• Do not stream batch workloads. Inference: If the freshness requirement is daily or slower, the always-on premium cache and autoscaled compute add cost with no real-time payoff — a batch lakehouse/warehouse pipeline is the right tool. Streaming is justified by an event- or seconds-driven freshness need, which is the whole design premise of RTI ^[S1].
• Enabling OneLake availability freezes schema/DDL (rename, type change, RLS, delete/purge) and re-enabling forces a full backfill — decide schema and security first ^[S3].
• Over-aggressive OneLake batching (near the 5-minute floor) produces many small files that degrade downstream query performance ^[S3].
• Ignoring cold-start warm-up (5–10 s) in latency-sensitive pipelines with idle gaps leads to missed SLAs ^[S3].
• Uncapped capacity risks the extreme-reactive throttling stage, where data loss becomes possible; use surge protection to bound each workspace's CU share ^[S3].

Internals#

Architecture & design#

The two-tier storage split is the core internal design: a KQL database physically separates a premium cache tier (hot dataset, governed by the caching policy) from a standard tier (full retention window, governed by the retention policy), and the two are billed independently ^[S3]. Records are organized by arrival time on ingestion, which is what enables the fast indexed queries over structured, semi-structured, and unstructured data ^[S3]. The OneLake copy is a single logical mirror of the same records rather than a second physical dataset, which is why it carries no extra storage charge and stays lifecycle-locked to the KQL retention policy ^[S3].

How it works internally#

On the cold path, OneLake availability uses an adaptive batching mechanism: writes to OneLake are delayed until enough data accumulates to produce optimally-sized Parquet files (typically 200–256 MB), with a default maximum delay of up to 3 hours that can be lowered to as little as 5 minutes via the mirroring policy — smaller delays trade file quality for freshness ^[S3]. Compute is fully managed and autoscaled, so a cold node reaches readiness within 5 to 10 seconds ^[S3]. The Eventstream internals — how its stream-processing operators execute filtering, windowed aggregation, and content-based routing — are a known depth gap in the KB (rti internals carry only L1–L3 conceptual claims), so that layer is Coming soon rather than described from grounded L4/L5 detail.

Performance characteristics#

Under load, the Eventhouse enforces a three-stage throttling ladder — proactive (queries slowed, ingestion intact), reactive (queries and ingestion paused, no data lost), and extreme-reactive (both paused, data-loss risk after a holding period) — which is the governing back-pressure behaviour to design capacity around ^[S3]. Capacity is metered by the UpTime CU formula (active seconds × autoscaler-allocated vCores) ^[S3], and read latency is bounded by which tier a query hits: cache-resident rows are sub-second while standard-tier and OneLake reads are higher-latency ^[S3]. Benchmarked internal query-engine numbers for KQL over the cache are not present in the KB and are marked Coming soon rather than invented.

Alternatives#

• Hot-only (Eventhouse-first), no cold offload. When history and cross-engine joins are not needed, skip OneLake availability and serve everything from KQL and Real-Time Dashboards ^{[S1] [S3]}. Simplest and cheapest for pure operational monitoring; loses cross-engine batch reuse.
• Hot + cold coexistence (recommended default). Route the same Eventstream feed to the KQL hot store and enable OneLake availability so Power BI Direct Lake, Warehouse, Lakehouse, and Spark can all read the Delta copy ^{[S1] [S3]}. Best when both seconds-fresh alerting and batch BI over long history are required.

Assumptions#

• The workload has a genuine event- or seconds-driven freshness requirement (otherwise see the batch anti-pattern above).
• One Fabric capacity backs the eventhouse; capacity sizing and surge protection are the operator's responsibility.
• Sources are among those Eventstream and the Real-Time hub support (Kafka, Kinesis, Pub/Sub, MQTT, CDC feeds) ^{[S1] [S2]}.

Open questions#

• Materialized view / update-policy execution semantics. The KB verifies materialized views exist as eventhouse child items ^[S3] but carries no claims on how update policies trigger or how materialized views refresh. Recommend the knowledge-curator ingest a source on KQL update policies and materialized views before relying on pre-aggregation performance claims.
• Eventstream processing internals (windowing, routing execution, throughput limits) are an L4/L5 gap; recommend curator coverage.
• Direct Lake behaviour over the KQL OneLake copy specifically — whether small-file batching interacts with Direct Lake resident-column limits — is not covered by verified claims here.

Source legend#