Event-driven pipeline orchestration with Activator

Recommended architecture#

The pattern replaces polling loops and fixed-schedule triggers with a continuous event-and-condition loop. Activator is a real-time engine that monitors data and reacts to events, in contrast to Data Factory's traditional fixed-schedule execution model ^[S1]. Real-Time Intelligence in Fabric is a complete end-to-end solution covering ingestion, transformation, storage, modelling, analytics, visualization, and automated actions, all purpose-built for event-driven and streaming scenarios ^[S2].

The reference architecture has five stages: event sources -> Eventstream -> Activator rules -> Copy Job / pipeline / Spark actions -> monitoring.

1. Event sources and the Real-Time hub#

The Real-Time hub functions as an organization-wide catalog of data in motion, surfacing active data streams, Microsoft-managed sources (including CDC feeds from Azure SQL DB, Cosmos DB, and PostgreSQL), and Fabric workspace item events, all accessible to every Fabric workload ^[S2]. Fabric events and Azure Blob Storage events exposed in the Real-Time hub can trigger Activator rules to start Spark jobs or Dataflows, enabling fully event-driven orchestration that requires no polling schedule ^[S2].

Inference: The hub is the natural entry point for this pattern because both the trigger side (Fabric/Azure item events) and the ingestion side (streams and CDC feeds) are cataloged in one place, so a team registers a signal once and reuses it across rules rather than re-plumbing each trigger per pipeline.

2. Ingestion and shaping: Eventstream#

Eventstreams provide a no-code pipeline for collecting, transforming, and routing high-volume real-time data to multiple destinations simultaneously; supported sources include Apache Kafka, AWS Kinesis, Google Cloud Pub/Sub, MQTT v3.1/v3.1.1, and an MQTT-based Real-Time Weather connector ^[S2]. They expose built-in stream-processing operators -- filtering, cleansing, windowed aggregations, deduplication, and content-based routing -- so data can be shaped before it lands at a destination, and derived eventstreams can be produced as reusable, shareable outputs ^[S2].

The connector lineup has matured. The Apache Kafka and Azure Service Bus source connectors for Eventstream have both reached general availability ^[S3]. The GA Apache Kafka connector adds support for the SASL_SSL security protocol alongside its existing SSL support, combining SASL-based authentication (e.g. PLAIN or SCRAM) with TLS encryption for connecting to production Kafka clusters ^[S3]. Custom CA and mutual TLS (mTLS) support, previously preview, is now generally available for the MQTT, Apache Kafka, AWS MSK, and Confluent Cloud source connectors, letting customers supply their own CA and client certificates from Azure Key Vault for mutually authenticated encrypted connections ^[S3].

Several connectors remain in preview and should be treated as such: an enhanced Azure IoT Hub source connector built on the Kafka Connect framework preserves full device-to-cloud message metadata ^[S3]; a new Oracle Database CDC source connector streams changes from on-premises or cloud-hosted Oracle databases and can route processed change events to Eventhouse, Lakehouse, Activator, or custom endpoints ^[S3]; and the Eventstream HTTP source connector now supports pagination in preview, natively walking multi-page REST API responses using page-based or cursor-based configurations ^[S3].

Inference: Because Eventstream can route to Activator directly ^[S3], the Eventstream stage does double duty here -- it both feeds a data store and hands a shaped signal to the rule engine, so the same feed that lands rows can also arm the trigger that moves them onward.

3. Rules and conditions: Activator#

Activator monitors data flowing through the Real-Time hub, KQL queries, or Power BI reports and fires automated actions -- such as sending notifications, launching Fabric pipeline or Spark job or Dataflow executions, calling User Data Functions, or triggering Power Automate workflows -- when threshold, pattern, or KQL-defined conditions are met ^[S2]. An Activator rule that invokes a Copy Job can be triggered from conditions such as a file being created in a monitored location, a table being updated (e.g. in OneLake), or a metric crossing a defined threshold ^[S1].

4. Actions: Copy Job, pipelines, and Spark#

Fabric Activator can now directly invoke a Copy Job as a rule action, without requiring an intermediate Data Factory pipeline, and this capability is Generally Available ^[S1]. Copy Job is described as Fabric's simplest and most scalable mechanism for moving data across multiple clouds and tenants ^[S1]. The recommended pattern for event-driven data movement is: define a Copy Job (source, destination, full-or-incremental mode, mappings) first, then create an Activator rule whose condition triggers that Copy Job as its action, replacing a scheduled trigger with an event trigger ^[S1]. Once the rule is enabled, it continuously monitors its defined condition and runs the Copy Job automatically only when that condition is met, rather than on every scheduled interval ^[S1].

Beyond Copy Job, Activator actions can launch Fabric pipelines, Spark jobs, Dataflows, or User Data Functions ^[S2], which is what lets the same trigger fabric drive both data movement and downstream transformation.

5. Monitoring#

Inference (hedged, tier 6): Recommended Eventstream health signals to monitor include end-to-end latency, event throughput drop versus the source rate, error rate from schema mismatches, and pending-event queue depth; a steadily growing queue indicates the destination is falling behind and the stream needs scaling or simplified transforms ^[S7]. Because ^[S7] is a tier-6 community tutorial, treat these signals as reasonable operational guidance to confirm against a tier-1 source, not as a product guarantee.

Data flow#

1. An event source (Kafka, Service Bus, IoT Hub, Oracle/CDC, Blob storage event, or a OneLake table update) raises a signal cataloged in the Real-Time hub ^{[S2] [S3]}.
2. Optionally, an Eventstream collects and shapes the feed with filtering, deduplication, windowing, or content-based routing, and can route directly to Activator ^{[S2] [S3]}.
3. An Activator rule evaluates its condition -- file created, table updated, or metric threshold crossed ^[S1].
4. When the condition is met, the rule fires an action: a Copy Job (GA, no intermediate pipeline) ^[S1], or a Fabric pipeline / Spark job / Dataflow / User Data Function ^[S2].
5. The action runs only on the event, not on a fixed interval ^[S1].
6. Operational health signals are watched to confirm the loop is keeping up (hedged ^[S7]).

Component responsibilities#

The fixed-schedule vs event-driven tradeoff#

This is the core decision the pattern exists to resolve.

Fixed-schedule data movement is characterized as having three recurring costs: compute waste from empty runs when no data changed, a latency-versus-cost tradeoff tied to schedule frequency, and operational overhead from managing many schedules across large table counts ^[S1]. An event-driven rule removes all three: it runs the Copy Job automatically only when its condition is met, rather than on every scheduled interval ^[S1].

Inference: The freshness/cost frontier inverts between the two models. Under a schedule, higher freshness means more frequent runs and therefore more empty runs and more cost; under event triggering, freshness tracks the event itself while cost tracks real work, so you stop paying for polling. The break-even favours events when data arrives irregularly or sparsely relative to the schedule; a genuinely continuous, high-rate feed that would fire the rule on essentially every interval anyway gains less from switching and is better served by a streaming ingestion path.

When to keep a schedule (inference): predictable, dense arrival where nearly every interval has work; contractual batch windows; or downstream systems that expect a fixed cadence. When to go event-driven ^[S1]: sparse or irregular arrival, large table counts whose schedules are an operational burden, or a latency requirement tighter than the cheapest tolerable schedule frequency.

Check-then-act maintenance automation#

Event-driven orchestration also applies to lakehouse maintenance, not just data movement. Running OPTIMIZE (Spark table compaction) on a fixed schedule regardless of actual table condition is called out as an antipattern: it wastes compute/capacity units on already-healthy tables while tables that truly need attention can still be missed ^[S4]. Delta table files can drift into a suboptimal physical layout over time even when the logical data hasn't changed, which is the underlying cause of gradually slowing queries ^[S4].

The recommended check-then-act pipeline pattern runs sp_get_table_health_metrics first inside an ETL/ELT pipeline, then conditionally triggers a Spark notebook to run OPTIMIZE only if an anomaly was detected -- implementable as a two-activity Fabric pipeline with an if-condition ^[S4]. That stored procedure is a built-in procedure exposed through the Fabric SQL analytics endpoint, giving a T-SQL-native way to check the structural/physical health of Lakehouse Delta tables ^[S4]; it reports file-size distribution across bins and row-count distribution so small-file problems and fragmentation are visible at a glance, and includes snapshot/checkpoint version information to judge whether a new Delta checkpoint is warranted ^[S4]. It is callable as standard T-SQL from Fabric pipelines, Azure Data Factory, dbt, or any SQL-based orchestration tool, since it does not require a Spark session to invoke ^[S4]. Because the SQL analytics endpoint is read-only, it cannot perform maintenance itself; the procedure only exposes the diagnostic signal, while the actual OPTIMIZE/compaction fix must still run through Spark or the Lakehouse engine ^[S4].

Inference: Check-then-act is the same event-driven principle applied to a synthetic event -- the anomaly detected by the health check is the trigger, so maintenance compute is spent only when a table is genuinely unhealthy, exactly mirroring the empty-run waste that fixed schedules incur on the data-movement side ^{[S1] [S4]}.

Governance & security: OAP for cross-workspace flows#

When the trigger and the action live in different workspaces, Outbound Access Protection (OAP) governs whether the flow is permitted. OAP for events is a preview capability ^{[S5] [S6]}; mark it as such in any plan.

Fabric Events and Azure Events in Real-Time hub now support Workspace OAP, bringing the same default-deny outbound network control already used by other Fabric workloads to cross-workspace event consumption ^[S5]. When OAP is enabled on a workspace, all outbound connections from items in that workspace are blocked by default unless explicitly permitted through the workspace's data connection rules ^[S5]. With OAP enabled, an Activator alert or Eventstream in that workspace can no longer consume Job or OneLake events emitted from a different workspace unless a data connection rule for the Real-Time Events connector is added to explicitly allow it ^[S5]. Event consumption between items within the same workspace is always allowed regardless of the OAP setting; OAP only affects consumption that crosses a workspace boundary ^[S5].

Enforcement is one-sided: OAP for events is enforced only on the consumer's workspace; the OAP configuration of the workspace producing the events has no effect on whether the cross-workspace event flow is allowed ^[S5]. OAP for events is positioned as one of three layered network controls -- OAP governs whether a consumer workspace can reach event sources elsewhere, workspace-level private links govern whether other workspaces can reach a source workspace that blocks public access, and tenant-level private links govern whether external Azure events (e.g. Azure Blob Storage events) can reach a tenant that blocks public internet access ^[S5].

On the action side, Activator now supports Workspace OAP: actions targeting Fabric items (notebooks, Spark jobs, pipelines, User Data Functions, Dataflows) in the same workspace are always allowed, while actions targeting items in other workspaces are blocked unless the workspace admin explicitly permits them via data connection rules ^[S6]. Notification actions are constrained under OAP: Microsoft Teams notifications are restricted to the same tenant and configurable per workspace via a Teams connection kind; email notifications are blocked by default and limited to same-tenant recipients; and Power Automate flows triggered from Activator are blocked outright and cannot currently be re-enabled through data connection rules ^[S6].

The seven-day pause window is the critical operational hazard. If OAP is turned on for a workspace that already has cross-workspace event consumers configured, Real-Time hub does not delete those consumers; it detects the policy change and moves the affected configurations into a Paused state, visible on the Fabric events or Azure events page with a reason shown via View details ^[S5]. Paused event configurations retain their events for up to seven days after the pause begins; if the access rule is not corrected within that window, the retained events are dropped rather than delivered ^[S5]. Recovery is not automatic: to resume a paused cross-workspace event consumer after adding the required data connection rule, the consumer configuration must be deleted and recreated rather than simply resuming automatically ^[S5].

OAP for Azure and Fabric events is described as the first stage of OAP integration for the Fabric eventing ecosystem, with plans to extend coverage to Power BI reports and maps next; that forward-looking scope is not yet shipped ^[S5].

Inference: For any cross-workspace event-driven design, add the Real-Time Events data connection rule on the consumer workspace before enabling OAP, or accept the delete-and-recreate recovery plus a bounded seven-day retention buffer ^[S5]. Keeping trigger and action in the same workspace sidesteps OAP entirely, since same-workspace consumption and same-workspace actions are always allowed ^{[S5] [S6]}.

Cost & capacity#

• Eliminating empty runs is the primary saving. Fixed schedules pay compute for empty runs when no data changed; event triggers run only on real events ^[S1].
• Operational overhead drops because you stop managing many schedules across large table counts ^[S1].
• Maintenance compute is spent only when needed under check-then-act, versus scheduled OPTIMIZE that burns capacity units on already-healthy tables ^[S4].
• Inference (hedged, tier 6): Direct Ingestion sends raw events straight to a destination and carries lower latency and lower capacity-unit cost than routing through transformation first; a reasonable default is to start with Direct Ingestion and push filtering/aggregation to query time, adding Event Processing only when volume reduction, enrichment, or content-based routing is specifically required ^[S7]. Confirm against a tier-1 source before committing cost claims.

Failure, retry, and delivery considerations#

The verified claims support only a limited, honest set of statements here:

• Hedged (tier 6): Eventstream provides at-least-once delivery, so duplicate events are possible (for example during connector restarts); the recommended mitigation is to deduplicate at the destination -- KQL arg_max() by event ID for Eventhouse, or Delta MERGE logic keyed on event ID for Lakehouse ^[S7]. Because this is a tier-6 source, treat the delivery guarantee as a claim to verify against Microsoft documentation before designing idempotency around it.
• The seven-day pause is itself a failure mode: a mis-timed OAP change silently pauses cross-workspace consumers, and events are dropped after seven days if the rule is not corrected -- with delete-and-recreate as the only recovery ^[S5].
• Retry/backoff semantics for Activator-fired Copy Jobs and the guaranteed-delivery model for the GA connectors are not covered by verified claims. Do not invent retry counts, dead-letter behaviour, or throughput numbers; see Open questions.

Risks & anti-patterns#

• Keeping fixed schedules for sparse data pays for empty runs, forces a latency-vs-cost compromise, and multiplies operational overhead across large table counts ^[S1].
• Scheduled OPTIMIZE regardless of table condition wastes capacity on healthy tables while still missing tables that need attention; use check-then-act instead ^[S4].
• Enabling OAP on a workspace with existing cross-workspace consumers without pre-staging data connection rules pauses those consumers and drops events after seven days, recoverable only by delete-and-recreate ^[S5].
• Relying on Power Automate actions from Activator in an OAP-protected workspace -- those flows are blocked outright and cannot currently be re-enabled via data connection rules ^[S6].
• Treating preview connectors (IoT Hub enhanced, Oracle CDC, HTTP pagination) as GA -- they are in preview ^[S3].
• Inference: Forcing an always-firing high-rate feed through event triggering buys little over a streaming ingestion path, since the rule would fire on essentially every interval anyway; reserve event triggers for sparse/irregular arrival where empty-run avoidance is the win ^[S1].

Internals#

Architecture & design#

The design pivots on a single inversion: replace a fixed-schedule trigger with an event/condition trigger, keeping the same downstream action. Activator is the real-time monitoring engine that watches the Real-Time hub, KQL queries, or Power BI reports and fires actions when threshold, pattern, or KQL conditions are met ^{[S1] [S2]}; Copy Job is the movement engine it invokes directly, with no intermediate Data Factory pipeline required (GA) ^[S1]. The check-then-act maintenance variant reuses the same shape -- a diagnostic read (sp_get_table_health_metrics over the read-only SQL analytics endpoint) produces the condition, and a Spark action performs the fix, since the endpoint cannot mutate tables itself ^[S4]. Cross-workspace flows are gated by OAP data connection rules on the consumer/action workspace, with same-workspace flows always allowed ^{[S5] [S6]}.

How it works internally#

On the eventing side, the connector transport internals matter operationally: the GA Kafka connector uses SASL_SSL (SASL auth plus TLS) ^[S3], and mTLS across MQTT/Kafka/MSK/Confluent draws customer-supplied CA and client certificates from Azure Key Vault for mutually authenticated encrypted connections ^[S3]. The preview IoT Hub connector is built on the Kafka Connect framework and preserves device metadata by copying it into user metadata with an __src__ key prefix, extractable in an Eventstream SQL operator via a GETMETADATAPROPERTYVALUE function ^[S3]. OAP enforcement is implemented as default-deny outbound at the workspace boundary, evaluated only on the consumer's workspace, with a policy-change detector that transitions existing cross-workspace consumers into a Paused state (seven-day event retention, then drop) rather than deleting them ^[S5]. The private-network path for connectors runs on a Streaming virtual network data gateway that provisions a bridge Azure VNet/subnet so connectors execute inside the customer's network, with connectivity to the private source established via VPN, ExpressRoute, private endpoints, or peering ^[S3]. The Eventstream operator execution internals -- how filtering, windowing, and content-based routing actually execute, and the ordering/latency semantics between the ingest path and the Activator hand-off -- are a known depth gap in the knowledge base and are Coming soon rather than described from grounded L4/L5 detail.

Performance characteristics#

Coming soon. The knowledge base carries no verified L4/L5 throughput, latency, or scaling benchmarks for Activator rule evaluation, Copy Job execution, or the Eventstream ingest-to-action path. The only performance-adjacent signals present are hedged tier-6 operational guidance -- monitor end-to-end latency, throughput drop versus source rate, schema-mismatch error rate, and pending-event queue depth as a growing-queue back-pressure indicator ^[S7] -- which should be confirmed against a tier-1 source before being treated as a performance contract. No throughput numbers are invented here.

Alternatives#

• Same-workspace, OAP-free (recommended default when possible). Keep the event trigger and the action in one workspace so same-workspace consumption and same-workspace actions are always allowed, sidestepping OAP data connection rules and the seven-day pause hazard entirely ^{[S5] [S6]}. Simplest to operate; loses domain/workspace isolation.
• Cross-workspace with OAP governance. When domains must be isolated, place trigger and action in separate workspaces and pre-stage the Real-Time Events data connection rule on the consumer workspace before enabling OAP ^[S5]. Stronger governance and blast-radius control; adds the delete-and-recreate recovery path and the seven-day retention window as operational obligations.

Assumptions#

• Data arrival is sparse or irregular enough that event triggering beats a schedule on cost; a continuously dense feed is better served by a streaming ingestion path (inference from ^[S1]).
• Sources are among those Eventstream/Real-Time hub support (Kafka, Service Bus, IoT Hub, Oracle CDC, Kinesis, Pub/Sub, MQTT, Blob events, OneLake table updates) ^{[S2] [S3]}.
• Preview connectors and OAP-for-events are acceptable at their current maturity, or the design is confined to GA surfaces (Kafka/Service Bus connectors, mTLS, Activator-to-Copy-Job) ^{[S1] [S3]}.
• One Fabric capacity backs the workloads; capacity sizing is the operator's responsibility.

Open questions#

• Activator/Copy Job retry, backoff, and dead-letter semantics. No verified claims describe what happens when an Activator-fired Copy Job or pipeline action fails. Recommend the knowledge-curator ingest a tier-1 source on Activator action reliability before designing around retries.
• Delivery guarantees of the GA connectors. The at-least-once/duplicate guidance here is tier-6 only ^[S7]; recommend curator coverage from Microsoft documentation before building idempotency assumptions.
• Eventstream operator execution internals and end-to-end latency (L4/L5) are a known gap; recommend the coverage-auditor track and the knowledge-curator source it.
• Throughput and scaling limits for Activator rule evaluation and Copy Job under event load are absent from the KB; do not assume numbers -- recommend curator coverage.

Source legend#