Fabric Data Factory: Pipelines, Copy, and Orchestration in Practice

What Is Fabric Data Factory?#

Fabric Data Factory is Fabric's data integration workload ^[S1]. It connects to data sources, moves data, transforms it, and orchestrates movement and transformation tasks from a single place, with connectivity to more than 170 data sources spanning cloud, multicloud, and on-premises systems reached through gateways ^[S1]. Microsoft positions it as the next generation of Azure Data Factory, publishing both a comparison guide and a migration path for teams moving existing Azure Data Factory or Azure Synapse pipelines into Fabric ^[S1].

The service is aimed at two different workload shapes at once: large-scale one-time migrations — moving a data lake or an enterprise data warehouse into OneLake — and high-volume recurring ingestion ^[S2]. Both are handled by the same underlying Copy engine, which Microsoft recommends specifically because it is serverless, performant at scale, and cost-effective ^[S2]. Serverless matters here in a concrete way: there is no compute to pre-provision or size before a migration starts.

For a practitioner deciding where Data Factory fits, the short version is: pipelines are the orchestration and large-scale movement tool, Dataflow Gen2 is the low-code shaping tool, and three distinct data-movement primitives — Copy job, Copy activity, and Mirroring — cover the actual transport of bytes. Increasingly, Data Factory is also the orchestration layer that reaches outside Fabric entirely, coordinating work across other clouds' data platforms from one pipeline.

Three Ways to Move Data#

Data Factory exposes three distinct data-movement options, each pitched at a different level of control and automation ^[S1]:

• Copy job — a simplified movement surface with built-in delivery styles: bulk load, incremental load, and change-data-capture replication. Microsoft describes it as Fabric's simplest and most scalable mechanism for moving data across multiple clouds and tenants ^[S3].
• Copy activity — fully customizable movement inside a pipeline, with manual control over parallelism and connector options. This is the right tool when the defaults are insufficient or when copy is one step in a larger orchestration.
• Mirroring — near-real-time database replication into OneLake, for analytical read access to operational data without building a separate ingestion pipeline ^[S1].

Microsoft publishes a data-movement decision guide specifically to help choose among the three ^[S1]. Declarative tools more broadly — Copy job among them, alongside third-party options like Fivetran and Airbyte — manage schema evolution and incremental-state tracking declaratively, which removes the need for hand-written change-detection logic in the pipeline itself ^[S4]. Inference: that declarative model is why Copy job trades some flexibility for a much smaller maintenance surface compared with hand-rolled Copy activity logic.

Run Your First Pipeline#

The fastest way to get a feel for Data Factory is to run one small copy end to end using Copy Job — the no-code Copy Job Activity that Fabric added with automatic schema detection and bulk, incremental, and change-data-capture load modes ^[S5].

1. 1
2. 2

   Create a Copy job From the Data Factory workload, create a new Copy job and pick a source connection. Copy job supports bulk load, incremental load, and change-data-capture replication as built-in delivery styles, so pick whichever matches how the source data actually changes ^[S3].

3. 3
4. 4

   Point it at a destination and store the connection Set a lakehouse table (or another supported destination) as the target. Store any source or destination credentials in a Data Factory connection rather than hardcoding them in the pipeline definition itself — the standard way to avoid embedding credentials directly ^[S5].

5. 5
6. 6

   Run it and check the Monitor hub Trigger the Copy job manually for this first run. Open the Monitor hub afterward and look at the per-activity detail — including row counts and skipped-row counts — which is the concrete place to confirm the copy actually moved the rows you expected ^[S5].

7. 7

A tier-6 practitioner tutorial describes Fabric pipelines adding a no-code Copy Job Activity — with automatic schema detection and bulk, incremental, and change-data-capture load modes — as GA per that post's own stated timeline ^[S5]. Hedge: that source is an independent tutorial series, not a Microsoft release note, so treat the GA date as the author's characterization rather than a confirmed Microsoft milestone until corroborated by a tier-1 source.

Event-Driven Data Movement#

Scheduled Copy jobs carry three recurring costs that a fixed cadence cannot avoid: wasted compute from runs that fire when nothing changed, a latency-versus-cost tradeoff baked into whatever frequency you pick, and operational overhead from managing many schedules once table counts grow ^[S3]. Fabric's answer, now generally available, is wiring Copy job execution to Fabric Activator instead of a clock.

The documented pattern is a two-step build ^[S3]:

1. Define the Copy job first — source, destination, full-or-incremental mode, and column mappings — exactly as you would for a scheduled job.
2. Create an Activator rule whose condition triggers that Copy job as its action, replacing the scheduled trigger with an event trigger.

This is a different mechanism from the storage event triggers pipelines have long supported (covered below) — it's Activator rules driving Copy jobs specifically, and it is GA, not preview ^[S3].

Copy job is Fabric's simplest and most scalable mechanism for moving data across multiple clouds and tenants.

Multi-Cloud Orchestration#

Data Factory ships built-in connectors and activities for Snowflake, Databricks, and Google BigQuery, plus Salesforce reached through SFTP-paired movement — Databricks specifically gets notebook and JAR trigger activities, not just a Copy connector ^[S6]. That connector breadth turns a single Fabric pipeline into a cross-cloud orchestrator rather than just an ingestion tool.

The documented shape of this is a multi-cloud orchestration pattern: one pipeline extracts from Google BigQuery, hands the data to a Databricks notebook for processing, and merges the result into a Snowflake table — all under one pipeline and one scheduler ^[S6]. None of the three systems needs its own orchestration tooling; Fabric's pipeline is the single control plane spanning all three clouds.

What makes that practical for anything beyond a linear chain is Data Factory's logic-heavy control-flow support: If-Condition, Until, and ForEach iterators are positioned as the differentiator for complex cross-system workflows, compared with simpler linear-chain schedulers ^[S6]. And the trigger for a cross-cloud pipeline doesn't have to be a clock either — Fabric pipelines can start automatically from event-based triggers, such as a new file landing in a Lakehouse or Blob Storage container ^[S6].

Pipelines as the Orchestration Layer#

Pipelines sit above individual data-movement activities and act as the orchestration layer ^[S1]. A single pipeline can chain Copy activities, Fabric notebook runs, Spark job definitions, HDInsight tasks, stored procedures, SQL scripts, and dbt jobs, wired together with loops and conditionals, all authored in a low-code visual designer ^[S1].

dbt jobs specifically run SQL-based model transformations directly against a Fabric data warehouse, with a no-code setup for building, testing, and deploying models ^[S1]. Teams that prefer code-first orchestration are not locked into the visual designer — Data Factory's Apache Airflow integration lets workflows be expressed as Python DAGs instead ^[S1].

Copilot for Data Factory can summarize what existing dataflow queries and pipelines do, and explain run errors with suggested fixes.

Copilot for Data Factory lets users author and edit pipelines and dataflows from natural-language prompts, and layers that same assistance onto explaining existing artifacts and diagnosing failed runs ^[S1].

ETL, ELT, and Choosing a Pattern#

Fabric Data Factory deliberately supports both ETL and ELT in the same solution, rather than forcing one pattern ^[S1]:

A practical division of labor follows from this: use pipelines when the problem is orchestration and large-scale movement, use Dataflow Gen2 when the problem is low-code data shaping, and consult Microsoft's data-movement decision guide for the Copy job / Copy activity / Mirroring choice underneath ^[S1].

The diagram below distills that decision into a scoring matrix — Pipeline Copy Activity, Dataflow Gen2, and Notebook Activity compared across strength, parallelism control, authoring surface, orchestration role, and watch-outs — alongside the separate Copy job / Copy activity / Mirroring data-movement decision.

Scheduling and Triggers#

A pipeline run is one execution instance of a pipeline, started one of three ways: on demand from the editor, on a time-based schedule, or from an event trigger firing on occurrences like files arriving in storage ^[S1]. Fabric pipeline schedules have no open-ended option — every schedule requires both a start and an end date, so a long-running pipeline is kept alive by setting an end date far in the future and adjusting it later ^[S1].

A single pipeline can carry up to 20 schedules, each with its own frequency, start/end dates, and time zone, letting one pipeline serve several delivery cadences without duplication ^[S1]. Schedules can pass parameter values into the pipeline at each run, sourced either as a direct static value or as a reference into a variable library — the latter centralizes configuration for promotion across environments ^[S1].

Migrating From Azure Data Factory and Synapse#

Moving existing orchestration into Fabric has moved from a manual rebuild exercise toward assisted tooling, though most of it is still preview.

A Pipeline Migration Assistant, described as in public preview, performs an assessment-first migration of Synapse pipelines into Fabric Data Factory: it checks activity compatibility, converts linked services into Fabric connections, and disables migrated triggers by default so they can be validated before activation rather than firing immediately ^[S7].

A separate, newer command-line skill goes further on automation. The Pipeline Migration skill — a companion to a Synapse Spark migration skill — moves Synapse Data Factory pipeline orchestration into Fabric Data Factory pipelines, rewiring notebook activities to point at the Fabric notebooks the Spark migration skill already created. Like its companion, it runs from the command line over REST APIs and starts with a read-only assessment of scope and blockers before changing anything ^[S8]. This tooling is explicitly described as preview ^[S8].

The remapping it performs is specific enough to plan around ^[S8]:

Inference: the read-only assessment-first design in both the CLI skill and the Migration Assistant point at the same underlying caution: Microsoft is treating pipeline migration as something to preview and validate before cutover, not a one-shot lift-and-shift, which tracks with triggers being deliberately excluded from the automated path.

Best Practices#

Scale out at the orchestration layer#

Rule: when one Copy activity hits its ceiling, fan out with a ForEach loop rather than tuning that single activity further.

Why: the documented scale-out path for ingestion beyond what one Copy activity delivers is to drive multiple Copy activities in parallel from a ForEach loop, partitioning the workload — by table or file range — across iterations ^[S2].

{
  "name": "ForEachTablePartition",
  "type": "ForEach",
  "typeProperties": {
    "items": "@pipeline().parameters.TableList",
    "isSequential": false,
    "activities": [
      {
        "name": "CopyTablePartition",
        "type": "Copy",
        "typeProperties": {
          "source": { "tableName": "@item().tableName" },
          "parallelCopies": 8
        }
      }
    ]
  }
}

Protect fragile systems with explicit parallelism caps#

Rule: cap degree of copy parallelism explicitly when the source or destination system cannot tolerate high concurrent load.

Why: explicitly lowering the degree of copy parallelism is the documented way to protect a fragile source or destination — capping thread count limits the concurrent load the copy places on the machines hosting those data stores ^[S2].

// Before: default parallelism can overwhelm a small on-prem SQL source
{ "source": { "type": "SqlSource" } }

// After: explicit cap protects the source during business hours
{
  "source": { "type": "SqlSource" },
  "parallelCopies": 2
}

Measure before tuning throughput#

Rule: identify the bottleneck before touching Intelligent Throughput Optimization (ITO) or parallelism settings.

Why: achievable end-to-end throughput is bounded by the minimum of three numbers — what the source store can serve, what the destination can absorb, and the network bandwidth between them ^[S2]. Raising ITO or parallelism cannot push throughput past whichever of those three is the actual constraint ^[S2].

// Wrong: raising ITO to Maximum when the source is a throttled REST API
ITO: Maximum, parallelCopies: 32   // no effect -- source IOPS is the ceiling

// Right: measure via activity monitoring first, then size to the real bottleneck
ITO: Balanced, parallelCopies: 4   // matched to source's actual throughput limit

Filter event triggers through the Subject field#

Rule: write folder/file event filters against the CloudEvent Subject field, not a separate filename property.

Why: storage event triggers can listen to OneLake or Azure Blob Storage events, and file name and folder path are encoded inside Subject rather than exposed as separate properties, so filtering has to target Subject directly ^[S9].

Track incremental state with a watermark, not a schedule guess#

Rule: persist the last-processed value in a control table and parameterize the next run's source query against it, rather than assuming a fixed lookback window will always be correct.

Why: a tier-6 practitioner tutorial documents a three-activity pattern for this: a Lookup activity reads the last watermark, a parameterized Copy activity pulls only newer rows, and a Stored Procedure activity writes the new watermark back only after the copy succeeds ^[S5]. Hedge: this is a widely-used pattern description from an independent tutorial rather than Microsoft documentation, but the sequencing — write the watermark back only on success — is the detail that actually prevents data loss on a failed run, so it is worth keeping even while treating the source as informal.

1. Lookup: SELECT last_watermark FROM control.LoadState WHERE table_name = 'Orders'
2. Copy:   SELECT * FROM Orders WHERE ModifiedDate > @{activity('Lookup').output.firstRow.last_watermark}
3. Stored Procedure (runs only if Copy succeeded):
   UPDATE control.LoadState SET last_watermark = @{utcNow()} WHERE table_name = 'Orders'

Gate promotions with an approval step, not a side channel#

Rule: when a pipeline promotes data from silver to gold, insert an Approval Activity ahead of final data-quality assertions instead of routing sign-off through email or a separate ticketing tool.

Why: a tier-6 tutorial describes this governance-oriented pattern as pausing pipeline execution until a designated approver responds, with a timeout branch on non-response, which keeps the approval step auditable inside the pipeline's own run history rather than external to it ^[S5]. Hedge: sourced from an independent tutorial, not Microsoft documentation — verify the Approval Activity's exact configuration surface against current product docs before relying on it for a compliance-sensitive workflow.

Authenticate Web/Copy calls with a token variable, not embedded credentials#

Rule: call an auth endpoint first to retrieve a bearer token into a pipeline variable, then pass that token as an Authorization header on subsequent Copy or Web activity calls.

Why: a tier-6 tutorial documents this as the standard way to avoid hardcoded credentials sitting inside the pipeline definition itself ^[S5]. Hedge: independent tutorial source — treat as a sound general pattern, not a Microsoft-endorsed reference implementation, and confirm secret handling against your tenant's credential-storage policy (e.g. a workspace-scoped connection or key vault reference) rather than a literal pipeline variable in production.

1. Web Activity "GetToken": POST https://auth.example.com/token  -> response.access_token
2. Set Variable "BearerToken": @activity('GetToken').output.access_token
3. Copy Activity source headers:
   Authorization: Bearer @variables('BearerToken')

Diagnose before you optimize a Lakehouse table#

Rule: check table health before running a Spark OPTIMIZE, instead of scheduling OPTIMIZE unconditionally.

Why: a documented check-then-act pattern runs a sp_get_table_health_metrics stored procedure first inside an ETL/ELT pipeline, then conditionally triggers a Spark notebook to run OPTIMIZE only if an anomaly was actually detected — implementable as a two-activity Fabric pipeline with an if-condition ^[S10].

1. Stored Procedure Activity: EXEC sp_get_table_health_metrics @table = 'sales.orders'
2. If Condition: @greater(activity('CheckHealth').output.fragmentation_pct, 20)
     True  -> Notebook Activity: run OPTIMIZE sales.orders
     False -> No-op (skip the compaction run)

What Goes Wrong#

Tuning without measuring#

Raising ITO presets or parallelism settings when the real bottleneck is the source store's IOPS or the network path cannot push throughput past that external limit — it just adds cost for no gain ^[S2].

Interval-schedule immutability in production#

Interval-based schedules (public preview) currently cannot be edited, enabled, or disabled after creation — they must be deleted and recreated — and time-slice monitoring and backfill are not yet available for them ^[S9]. That limits their suitability for production windowed loads that need replay.

Silent parameter mismatches#

Parameter names configured on a schedule must match the pipeline's parameter names exactly. A mismatch is silently ignored at run time rather than raising an error, so a typo produces a run on default values instead of a visible failure ^[S9].

Email notifications do not cover on-demand runs#

Email failure notifications can be configured per schedule, but they fire only for scheduled runs, never for on-demand runs — a manual test execution that fails will not alert anyone ^[S9].

Private-link constraints on Copy destinations#

Pipeline Copy activity to and from Fabric Data Warehouse is unsupported when private links are enabled, and Eventhouse is unsupported as a direct Copy pipeline destination under both tenant-level and workspace-level private link configurations ^[S11]. This needs to be verified early in any deployment that mandates private networking.

A stray byte-order mark can silently corrupt the first column#

CSV source files carrying a byte-order mark can cause Copy Activity to misread the first column's name entirely, per a tier-6 practitioner tutorial ^[S5]. Hedge: independent tutorial source, but the fix it describes is cheap insurance regardless: enable the source dataset's Skip BOM option rather than relying on positional column mapping that will silently drift if the header ever changes.

Fixed schedules carry a cost you don't see until it's large#

Before reaching for event-driven Copy jobs, know what a fixed schedule is actually costing: wasted compute on runs where nothing changed, a latency-versus-cost tradeoff tied directly to how tight the schedule is, and rising operational overhead as the number of scheduled tables grows ^[S3]. None of these show up as a single alarming number — they accumulate.

Internals#

Architecture & design#

The Copy activity's serverless architecture exposes parallelism at three distinct, independently tunable levels ^[S2]:

1. Control-flow concurrency — many Copy activities can run at once, orchestrated by constructs like a ForEach loop.
2. Intelligent Throughput Optimization (ITO) — a per-activity compute-sizing lever.
3. Degree of copy parallelism — a thread cap within a single activity.

Event-based triggers are not a bespoke subsystem bolted onto pipelines — they are built on existing Fabric platform plumbing. Defining a storage event trigger creates an eventstream object plus a Data Activator alert stored as a Reflex item in the workspace, and triggers can respond to file events, job events, and workspace events ^[S9]. This means a trigger is really a thin pipeline-facing wrapper over Fabric's general eventing and alerting primitives, not a parallel implementation. The GA event-driven Copy job pattern follows the same shape one layer up: an Activator rule's condition is the trigger, and the Copy job it fires is the action — the same Activator/Reflex machinery, aimed at a Copy job instead of a pipeline ^[S3].

The multi-cloud connector surface — Snowflake, Databricks, BigQuery, Salesforce — sits inside this same architecture as additional connector implementations behind the Copy activity and pipeline activity model, rather than a separate execution path; the If-Condition/Until/ForEach control-flow primitives that make multi-cloud orchestration practical are the same general-purpose constructs used for any pipeline, not multi-cloud-specific logic ^{[S6] [S6]}.

At the data-plane level, bulk-load connectors converge on shared plumbing too: SSIS, Informatica, and Azure Data Factory's SQL bulk-load connectors all rely on BCP API semantics under the hood, so they share the same client-side bulk-ingestion model transparently across tools ^[S12]. Inference: this suggests Fabric's own SQL-destination Copy paths sit on comparable bulk-load semantics, though the claim as sourced covers the broader tooling ecosystem rather than confirming Fabric Data Factory's Copy activity specifically uses BCP API internally.

How it works internally#

ITO is the compute-sizing decision inside the Copy activity: the service weighs CPU, memory, network allocation, and the expected cost of the run to choose a throughput level, accepting the presets Auto, Standard, Balanced, and Maximum, or an explicit numeric value between 4 and 256 ^[S2]. The presets trade cost against speed — Auto lets the service pick freely for the source-destination pair and observed data pattern, Standard stays within standard compute resources, Balanced weighs throughput against available compute, and Maximum applies all available compute to the run ^[S2].

Degree of copy parallelism is a separate, orthogonal lever: it bounds the maximum number of concurrent reader/writer threads inside one Copy activity, must be an integer of at least 1, and at run time the service may actually use fewer threads than configured if that yields better performance ^[S2]. By default the service picks an optimal parallelism dynamically from the source-destination pair and the observed data pattern, rather than requiring a fixed value up front ^[S2].

On the trigger side, the events delivered follow the CloudEvents schema (specversion 1.0), with top-level properties including source, subject, type, time, id, and a data payload object supplied by Event Grid ^[S9]. Trigger metadata such as the triggering file name reaches pipeline expressions via parameters parsed out of the event's Subject and Topic fields, referenced with null-safe syntax like @pipeline()?.TriggerEvent?.FileName — necessary because during a manual test run these values are NULL, and an unguarded reference would fail the run ^[S9].

The Synapse-to-Fabric migration skill's remapping logic gives a second, complementary window into pipeline internals: because Fabric has no separate Dataset item, a Synapse dataset's properties fold directly into the consuming activity, and a Synapse global parameter becomes a variable-library entry referenced through the expression pipeline().libraryVariables rather than a pipeline-scoped global ^[S8]. Notebook activities are repointed by GUID rather than by name when rewritten from SynapseNotebook to TridentNotebook, which is why the migration skill runs a read-only assessment first — it has to resolve those GUID mappings before it can safely rewrite anything ^{[S8] [S8]}.

Performance characteristics#

Coming soon — this depth isn't in the knowledge base yet. It needs an L4/L5 source such as a Microsoft engineering blog benchmarking Copy activity throughput at scale, or a case study quantifying ITO preset impact on real workloads with numbers attached. Tracked in content/queue.md.

Worked Example: Event-Driven Incremental Load#

Consider a team that receives sales files in an Azure Blob Storage container and wants to load each file into a Fabric lakehouse as it arrives.

Step 1 — Event trigger. Create a storage event trigger on the pipeline, subscribed to blob-created events on the target container. The trigger automatically provisions an eventstream and a Reflex alert in the workspace ^[S9]. Configure the filter to match against the Subject field, since folder path is encoded there rather than exposed as a separate property ^[S9].

Step 2 — File name parameter. Add a pipeline parameter FileName. Bind it to the trigger event using null-safe syntax so a manual test run does not fail on a NULL value:

@pipeline()?.TriggerEvent?.FileName

Step 3 — Copy activity. Add a Copy activity sourced from the blob container, scoped to the arriving file via the FileName parameter, writing to the target lakehouse.

Step 4 — Throughput sizing. Leave ITO on Auto for initial runs, then use activity monitoring to see whether the real constraint is source IOPS, destination write throughput, or network bandwidth ^{[S2] [S9]}. Only adjust ITO or parallelism after that measurement — raising either past the actual bottleneck has no effect ^[S2]. The Monitor hub's per-activity detail — including Copy Activity row counts and skipped-row counts — is the concrete place to look for that evidence ^[S5].

Step 5 — Failure notifications. Configure email notifications on the pipeline's schedule for retries. Remember these alerts do not fire for manual test runs of the same pipeline ^[S9].

Inference: if file volume grows past what one Copy activity handles comfortably, wrap the Copy activity in a ForEach loop partitioned by folder prefix — this is the same scale-out pattern documented for high-volume ingestion generally, just applied to an event-driven trigger instead of a schedule ^[S2].

Inference: the same shape extends naturally to Copy job instead of Copy activity — swap Step 3 for a Copy job and Step 1's storage event trigger for an Activator rule, and the pipeline becomes the GA event-driven Copy job pattern rather than a custom event-triggered pipeline ^[S3]. Whether to use a pipeline with Copy activity or a bare Activator-triggered Copy job comes down to whether the surrounding orchestration needs pipeline-level control flow (parameters, conditionals, retries across multiple activities) or is genuinely a single copy step, in which case Copy job's smaller surface is the simpler build.