What Data Factory does#
Data Factory is Fabric's data integration workload: it connects to data sources, moves and transforms data, and orchestrates that sequence from one place, with connectivity to more than 170 sources across cloud, multicloud, and on-premises systems reached through gateways [S1]. Microsoft positions it as the next generation of Azure Data Factory, with a migration guide for moving existing pipelines into Fabric [S1].
Pipelines: the orchestration layer#
A pipeline is a logical grouping of activities deployed and scheduled as one unit, not managed activity by activity [S2]. Activities fall into three categories: data movement, data transformation, and control flow (conditions, loops, error handling) [S2]. A pipeline can chain movement, transformation steps such as notebooks or dbt jobs, and control-flow constructs, all in a low-code designer [S1].
Choosing how data moves#
Data Factory offers three data-movement options [S1]: Copy job for simplified movement with built-in bulk, incremental, and change-data-capture delivery; Copy activity for fully customizable movement with manual control over parallelism; and Mirroring for near-real-time database replication into OneLake. Copy activity is Microsoft's recommended choice for large-scale migrations and high-volume recurring ingestion, since it's serverless, performant at scale, and cost-effective [S3].
Running and monitoring pipelines#
A pipeline run is one execution instance, started on demand, on a schedule, or from an event trigger such as a file landing in storage [S4]. Every run gets a unique run ID for tracking [S2]. Schedules have no open-ended option — every schedule needs a start and an end date, so a long-running pipeline stays alive by setting the end date far ahead [S4]. Failure emails fire only for scheduled runs, never on-demand ones [S4]. The Monitor hub shows per-run and per-activity detail, including row counts from Copy activity, and lets a run resume from just the failed activity instead of restarting whole [S5].
A worked example#
You land a partner's daily CSV export into a lakehouse: a pipeline with Copy activity pulling from an SFTP source into a lakehouse table [S1], on a daily schedule with a far-future end date and an email alert on failure [S4]. Months later the file arrives with a byte-order mark and the first column name is misread — the fix is enabling Skip BOM in the source dataset settings [S6]. In the Monitor hub you see exactly which activity failed, and resume from that point instead of restarting the whole pipeline [S5].
What goes wrong#
- Typo'd schedule parameters fail silently. A mismatched parameter name is ignored at run time rather than raising an error [S4].
- On-demand runs don't alert like scheduled ones. Failure emails only fire on scheduled runs; a failed manual test notifies no one [S4].
- Forgetting the mandatory end date. Omitting a far-future end date can quietly stop a pipeline you assumed was permanent [S4].