Metadata-Driven Architecture in Microsoft Fabric

Most Fabric estates start the same way: a handful of source systems, a handful of pipelines, each one built by hand. That works until source fifteen arrives, then source fifty. At that point the team is no longer building pipelines, it's maintaining a fleet of near-identical copies that all need the same bug fix applied fifty times. Metadata-driven architecture is the answer: describe what needs to be loaded as data — rows in a control table, entries in a variable library, a JSON or YAML contract — and let one reusable pipeline and one reusable notebook execute that description for every source, rather than writing a new pipeline per source.

This matters because Fabric Data Factory already gives you the raw material for it. A pipeline is a logical grouping of activities deployed and scheduled as one unit ^[S1]. Parameters exist specifically so the same pipeline can process different datasets or configurations across runs, while variables hold values that change within a single run ^[S2]. Recommended pipeline design practice is explicit about the payoff: start with simple data movement, then parameterize connections and file paths so pipelines stay reusable instead of hard-coded ^[S1]. That single sentence from Microsoft's own pipeline documentation is effectively the mission statement for this whole pattern.

Parameterizing connections and file paths keeps pipelines reusable rather than hard-coded.

The payoff compounds as source count grows. One pipeline plus one row of metadata per source scales linearly in configuration, not in pipeline count. A bug fix, a new column mapping, a new retry policy — all of it lands once, in the shared pipeline or notebook, instead of being copy-pasted across dozens of near-duplicate artifacts.

The three moving parts: contract, engine, ledger#

Every metadata-driven implementation in Fabric has the same three pieces, whether it's five lines of variable-library JSON or a fully externalized control-table schema.

The metadata contract is the data that describes the work: which source, which target, what load mode, what quality rules. Data Factory gives you two places to hold it. Pipeline parameters are assigned a value when a run starts and stay constant for the rest of that run, distinguishing them from variables that can be reassigned mid-run ^[S3]. Schedules can pass parameter values in either as a direct static value or as a reference into a variable library, and the latter centralizes configuration so it can be promoted across environments — dev, test, prod — without editing the pipeline itself ^[S4]. A connection used inside a pipeline can itself be parameterized: capture its GUID connection ID from Manage connections and gateways, then substitute that GUID dynamically via a string parameter so one pipeline can target a different connection per run ^[S3].

The engine is the reusable orchestration and transformation logic that reads the contract and acts on it. On the orchestration side, Data Factory's control-flow activities — If-Condition, Until, and ForEach — are the differentiator that makes complex, logic-heavy cross-system workflows possible from a single pipeline ^[S5]. To scale ingestion beyond one Copy activity, the documented pattern is to fan out at the orchestration layer: drive multiple Copy activities in parallel from a ForEach loop, partitioning the workload (by table, by file range) across iterations ^[S6]. On the transformation side, incremental ingestion in Fabric notebooks is commonly parameterized using mssparkutils.runtime.getParameterValue, so a pipeline can pass in a run date and the notebook reads only that day's partition ^[S7] — the same contract-then-engine split, just on the Spark side of the house.

The run ledger is what makes the whole thing restartable and auditable rather than a black box. Every pipeline run gets its own unique run ID for tracking ^[S8], and Fabric's monitoring surfaces already give real-time run progress, run history, execution-time and resource-usage metrics, and an audit trail of who ran what and when ^[S8]. Critically, a failed pipeline run can be rerun either in its entirety or resumed starting only from the failed activity, rather than always requiring a full restart ^[S9] — which is exactly the restartability a metadata-driven framework needs when source forty-two fails but sources one through forty-one succeeded.

A worked pattern: watermark-driven incremental load#

The canonical metadata-driven building block in Fabric is the watermark pattern: a Lookup activity reads the last processed watermark value, a Copy activity's source query is parameterized with that watermark to pull only newer rows, and a Stored Procedure activity writes the new watermark back to a control table only after the copy succeeds ^[S10]. Applied across many sources, the control table becomes the metadata contract and the three-activity sequence becomes the reusable engine:

-- control table: one row per source, doubling as both
-- the metadata contract and the run ledger
CREATE TABLE dbo.LoadControl (
    SourceName      NVARCHAR(100) PRIMARY KEY,
    SourceQuery     NVARCHAR(MAX),   -- parameterized with @watermark
    TargetTable     NVARCHAR(200),
    LastWatermark   DATETIME2,
    LastRunStatus   NVARCHAR(20),
    LastRunId       NVARCHAR(50)
);

A parent pipeline does a Lookup against LoadControl to get the list of active sources, then drives a ForEach loop over that list — one child-pipeline invocation per source, each with the source's own query and watermark passed in as parameters ^{[S6] [S3]}. Each child run's Stored Procedure activity only advances LastWatermark after its Copy activity reports success ^[S10], so a failed run simply doesn't move the watermark, and re-running that one source picks up exactly where it left off.

Best practices#

Centralize environment-specific values in a variable library, not hard-coded parameter defaults. The reason: a variable library reference lets the same schedule definition promote across dev/test/prod without editing the pipeline, where a static default value has to be hand-edited at each promotion ^[S4].

// Bad: environment baked into the pipeline's default parameter value
{
  "parameters": { "targetConnectionId": { "defaultValue": "guid-for-dev-sql-server" } }
}

// Good: schedule references a variable library entry, resolved per environment
{
  "parameters": {
    "targetConnectionId": { "value": "@pipeline().libraryVariables.TargetSqlConnectionId" }
  }
}

Match schedule parameter names to pipeline parameter names exactly. A mismatched name is silently ignored at run time rather than raising an error, so a typo produces a default value instead of a visible failure ^[S4] — which is exactly the kind of silent-wrong-data bug a metadata-driven framework is supposed to prevent, not introduce.

Guard trigger-derived metadata with null-safe expressions. Trigger metadata such as the triggering file name reaches pipeline expressions via parameters parsed from the event's Subject and Topic fields; because these are NULL during a manual test run, referencing them requires null-safe syntax like @pipeline()?.TriggerEvent?.FileName rather than an unguarded reference that would fail during testing ^[S11].

Split structured configuration from runtime control flow in Spark Job Definitions. SJD parameterization separates two concerns: structured, versioned configuration data (zone or load-group settings) is best held in YAML files read from OneLake via ABFSS paths, while runtime control flow is best handled through argparse command-line arguments with type enforcement and validation ^[S12]. Mixing the two — stuffing zone configuration into command-line flags, or trying to pass a run date through a YAML file — loses the versioning benefit of the former and the validation benefit of the latter.

What goes wrong#

Two failure modes recur often enough to call out explicitly.

Config-schema creep. A metadata contract that starts as three or four fields (source, target, load mode) tends to accumulate special-case flags over time — one more boolean for one more edge case — until the "generic" contract has quietly become an untyped, undocumented programming language that only its original author can safely extend. Inference: this failure mode is not asserted by a single cited source but follows directly from the contract/engine split described above — the fix is narrower, purpose-built contracts for genuinely different load shapes, not one contract stretched to cover everything.

Tuning throughput before finding the bottleneck. Raising Copy activity parallelism or intelligent throughput optimization cannot push a run past a bottleneck that lives outside the activity: if the constraint is the source store's IOPS or the network path, more threads or more compute simply queue up behind that external limit ^[S5]. A metadata-driven framework that fans out many sources in parallel via ForEach makes this worse, not better, if the underlying source system is the shared constraint — explicitly lowering the degree of copy parallelism is the documented way to protect a fragile source system by capping the concurrent load a fan-out places on it ^[S5].

Governing the metadata layer#

A metadata-driven framework doesn't just move data — it becomes the thing that decides what data goes where, which makes it a governance surface in its own right. Fabric's governance model gives this a natural home. Governance settings operate in a three-tier hierarchy: tenant-wide defaults, domain-level overrides for delegated settings, and workspace-level controls for the most granular scope ^[S15] — the same hierarchy a metadata contract's own precedence rules (tenant default, then domain override, then per-source override) should mirror rather than invent from scratch.

Lineage tracking closes the loop between the contract and what actually happened. Because Fabric integrates with Microsoft Purview, data lineage can be tracked end-to-end from an external source through to whatever ultimately consumes the data ^[S4], and metadata scanning exposes admin REST APIs that external cataloging tools can call to retrieve item-level metadata across all Fabric items — name, ID, sensitivity label status, endorsement status ^[S15]. A practical layered governance pattern applies different controls per medallion layer: bronze restricted to engineering roles with sensitivity labels at ingestion, silver adding schema validation and row filters, gold requiring catalog certification and a completed DLP scan before being marked discoverable ^[S11]. A metadata-driven framework is a natural enforcement point for exactly this kind of per-layer policy, since every object passing through it already carries its layer and classification as contract fields.

Internals#

Architecture & design#

The shape described above — contract, engine, ledger — maps directly onto Fabric's own building blocks rather than requiring anything bespoke. The contract lives in pipeline parameters, a variable library, or an external control table ^{[S3] [S4]}. The engine is the pipeline's control-flow layer (ForEach, If-Condition, Until) driving Copy activities and notebook/Spark Job Definition activities ^{[S5] [S6]}, with Fabric Data Factory able to chain data movement, transformation activities (notebooks, Spark job definitions, stored procedures, SQL scripts, dbt jobs), and control-flow constructs all from one low-code designer ^[S12]. The ledger is the run history, monitoring hub, and, optionally, a control table's own status columns ^{[S8] [S9]}.

Event-driven triggering extends this architecture past scheduling. Rather than a bespoke trigger service, event-based pipeline triggers are built on existing Fabric platform plumbing: defining a storage event trigger creates an eventstream object plus a Data Activator alert stored as a Reflex item in the workspace, and triggers can respond to file events, job events, and workspace events ^[S11]. The recommended pattern for event-driven data movement replaces a scheduled trigger with an event trigger entirely: define a Copy job first, then create an Activator rule whose condition triggers that Copy job as its action ^[S13]. This matters for metadata-driven frameworks specifically because fixed-schedule data movement carries three recurring costs — compute waste from empty runs when nothing changed, a latency-versus-cost tradeoff tied to schedule frequency, and operational overhead from managing many schedules across a large table count ^[S13]. Event-driven triggering removes exactly the schedule-sprawl cost that a metadata-driven framework would otherwise multiply across every source it manages.

How it works internally#

At the notebook layer, the contract-to-execution handoff runs through mssparkutils.runtime.getParameterValue, letting an orchestrating pipeline pass a run date (or any other contract field) into a notebook so it reads only the corresponding partition ^[S7]. At the packaged-application layer, a Fabric Spark Job Definition is the platform's equivalent of spark-submit: a packaged, parameter-driven application runner, distinct from the cell-by-cell interactive model of a notebook ^[S9]. Configuring one requires five components — a single entry-point script, optional reference files, optional command-line arguments, a lakehouse reference that sets the default metastore context, and an environment reference supplying libraries and Spark pool configuration ^[S9]. Unlike a notebook, a Spark Job Definition does not auto-inject a SparkSession or pre-load common imports; the entry-point script must explicitly instantiate its own session and import any helper utilities it needs ^[S9] — a detail that matters because a metadata-driven Spark engine built as an SJD carries more explicit bootstrapping than the equivalent notebook version.

On the read side, the metadata layer that resolves table structure (as opposed to pipeline configuration) has its own internals worth understanding, because a metadata-driven framework's target tables are usually queried back out through the SQL analytics endpoint. Automatic metadata discovery for a lakehouse's SQL analytics endpoint runs as a single background sync instance per Fabric workspace, and that one instance is responsible for scanning every lakehouse in the workspace for committed changes — the scan work is serialized across lakehouses rather than parallelized per lakehouse ^[S14]. A newer metadata sync architecture, in preview, replaces log re-scanning with an external-tables-based approach for parsing Delta logs and building the SQL catalog, but it only applies to newly created SQL analytics endpoints in workspaces that opt in; existing endpoints stay on the legacy sync path ^[S14]. That new path also has documented gaps: it doesn't support Delta's deprecated multi-part checkpoint feature, and it currently cannot be enabled on workspaces using workspace private link ^[S14].

Performance characteristics#

The single-instance, serialized nature of metadata sync is a documented, workspace-level scaling ceiling: because sync is one instance per workspace, a workspace holding many lakehouses is a documented cause of increased sync latency, and Microsoft's stated mitigation is to split lakehouses across separate workspaces so each workspace's automatic metadata discovery scales independently ^[S14]. A metadata-driven framework that fans many sources into a shared lakehouse should treat this as a real constraint on how many target lakehouses one workspace can host before schema-visibility latency becomes noticeable to downstream SQL consumers.

At the Copy activity layer, the two levers a metadata-driven engine actually tunes — intelligent throughput optimization (ITO) and degree of parallelism — are documented as orthogonal: ITO sizes the compute resources for the run, degree of parallelism controls thread fan-out, and by default the service chooses parallelism dynamically from the source-destination pair and observed data pattern ^[S5]. ITO itself trades cost against speed across four presets: Auto lets the service pick freely, Standard stays within standard compute resources, Balanced weighs throughput against available compute, and Maximum applies all available compute to the run ^[S5]. Neither lever helps once the real constraint sits outside the activity — raising ITO or parallelism cannot push throughput past a source store's IOPS ceiling or network path limit ^[S5] — which is why explicitly lowering the degree of parallelism, not raising it, is the documented way to protect a fragile source system that a metadata-driven fan-out would otherwise hit with many concurrent copies at once ^[S5].

At the pipeline-fleet layer: a single pipeline can carry up to 20 schedules ^[S3], and a workspace has a documented ceiling of 150 combined warehouse and SQL analytics endpoint items, after which creating another requires deleting one first ^[S15] — both real, numbered constraints on how large a single-workspace metadata-driven deployment can grow before it needs to be split across workspaces.

Worked example: an event-driven, metadata-driven ingestion framework#

Pulling the pieces together into one end-to-end scenario: an analytics team ingests daily extracts from twelve regional sales databases into one bronze lakehouse.

Contract. A control table holds one row per region, with the source connection GUID (captured from Manage connections and gateways and substituted dynamically via a string parameter ^[S3]), the target table name, the last successful watermark, and the last run's status and run ID.

Trigger. Instead of twelve schedules (bumping toward the 20-schedules-per-pipeline ceiling ^[S3] once other pipelines share the workspace), a Copy job for each region's landing path is set up first, then an Activator rule triggers it on file arrival ^[S13] — removing the fixed-schedule costs of wasted empty runs and per-schedule operational overhead ^[S13].

// simplified Activator rule condition (illustrative shape, not a literal API payload)
{
  "condition": "OneLakeFileEvent.path startswith '/bronze/regionsales/{region}/incoming/'",
  "action": { "type": "CopyJob", "target": "CopyJob_RegionSales_{region}" }
}

Engine. A parent pipeline does a Lookup against the control table for active regions, then a ForEach loop invokes a child pipeline per region ^[S6], parameterized with that region's connection GUID and last watermark ^[S3]. The child pipeline's Copy activity pulls only rows newer than the watermark ^[S10], writes to the bronze lakehouse, and hands off to a notebook that reads its own run-date parameter via mssparkutils.runtime.getParameterValue ^[S7] to apply schema conformance before writing silver.

Ledger. On success, a Stored Procedure activity advances that region's watermark and status in the control table ^[S10]; the pipeline's own run ID and the monitoring hub's audit trail ^[S8] give a second, independent record of what ran and when, so the control table and Fabric's own monitoring data cross-check each other.

Restart. If region seven's source database times out mid-run, the other eleven regions' watermarks have already advanced independently (each ForEach iteration is its own child pipeline run), and the failed run for region seven can be resumed from the failed activity rather than restarted from scratch ^[S9] — re-reading the same watermark it failed on, because that watermark was never advanced in the first place ^[S10].

Inference: the Activator-rule JSON above is illustrative of the shape described in the cited pattern, not a literal reproduction of a Microsoft API payload — exact condition/action syntax should be verified against the current Activator authoring UI.