Overview#

Fabric data engineering spans code-first Spark, orchestration-focused Data Factory pipelines, high-volume Copy activity, and low-code Dataflow Gen2 preparation [S1] [S2] [S3]. These tools are complementary because they can all land data into OneLake and hand work to the next engine without leaving the Fabric environment [S1] [S2].

Fabric Spark Data Engineering

Core concepts#

Spark notebooks and Spark job definitions are the code-first path for scalable transformation, custom logic, and production engineering jobs [S1]. Data Factory pipelines orchestrate activities and data movement, while Copy activity is the preferred path for large-scale recurring or migration-style ingestion [S2] [S4]. Dataflow Gen2 uses Power Query for low-code preparation and can write transformed outputs into Fabric destinations including lakehouses [S3].

The practical decision is usually about authoring surface and workload shape. Use pipelines for orchestration, Copy activity for bulk movement, Dataflow Gen2 for low-code shaping, and Spark for complex transformation or jobs that need programmatic control [S1] [S2] [S3] [S4].

Data Factory ingestion decision matrix

How it works and best practices#

Separate movement from transformation. A common Fabric pattern is to land source data with Copy activity or shortcuts, then transform it in Spark or another compute engine over OneLake [S2] [S4]. Dataflow Gen2 is strongest when business users need a visual preparation surface before data lands in curated tables [S3].

Tune Spark through pool choice and file layout. Starter pools reduce interactive startup time, custom pools expose node and scaling choices, and file settings such as optimized write, compaction, and V-Order shape downstream read performance [S1] [S5] [S6].

Implementation example#

A robust engineering flow separates orchestration, movement, and transformation. Data Factory coordinates the run, Copy activity handles scalable movement, Spark applies code-first transformations, and Dataflow Gen2 is reserved for low-code Power Query preparation where that authoring surface is the right fit [S1] [S2] [S3] [S4].

yaml
pipeline:
  trigger: nightly
  activities:
    - name: copy_source_extracts
      tool: data_factory_copy_activity
      output: lakehouse.files/raw
    - name: transform_silver_gold
      tool: spark_job_definition
      input: lakehouse.files/raw
      output: lakehouse.tables/gold
    - name: business_owned_cleanup
      tool: dataflow_gen2
      when: low_code_power_query_is_preferred

Inference: this YAML is an orchestration sketch. The sourced mechanics are Data Factory orchestration, Copy activity scale controls, Spark engineering workloads, and Dataflow Gen2's low-code Power Query model [S1] [S2] [S3] [S4].

python
# Illustrative notebook transform: land raw data once, then write curated Delta tables.
raw = spark.read.format("parquet").load("Files/raw/orders/")
silver = raw.dropDuplicates(["OrderId"]).filter("OrderDate IS NOT NULL")
silver.write.format("delta").mode("overwrite").saveAsTable("silver_orders")

The code illustrates a code-first Spark transform path; performance-sensitive versions should add file-layout controls such as compaction, optimized write, and V-Order decisions based on the workload's read/write pattern [S5] [S6] [S7].

What goes wrong#

Using notebooks as a scheduler makes lineage and operational control harder than using pipelines for orchestration [S2]. Using Dataflow Gen2 for heavyweight engineering logic can also hide complexity in a low-code surface better suited to repeatable preparation steps [S3].

Internals#

Architecture & design#

Fabric Spark runs on managed pools and writes Delta/Parquet tables into OneLake, while pipelines coordinate activities that can move data, invoke notebooks, or run transformations [S1] [S2]. Dataflow Gen2 uses the Power Query authoring model and writes prepared results into supported destinations [S3].

How it works internally#

Spark file-layout controls change the physical Parquet output before downstream engines read it [S5] [S6]. Native execution engine support can accelerate eligible Spark workloads by offloading supported plan fragments to a native vectorized execution path while preserving fallback for unsupported operations [S7].

Performance characteristics#

Copy activity exposes parallelism controls for scalable movement, while Spark performance depends on pool sizing, execution engine support, shuffle behavior, and file count [S4] [S5] [S7]. Overproducing small files harms later SQL and BI reads, so compaction and optimized write are engineering controls as much as storage controls [S5] [S6].

Worked example#

For a nightly customer data pipeline, use Copy activity to land raw extracts, Spark to cleanse and conform the data into Delta tables, and a pipeline to orchestrate dependencies and failure handling [S1] [S2] [S4]. Use Dataflow Gen2 instead of Spark for source-specific shaping when the logic is business-owned and naturally expressed in Power Query [S3].