Edit

Share via


Quickstart: Create a Copy job

Copy jobs in Data Factory ingest data without the need to create a Fabric data pipeline. It brings together various copy patterns such as bulk or batch, incremental or continuous copy into a unified experience. If you only need to copy data without transformations, use a Copy job.

This quickstart guide walks you through how to copy data incrementally from a Fabric Warehouse table to a Fabric Lakehouse table using Copy job.

For more information about Copy jobs in general, see:

Prerequisites

Before you start, complete these prerequisites:

  • A Microsoft Fabric tenant with an active subscription. You can create a free account.

  • A Microsoft Fabric workspace..

  • A Fabric Warehouse.

  • A table in your warehouse that includes an incremental column, like a timestamp or an increasing integer column, that can serve as a watermark for change detection. You can also use this script to create a sample Employee table:

    CREATE TABLE dbo.Employee 
    
    ( 
        EmployeeID INT NOT NULL, 
        FirstName VARCHAR(40), 
        LastName VARCHAR(40), 
        Position VARCHAR(60), 
        ModifiedDate DATETIME2(3) 
    ); 
    

    Insert sample data:

    INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) 
    VALUES  
    (1, 'Alice', 'Smith', 'Data Analyst', SYSDATETIME()), 
    (2, 'Bob', 'Johnson', 'Engineer', SYSDATETIME()), 
    (3, 'Carol', 'Lee', 'Manager', SYSDATETIME()), 
    (4, 'David', 'Wong', 'Data Scientist', SYSDATETIME()), 
    (5, 'Eve', 'Garcia', 'Product Owner', SYSDATETIME());
    

    Screenshot of the created employee table.

Create a Copy job

  1. In your Microsoft Fabric workspace select + New Item, and under Get data choose Copy Job.

    Screenshot of the Fabric workspace with the new item button selected, and the copy job highlighted under get data.

  2. Name your Copy job, and select Create.

    Screenshot of the named copy job with the create button highlighted.

Configure Incremental Copy

  1. In the Choose data source page of the Copy job wizard, select your Fabric Warehouse.

    Screenshot of the choose data source page of the copy job creation wizard with a warehouse selected.

  2. In the Choose data page, select the source Warehouse table that includes the incremental column. Select Next.

    Screenshot of the choose data page with the employee SQL table selected.

  3. In the Choose data destination page, select Lakehouse under New Fabric item.

    Screenshot of the choose data destination page, with Lakehouse selected.

  4. Provide a name for the new Lakehouse and select Create and connect.

    Screenshot of the naming window for the new lakehouse with create and connect selected.

  5. On the Map to destination page, choose Tables, optionally rename the destination table, and select Next.

    Screenshot of the map to destination page with the employee table selected.

  6. In the Settings step, choose Incremental copy as the Copy Job mode. Select the column that serves as the incremental column. For the sample table, that's ModifiedDate.

    Screenshot of the settings page with incremental copy selected and the incremental column set to ModifiedDate.

Run and monitor the Copy job

  1. On the Review + save page, verify your settings. Leave the default option to Start data transfer immediately and optionally set the Copy job run every 1 minute for faster change tracking.

  2. Save the Copy Job and start the first execution by selecting the Save + Run button.

    Screenshot of the review and save page, with the run options set to run on schedule every one minute.

  3. Once the Copy job artifact is saved successfully, it starts the first run to bring in the initial data from the source table.

  4. Use the Copy Job panel or Monitoring Hub to monitor progress. Learn more about monitoring Copy Jobs.

    Screenshot of the monitoring hub, showing the successful job with five rows read and five rows written.

Simulate changes with new data

  1. If you used the sample table, use the following SQL query to insert new rows into the Source Fabric Warehouse table.

    INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) VALUES (6, 'John', 'Miller', 'QA Engineer', SYSDATETIME()); 
    INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) VALUES (7, 'Emily', 'Clark', 'Business Analyst', SYSDATETIME()); 
    INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) VALUES (8, 'Michael', 'Brown', 'UX Designer', SYSDATETIME()); 
    
  2. The Copy job uses the incremental column to detect these rows during its next scheduled run.

    Screenshot of the monitoring page showing the successful job with three rows read and three rows written.

  3. After the next run, query the target Fabric Lakehouse table to confirm the table was moved.

    Screenshot of the Fabric Lakehouse table, showing all rows written into the table.