Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Copy jobs in Data Factory ingest data without the need to create a Fabric data pipeline. It brings together various copy patterns such as bulk or batch, incremental or continuous copy into a unified experience. If you only need to copy data without transformations, use a Copy job.
This quickstart guide walks you through how to copy data incrementally from a Fabric Warehouse table to a Fabric Lakehouse table using Copy job.
For more information about Copy jobs in general, see:
Prerequisites
Before you start, complete these prerequisites:
A Microsoft Fabric tenant with an active subscription. You can create a free account.
A table in your warehouse that includes an incremental column, like a timestamp or an increasing integer column, that can serve as a watermark for change detection. You can also use this script to create a sample Employee table:
CREATE TABLE dbo.Employee ( EmployeeID INT NOT NULL, FirstName VARCHAR(40), LastName VARCHAR(40), Position VARCHAR(60), ModifiedDate DATETIME2(3) );
Insert sample data:
INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) VALUES (1, 'Alice', 'Smith', 'Data Analyst', SYSDATETIME()), (2, 'Bob', 'Johnson', 'Engineer', SYSDATETIME()), (3, 'Carol', 'Lee', 'Manager', SYSDATETIME()), (4, 'David', 'Wong', 'Data Scientist', SYSDATETIME()), (5, 'Eve', 'Garcia', 'Product Owner', SYSDATETIME());
Create a Copy job
In your Microsoft Fabric workspace select + New Item, and under Get data choose Copy Job.
Name your Copy job, and select Create.
Configure Incremental Copy
In the Choose data source page of the Copy job wizard, select your Fabric Warehouse.
In the Choose data page, select the source Warehouse table that includes the incremental column. Select Next.
In the Choose data destination page, select Lakehouse under New Fabric item.
Provide a name for the new Lakehouse and select Create and connect.
On the Map to destination page, choose Tables, optionally rename the destination table, and select Next.
In the Settings step, choose Incremental copy as the Copy Job mode. Select the column that serves as the incremental column. For the sample table, that's ModifiedDate.
Run and monitor the Copy job
On the Review + save page, verify your settings. Leave the default option to Start data transfer immediately and optionally set the Copy job run every 1 minute for faster change tracking.
Save the Copy Job and start the first execution by selecting the Save + Run button.
Once the Copy job artifact is saved successfully, it starts the first run to bring in the initial data from the source table.
Use the Copy Job panel or Monitoring Hub to monitor progress. Learn more about monitoring Copy Jobs.
Simulate changes with new data
If you used the sample table, use the following SQL query to insert new rows into the Source Fabric Warehouse table.
INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) VALUES (6, 'John', 'Miller', 'QA Engineer', SYSDATETIME()); INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) VALUES (7, 'Emily', 'Clark', 'Business Analyst', SYSDATETIME()); INSERT INTO dbo.Employee (EmployeeID, FirstName, LastName, Position, ModifiedDate) VALUES (8, 'Michael', 'Brown', 'UX Designer', SYSDATETIME());
The Copy job uses the incremental column to detect these rows during its next scheduled run.
After the next run, query the target Fabric Lakehouse table to confirm the table was moved.