ADF incremental loads error

Andres Esteban 156 Reputation points
2022-09-29T10:45:05.423+00:00

I am trying to make a pipeline in ADF and I have the following problem.

To start with the data flow works as follows: starting from two tables, customers and alarms, we must join them both by a common id in both tables to get a single table. So far so good and it works correctly. The problem is when I want to make the data flow incremental, that is to say, that it only copies the new data to the output. For this, both source tables have a column called "created_time" which is in UNIX format or rather BIGINT in postgres (which is the source of both tables) so ADF recognises it as long.

When trying to do the incremental load I check the option that ADF brings as shown in the following image:

245979-image.png
245969-image.png

however, when activating the trigger I get the following error: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): Incompatible CDC option,watermarkColumn is not present when it tells us that there is no watermark column, I deduce that it is because no column has been selected, so I select the previously mentioned "created_time":

246035-image.png
246036-image.png

but I get an error again: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): not a valid data-type should be one of { date , datetime , timestamp } so I try changing the format of this column to timestamp

Original:
246081-image.png
Tranformed:
246082-image.png

but the same error: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): not a valid data-type should be one of { date , datetime , timestamp }

Could someone help me with this error so I can do incremental data loads?
Thank you in advance

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
{count} votes