Share via

ADF incremental loads error

Andres Esteban 156 Reputation points
2022-09-29T10:45:05.423+00:00

I am trying to make a pipeline in ADF and I have the following problem.

To start with the data flow works as follows: starting from two tables, customers and alarms, we must join them both by a common id in both tables to get a single table. So far so good and it works correctly. The problem is when I want to make the data flow incremental, that is to say, that it only copies the new data to the output. For this, both source tables have a column called "created_time" which is in UNIX format or rather BIGINT in postgres (which is the source of both tables) so ADF recognises it as long.

When trying to do the incremental load I check the option that ADF brings as shown in the following image:

245979-image.png
245969-image.png

however, when activating the trigger I get the following error: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): Incompatible CDC option,watermarkColumn is not present when it tells us that there is no watermark column, I deduce that it is because no column has been selected, so I select the previously mentioned "created_time":

246035-image.png
246036-image.png

but I get an error again: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): not a valid data-type should be one of { date , datetime , timestamp } so I try changing the format of this column to timestamp

Original:
246081-image.png
Tranformed:
246082-image.png

but the same error: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): not a valid data-type should be one of { date , datetime , timestamp }

Could someone help me with this error so I can do incremental data loads?
Thank you in advance

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.


Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.