ADF incremental loads error
I am trying to make a pipeline in ADF and I have the following problem.
To start with the data flow works as follows: starting from two tables, customers and alarms, we must join them both by a common id in both tables to get a single table. So far so good and it works correctly. The problem is when I want to make the data flow incremental, that is to say, that it only copies the new data to the output. For this, both source tables have a column called "created_time" which is in UNIX format or rather BIGINT in postgres (which is the source of both tables) so ADF recognises it as long.
When trying to do the incremental load I check the option that ADF brings as shown in the following image:
however, when activating the trigger I get the following error: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): Incompatible CDC option,watermarkColumn is not present when it tells us that there is no watermark column, I deduce that it is because no column has been selected, so I select the previously mentioned "created_time":
but I get an error again: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): not a valid data-type should be one of { date , datetime , timestamp } so I try changing the format of this column to timestamp
Original:
Tranformed:
but the same error: Job failed due to reason: at Source 'Customer'(Line 1/Col 0): not a valid data-type should be one of { date , datetime , timestamp }
Could someone help me with this error so I can do incremental data loads?
Thank you in advance