A community member has associated this post with a similar question:
ADF incremental load error

Only moderators can edit this content.

ADF incremental load error

Andres Esteban 156 Reputation points
2023-08-07T16:29:13.6866667+00:00

I am trying to perform an incremental load from a date in UNIX format (bigint) and for this, I am using a watermark. So I have the following fonts:

  • Source DB (PostgreSQL)
  • DB where the new data is copied to (Azure PostgreSQL)
  • Watermarked table (Azure SQL Database) The copy process is as follows, with two searches to copy the data > to the watermark date and <= to the maximum date of the source DB. Once the data has been copied (THIS WORKS WELL FOR ME) the watermark date must be updated so that the next copy copies from there.

User's image

The problem I have is when updating the watermark. To do this, it looks for the maximum date in the source DB (bigint format) and must update it in the watermark table (configured as bigint).

User's image

However, the mapping is done as int64 and I get the following error when updating the watermark

"Cannot create Sql Source. Please double check the connection string, stored procedure are set with correct format. Error: The value of the property 'Value' is invalid for the stored procedure parameter 'watermark_value'."

Does anyone know how to fix it so that the watermark table is updated with the new value of the maximum date (UNIX)?

Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
{count} votes