Yes, you can do this using a source query either from a Copy Activity or a Data Flow inside your ADF pipeline
ETL Process PostgreSQL in Azure Data Factory

Andres Esteban
51
Reputation points
Hello community
I have to perform an ETL process which consists of copying certain data (i.e. filtering is required) from a PostgreSQL DB to Azure PostgreSQL via Azure Data Factory, and ideally only new records should be copied once a day. I have been reading and as far as I can see there are two options:
- Create a Power Query: would this be possible for Postgres and would it be possible to only copy the new data with each query?
- Create a Lookup + copy data: would it be possible for Postgres and would it be possible to copy only the new data with each query?
The query I make to visualise the data I want is something similar to this:
SELECT *
FROM table
where column1='condition'
and user_name<>xxx@microsoft.com'
Is an ETL process doing this kind of queries possible in Azure Data Factory?
Thank you
1 answer
Sort by: Most helpful
-
MarkKromer-MSFT 5,136 Reputation points Microsoft Employee
2023-01-03T20:50:23.71+00:00