ETL Process PostgreSQL in Azure Data Factory

Andres Esteban 51 Reputation points
2023-01-03T15:44:06.28+00:00

Hello community

I have to perform an ETL process which consists of copying certain data (i.e. filtering is required) from a PostgreSQL DB to Azure PostgreSQL via Azure Data Factory, and ideally only new records should be copied once a day. I have been reading and as far as I can see there are two options:

  • Create a Power Query: would this be possible for Postgres and would it be possible to only copy the new data with each query?
  • Create a Lookup + copy data: would it be possible for Postgres and would it be possible to copy only the new data with each query?

The query I make to visualise the data I want is something similar to this:

SELECT *
FROM table
where column1='condition'
and user_name<>xxx@microsoft.com'

Is an ETL process doing this kind of queries possible in Azure Data Factory?

Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,294 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MarkKromer-MSFT 5,136 Reputation points Microsoft Employee
    2023-01-03T20:50:23.71+00:00

    Yes, you can do this using a source query either from a Copy Activity or a Data Flow inside your ADF pipeline