Time Zone Parameter Ignored in Azure Data Factory PostgreSQL Connector After Migration

icatii 25 Reputation points
2024-10-11T11:52:38.2166667+00:00

I'm experiencing an issue with Azure Data Factory (ADF) when using the PostgreSQL connector.

  • Problem:

The "Time Zone" parameter in the Linked Service appears to be ignored, and timestamps are always retrieved in UTC, despite attempts to set the time zone to Poland (Europe/Warsaw). This problem began after migrating to the new version of the ADF PostgreSQL connector.

  • Details: Source Database Time Zone: Europe/Sarajevo Objective: Retrieve and process timestamps in the Europe/Warsaw time zone, including daylight saving time adjustments.
  • Constraints: • The PostgreSQL database is managed by a third party; we cannot make any modifications to it. • Previously, with the old ADF PostgreSQL connector, timestamps were correctly converted. • We have hundreds of datasets and pipelines, so modifying each individually is impractical.

What we tried (among other things :)):

  • Set "Time Zone" Parameter in Linked Service: Tried values:
    	○ Europe/Warsaw
    
    	○ 'Europe/Warsaw'
    
    	○ Poland
    
    	○ +01:00
    
    	○ UTC+01:00
    
    	○ CET
    
    	○ CEST (raises an error)
    
    • Result: Returned timestamp values from Source remain in UTC; parameter seems to have no effect. However I executed the SHOW TIMEZONE command in pipeline->CopyData->Query window and the value was correct and aligned with Linked Service Time Zone parameter.
  • Modify Source Query: • Used AT TIME ZONE 'Europe/Warsaw' in SELECT statements. • Result: Timestamps properly converted • Limitation: Not feasible to modify every query due to the large number of datasets.
  • Questions: • Is the "Time Zone" parameter functional in ADF's PostgreSQL connector? • What are the accepted values for this parameter?
  • Notes: • Problem - timezone parameter in Linked Service seems to be ignore when retrieveing data • The source PostgreSQL database's time zone is set to Europe/Sarajevo. • We cannot make any changes to the source database. • The issue started after migrating to the new ADF PostgreSQL connector; the previous version handled time zone conversion correctly. • Modifying hundreds of datasets and pipelines individually is not feasible.
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,870 questions
Azure Database for PostgreSQL
{count} vote

Accepted answer
  1. Sina Salam 12,166 Reputation points
    2024-10-11T15:04:02.38+00:00

    Hello icatii,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    I understand that you are having issues with Time Zone Parameter Ignored in Azure Data Factory PostgreSQL Connector After Migration.

    You would need to handle time zone conversions across your datasets without altering individual queries by Mapping Data Flows in Azure Data Factory - https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-overview this method will convert timestampColumn to the Europe/Warsaw timezone, accounting for daylight saving time adjustments and you should also report this issue to Azure Support via your Azure Portal to see if there is a bug fix or patch for the PostgreSQL connector's "Time Zone" parameter. I will advise, automation with PowerShell or post-processing transformations which are also valid approaches to solve this at scale https://learn.microsoft.com/en-us/powershell/module/az.datafactory/?view=azps-9.0.0

    I hope this is helpful! Do not hesitate to let me know if you have any other questions.


    Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.