getting only 1 record while transferring data from TDV to Cloud Postgres DB using ADF

Lakshmi Narayana Sarma Bhamidipati 5 Reputation points
2023-09-13T12:34:16.1833333+00:00

trying to load data from TDV(Tibco Data Virtualization ) to Cloud PostgresSQL DB using Azure DAta Factory. But we are getting only 1 record every time. Please let me know what the issue is here. Thanks in advance.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
8,054 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amira Bedhiafi 4,631 Reputation points
    2023-09-13T15:16:43.61+00:00

    When transferring data from TDV to Cloud PostgresSQL DB using Azure Data Factory (ADF) and only one record is transferred, you should first check the query in TDV to ensure it's pulling multiple records. Next, review the source and destination settings in ADF to ensure no filters, limits, or transformations are limiting the transfer. Examine the logs in ADF for any error messages or signs of connection issues. Ensure data integrity in TDV doesn't have formatting issues that could limit transfer. Finally, confirm the versions of TDV, Postgres, and ADF are compatible. If issues persist, consider reaching out to Azure or Tibco support for more detailed troubleshooting.

    0 comments
  2. SatishBoddu-MSFT 18,611 Reputation points
    2023-09-15T04:06:20.4733333+00:00

    Hello @Lakshmi Narayana Sarma Bhamidipati Please let us know if you need to add more info so that we better assist you!
    Please send an email to us and we will work with you offline.

    Additionally you may want to try the following steps:

    Increase the timeout value: The "Activity timed out" error suggests that the ADF activity is taking longer than the default timeout value to complete. You can try increasing the timeout value in the ADF pipeline settings to allow more time for the activity to complete.

    Check the ODBC driver version: Ensure that you are using the latest version of the ODBC driver for TDV. You can check the version of the driver in the ODBC Data Source Administrator.

    Check the network connectivity: Ensure that there are no network connectivity issues between the ADF integration runtime and the TDV service. You can test this by pinging the TDV service from the integration runtime.

    Check the query: Ensure that the query you are using to pull data from TDV is correct and is not limiting the number of records returned. You can test this by running the query manually in TDV.

    Check the destination settings: Ensure that the destination settings in ADF are correct and are not limiting the number of records that can be transferred. You can check this by reviewing the destination settings in the ADF pipeline.

    0 comments