Getting data from QAD ERP system is challenging

Question

Hi all,

I have been working on the data warehouse project where I am trying to integrate multiple ERP system from different country using Azure data factory as my orchestrate tool and Azure sql database as my storage and transformation layer. Azure analysis service as my analysis layer.

I could able to get data from ERP system like ISAH from different regions in EU but the performance of getting QAD data from US is way slower and am getting time out error in my night run.

I have a stored procedure to get the tables from the QAD using a preselectquerybatch where I get data from 2022 , but the pipeline keeps running and does not retrieve any data from source system. Only the table named(ac_mstr) I could able to succeed in fetching data, the other table kept running where the data is neither read nor written(look at the screenshot attached).

Pipeline running for nearly 30 mins did not retrieve a single line of record( See the screenshot attached) .

People who worked with QAD could you please provide some suggestion on how to get data from QAD and the performance should be better.

Thank you in advance.

Answer 1

The issue with retrieving data from the QAD ERP system might be addressed by examining the query optimization, connection parameters, and timeout settings. Begin by analyzing the specific queries being run, optimizing them for performance, and employing proper indexing where needed. Additionally, tuning the connection parameters within Azure Data Factory to ensure that they align with the requirements of the QAD system could help. Increasing the timeout settings in the Data Factory pipeline might prevent the timeout errors, giving the system more time to process the data. Collaborating with QAD's support team or leveraging their documentation for specific connection guidelines and best practices could also provide insights tailored to your unique environment.