Tracing Stored Procedure Event in Azure Synapse pool & Handling Duplicates (No Composite Primary Key)

Swapnil Kalkar 66 Reputation points
2020-06-17T16:36:46.497+00:00

Hi Team,

I am loading CSV file data from Azure Blob to Azure DW in external table.
Through ADF pipeline I am calling Stored Procedure which loads data from ext. to staging table.
I have created a Trace table where I wanted log info such as Date, SP name, No of records inserted each time when stored procedure is called
so that I can track easily daily process.

Other scenario is that, as we don't have Composite Primary Key concept in Azure DW,
How to deal with duplicates while loading data from Azure Blob to Azure external table? How to deal with any bad data getting loaded from Blob as we cant go with some constraints?

Your help is appreciated in this regard

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,599 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,653 questions
{count} votes

Accepted answer
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2020-06-23T22:10:21.487+00:00

    Hello @SwapnilKalkar-4467 ,
    Thanks for your patiance here .
    On checking duplicate part of the question , i think you are refering to the use case where you do not want to have duplicate records on the table , if that the ase you can use WHERE NOT EXISTS ().

    Since in your case you are using the SP , i think you can put the dedup logic in the SP itself .

    Let me know if you have any further questions .

    Thanks & stay safe

    Himanshu

    Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful