Testing load logic efficacy

Question

Testing load logic efficacy

Flint Harrell 1

So I have flat files that are being pulled from Blob storage and being transformed into an analytical model in Azure SQL database. Is there a way to test the data loaded matches what was in the flat files natively? I was thinking I could use Databricks manually for this type of thing, but ideally I would like it to occur as part of the pipeline so I can implement failover.

ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2021-08-16T07:32:32.157+00:00

Hi @Flint Harrell ,

Welcome to Microsoft Q&A Platform. Thank you for posting your query here.

Could you please confirm which activity you are using for data load in Azure SQL database? Is it copy activity or data flow activity?

Copy activity or Data flow activity output json will have some useful information related to your data movement. is that's not helpful in your case ?
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2021-08-19T05:08:57.757+00:00

Hi @Flint Harrell ,

Just checking did you get chance to share details on above commented asks? Thank you.
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2021-08-24T07:25:43.11+00:00

Hi @Flint Harrell ,

Did you get chance to share details on above commented asks? Thank you.