How to share large data sets to third party data consumers

Question

How to share large data sets to third party data consumers

Shashank Singh 6,251

Hi,

Lets assume I have a client who has plethora of data related to railways(Signals, tracks, train timings, hazard. offers etc). There are various internal department in railways which wants that data. Just like various weather websites get data from weather department and show that data on their website. Similar is my requirement that I want to share the data secuely with other department and services. I want to look at best method to share the data to other services as quickly as possible when the data is available.

Possible Solution

API based: Create API for each department and share them data via API. This has its own pros and cons. This is something which came to my mind but we would have to create lot go API's I was looking if Azure and AWS has any other service which can do the same.
Azure based solution: I am looking for help in this if Azure and services it provides can help. ADF, Databricks, service bus etc would not solve the problem as per me. Let me know if I am incorrect.
AWS based solution: Is there any service in AWS which can help here? I understand this being MS forum I would not get much traction on this option but I have put this for clarity.
Any other solution ?

MartinJaffer-MSFT 26,161 Reputation points

2023-02-10T19:02:45.92+00:00

@Shashank Singh depending upon where you store the data and how you want to share, check out Azure Data Share
Shashank Singh 6,251 Reputation points

2023-02-13T05:33:12.8233333+00:00

I understand this is just a way of sharing data. Is there any real time service which can extract data and send it to third party API's or storage account.
MartinJaffer-MSFT 26,161 Reputation points

2023-02-17T20:23:21.86+00:00

@Shashank Singh Depends upon whether this is pull model or push model.

If things get written to Azure Storage (blob or gen2), then Event Grid can leverage file creation events to make something else happen.

If you have streaming data, Event Hubs or Stream Analytics can help.

If you are willing to make custom code, HDInsight is an option.

You mentioned railway and weather data. Those sometimes come from IOT devices. In those cases try IOThub.
AnnuKumari-MSFT 34,571 Reputation points Microsoft Employee Moderator

2023-02-24T15:37:24.0566667+00:00

@Shashank Singh ,

Just checking if the above suggestion helped? Please let us know if the query has been resolved for you