Exposing Azure Real Time data in data lake to Other Cloud Vendors

Question

Exposing Azure Real Time data in data lake to Other Cloud Vendors

V S Deepak 1

Hi All,

We have a requirement of exposing the real time data that is coming from multiple sensors with latency of seconds. At the present the data is residing in Azure data lake. We need to expose these data to customer clouds that might be AWS, Azure ,GCP or any xyz cloud. Requirement is to build a generic solution so that we don't need to build a multiple solutions for each cloud vendors.

As of now we have two options Push vs Pull.

Push Mechanism - Build ADF Pipelines which will push data to Customer SFTP On daily basis aggregating all the files per day, since its real time we will receive multiple files depending upon the threshold. This is more secure as we don't allow customers to talk to our data lake. Pipelines will be data driven based on config tables. Apart from egress charges per TB , also need to pay for Data factory instance. Downside is on cost and also we need to maintain a log for all vendors of when the data is pushed along with retry mechanism is case of failure.
Pull Mechanism - Expose the Azure data lake data via Azure API management ,Similarly data would be pulled by client on a daily basis in a consolidated file. Along with egress we have API cost here which I suppose will be quite less when compared to ADF.

Let me know , which approach is more scalable as requirement changes with near real time to expose data to other vendors.

Many Thanks,
Deepak

MartinJaffer-MSFT 26,161 Reputation points

2022-08-23T20:59:55.843+00:00

Hello @V S Deepak and welcome to Microsoft Q&A.

As I understand, the ask is for advice on how to get your streamed real-time data to your client/customers who may be in other clouds.
What confuses me, is your 2 listed options are both batch operations (opposite of streaming).

I was thinking, "why not use Event Hubs to split your stream and allow the customers to subscribe / pull data from a queue at their own pace?" This would allow lower latency than the daily aggregation present in both of your push/pull options. IOT Hub is like Event Hubs, but specialized for IOT data like sensors.

On another note, you could use Azure Data Share to push folders or even entire containers from your data lake to another data lake (or other select services), one owned by the customer. Data Share also tracks snapshot history and success. For lake-to-lake sharing, it can do incremental and probably easier to manage than Data Factory. You can revoke the process at any time, stopping their access to updates.
V S Deepak 1 Reputation point

2022-08-29T07:21:10.173+00:00

Hi @MartinJaffer-MSFT ,

Thanks for your suggestions and Data share looks promising for Azure to Azure Cloud migration. Which I am currently looking into

1 answer

Your answer

MartinJaffer-MSFT 26,161 Reputation points

2022-08-23T20:59:55.843+00:00

Hello @V S Deepak and welcome to Microsoft Q&A.

As I understand, the ask is for advice on how to get your streamed real-time data to your client/customers who may be in other clouds.
What confuses me, is your 2 listed options are both batch operations (opposite of streaming).

I was thinking, "why not use Event Hubs to split your stream and allow the customers to subscribe / pull data from a queue at their own pace?" This would allow lower latency than the daily aggregation present in both of your push/pull options. IOT Hub is like Event Hubs, but specialized for IOT data like sensors.

On another note, you could use Azure Data Share to push folders or even entire containers from your data lake to another data lake (or other select services), one owned by the customer. Data Share also tracks snapshot history and success. For lake-to-lake sharing, it can do incremental and probably easier to manage than Data Factory. You can revoke the process at any time, stopping their access to updates.
V S Deepak 1 Reputation point

2022-08-29T07:21:10.173+00:00

Hi @MartinJaffer-MSFT ,

Thanks for your suggestions and Data share looks promising for Azure to Azure Cloud migration. Which I am currently looking into

Answer 1

Sander van de Velde | MVP 37,066 MVP

Hello @V S Deepak ,

there as several approaches to tackle these real-time distribution requirements.

If you want to keep the data lake approach, an IoT Hub can route (sets of) telemetry to an Azure storage blob container, both in JSON and AVRO format. You can play with the size of the blobs. If you trigger an Azure Function based on new files added, you can expose eg. a message on a (public) event hub, call a webhook or expose a generic cloud event using EventGrid.

You can also expose cloud events for each incoming telemetry message which will give you real-time behavior. Check out EventGrid support for IoT Hub.

If you want to change the telemetry first, you will need to add other azure logic between IoT Hub and your point of exposure.

I do not recommend exposing the 'eventhub compatible endpoint' of the IoT Hub directly. Yes, it looks like an EventHub but it does not offer any configuration flexibility and control over scalability compared to the other services mentioned above.

MartinJaffer-MSFT 26,161 Reputation points

2022-08-24T19:01:54.787+00:00

Those are some really good points @Sander van de Velde | MVP

While I do touch upon Event Hubs in my work, IOT Hub is not something I work with. I didn't know about the difference of configuration flexibility. Thank you for the insights. Keep it up!

So you think streaming is better than the push or pull?
Sander van de Velde | MVP 37,066 Reputation points MVP

2022-08-24T20:52:41.447+00:00

Hello @V S Deepak ,

I you want to learn more about the Azure IoT Hub, check the documentation of follow the free learning modules (part of several learning paths regarding Azure IoT).

Using the streaming capabilities is in fact push.

Again, the right solution depends on your (non-functional) requirements.

----------

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. By doing so, all community members who have a similar issue will benefit. Your contribution is highly appreciated.

Share via

Exposing Azure Real Time data in data lake to Other Cloud Vendors

1 answer

Your answer