Architecture for Log data analysis

Question

Architecture for Log data analysis

Divakar Krishnan 106

Dear Team,

We have a requirement to ingest the Network user log traffic data and build the Power BI report on top of it.

Since it’s a log data, we don’t want to store it in our Azure SQL Warehouse DB, and we would like to implement any other effective architecture in azure to ingest and store this data.

Log Data Sample Below:

Source for this file:
Network application from vendor will generate this file on hourly basis and transfer this file to our SFTP. So we can say that we need to process 24 files on each day with the huge log data in each file.

Please advise me on which architecture and which service in azure is best fit for this scenario, I prefer to use some live streaming to ingest the data. Kindly share your suggestion.
Thanks,
Divakar

PRADEEPCHEEKATLA 91,866 Reputation points

2022-04-26T06:48:04.47+00:00
Hello @Divakar Krishnan ,

Following up to see if the below suggestions was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

Answer accepted by question author

Nandan Hegde 36,886 MVP Volunteer Moderator

Hey,
Assuming you do have Azure SQL DW already available,
you can leverage ADF to upload files from SFTP to Azure blob , Then leverage external table to read data from file rather than explicitly loading data into synapse.
And you can query the external table via PowerBI.

Else you can leverage ADF to upload in blob and access blob file directly via PowerBI.

Divakar Krishnan 106 Reputation points

2022-04-22T15:56:48.817+00:00

Hi @Nandan Hegde ,

With the accessing blob files directly via Power BI, will there be any impact in performance in power BI?

24 files will be generated, and I expect each file may have close to 30,000 to 50,000 records. So per day it's 1.2 million records.

I am just curious about Power BI performance capabilities with this approach. Please suggest.

Thanks,
Divakar
Nandan Hegde 36,886 Reputation points MVP Volunteer Moderator

2022-04-25T04:12:31.263+00:00

Hey, Accessing files from blob with the current amount of data shouldnt have any performance issues.

1 additional answer

Your answer

PRADEEPCHEEKATLA 91,866 Reputation points

2022-04-26T06:48:04.47+00:00

Hello @Divakar Krishnan ,

Following up to see if the below suggestions was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.
Divakar Krishnan 106 Reputation points

2022-04-22T15:56:48.817+00:00

Hi @Nandan Hegde ,

With the accessing blob files directly via Power BI, will there be any impact in performance in power BI?

24 files will be generated, and I expect each file may have close to 30,000 to 50,000 records. So per day it's 1.2 million records.

I am just curious about Power BI performance capabilities with this approach. Please suggest.

Thanks,
Divakar
Nandan Hegde 36,886 Reputation points MVP Volunteer Moderator

2022-04-25T04:12:31.263+00:00

Hey, Accessing files from blob with the current amount of data shouldnt have any performance issues.

Answer 1

David Broggy 6,801 MVP

Hi @Divakar Krishnan
plan B would be to use Azure Data Factory with Log Analytics.

2M records for a log analytics workspace should be a joke.
I work with billions of records per day using Sentinel, which depends on log analytics for everything.

If this helps please accept my solution and upvote.
Or just have a nice day.

0 comments