The Architecture

Overview

Implementation details

Core DB

Azure Data Factory

Data Warehouse

Azure Analysis Services

Provisioning the pipeline

Virtual Network

Azure Data Factory

Azure SQL Server

Adding a customer

1. Core DB + network configuration allowing access to it
2. Data warehouse database
3. ADF pipeline configured with connection details for the core DB and the customer's data warehouse database
4. Analysis services tabular model, both in Visual Studio and in AAS

Core DB / network

Data warehouse database

Analysis services model

ADF pipeline

Replicating the Analysis Services Model Template

1. Copy the 'aas' project folder. Rename the new folder something like 'aasCustomerA'.
2. File aasTemplate.smproj
   1. Rename the file to something like 'aasCustomerA.smproj'.
   2. Change <PropertyGroup>.<Name> from 'aasTemplate' to something like 'aasCustomerA'.
   3. Change all instances of <PropertyGroup>.<DeploymentServerDatabase> to something like 'aasCustomerA'.
   4. Change all instances of <PropertyGroup>.<DeploymentServerName> to match your AAS server name.
3. File Model.bim
   1. Change all instances of 'dwCustomerX' to something like 'dwCustomerA'. This is the data warehouse database name and must match above.
4. File Model.bim_xxx.settings
   1. Change <ModelUserSettings>.<ServerName> to match your AAS server name.
   2. Change <ModelUserSettings>.<DatabaseName> to simply 'aasCustomerA'. This is the analysis services database name. It does not need to match anything.
5. In Visual Studio, right-click the solution and Add, Existing Project. Select 'aasCustomerA.smproj'.
6. Open Model.bim in the added project. You may have to authenticate to the AAS server.
7. Process the model. This will involve authenticating to the data warehouse database. Save the solution.
8. Over time, there will be more and more AAS projects – one per customer – and hopefully a lot of them.
9. In the Azure Portal, take a look at your AAS server. You will see a new database/model for CustomerA. Congrats!

Creating the Process Analysis Services Model task

• docker run -p 8080:80 –rm -it funcv2

POST https://videobreakdown.azure-api.net/Breakdowns/Api/Partner/Breakdowns ?name=GregFrontFacialWithAudio &privacy=Private &videoUrl=https%3A%2F%2Fgolivearmstorage.blob.core.windows.net%2Fpublic%2FGregFrontWithAudio.MOV

GET https://videobreakdown.azure-api.net/Breakdowns/Api/Partner/Breakdowns/cce0136440/State

As before, you'll need the Ocp-Apim-Subscription-Key header key and value. You'll get back something like this:

These are the things that must be put into place in order to see telemetry from your Windows VM going into an Event Hub:

1. Create and configure an event hub
2. Write the public and private configuration files
3. Install the diagnostics agent along with this configuration into the VM

For #1, please refer to this article: Create an Event Hubs namespace and an event hub using the Azure portal

For #2, let's start with a trivial configuration file. It doesn't do anything but install correctly and demonstrate the structure of the file. Copy the text and save to a local file. Edit a few of the values to match your configuration: myEHNamespace, myEHHubName, mystorageaccountname, the storage account key, the EH shared access policy name and value. [js] { "PublicConfig": { "WadCfg": { "DiagnosticMonitorConfiguration": { "overallQuotaInMB": 4096, "sinks": "HotPath" }, "SinksConfig": { "Sink": [ { "name": "HotPath", "EventHub": { "Url": "https://myEHNamespace.servicebus.windows.net/myEHHubName", "SharedAccessKeyName": "RootManageSharedAccessKey" } } ] } }, "storageAccount": "mystorageaccountname" }, "PrivateConfig": { "storageAccountName": "mystorageaccountname", "storageAccountKey": "NotReallyMyStorageAccountKey==", "storageAccountEndPoint": "https://core.windows.net/", "EventHub": { "Url": "https://myEHNamespace.servicebus.windows.net/myEHHubName", "SharedAccessKeyName": "RootManageSharedAccessKey", "SharedAccessKey": "Jf9JhNotReallyMyKeyCC/ZCSl6h5pq4lnBDtsUoXYZ=" } } } [/js]

For #3, let's start with PowerShell as the installation tool. The file format shown above is suitable for use with Set-AzureRmVmDiagnosticsExtension, one of the Azure PowerShell cmdlets. To get Azure PowerShell, go here. Find PowerShell near the bottom of the page in the Command-line Tools section.

Use this command line to run the cmdlet:

When this runs you should see feedback similar to this:

Last thing, let's add some configuration that will cause some output to the event hub. Insert these lines after line 6 and re-run the command line above:

If you now go to your event hub blade you should see the graph start to register incoming messages. I'll add to this blog shortly with information on how to do the same thing with the Azure CLI.

Cheers!

 There are several scenarios that must be addressed when thinking about getting telemetry from Azure resources to Splunk. What if Splunk is on premises? What if you’re using Splunk Cloud? You could be using a private network connection to Azure, or not. The Splunk add-on approach isn’t suited to all of these. In this blog and others in this series, I’ll introduce some new architectural elements and go into detail on each.

 In this article, “the add-on” refers to this.

 These specific scenarios will be dealt with:

• Cloud-based Splunk, using the add-on (Part 2) 
• Cloud-based Splunk, using the HTTP Event Collector (Part 3) 
• Premises-based Splunk, using private network (Part 4) 
• Premises-based Splunk, via the internet (this article)

 In this, Part 5, I’ll go into Premises-based Splunk via the internet.

In this configuration, Splunk is on prem, behind a firewall and no private network, but no proxy server so the Splunk server can see to resolve names over the internet through port 443.

The solution is another Azure service: Azure Relay. With Azure Relay, the Splunk server takes the “listener” role and establishes an “open phone line” in the cloud via a call over port 443. The “sender” role is played by an Azure Function which can see that open phone line and establish a channel for communications over it.

The components that you need to get going with this are:

• Azure Function for Splunk, located here.
• A Splunk add-on that knows how to work with Azure Relay. That’s here.

In this case, the Azure Function should be configured to run with the Relay output binding. The installation instructions in the README.md covers those details.

 There are several scenarios that must be addressed when thinking about getting telemetry from Azure resources to Splunk. What if Splunk is on premises? What if you’re using Splunk Cloud? You could be using a private network connection to Azure, or not. The Splunk add-on approach isn’t suited to all of these. In this blog and others in this series, I’ll introduce some new architectural elements and go into detail on each.

 In this article, “the add-on” refers to this.

 These specific scenarios will be dealt with:

• Cloud-based Splunk, using the add-on (Part 2) 
• Cloud-based Splunk, using the HTTP Event Collector (Part 3) 
• Premises-based Splunk, using private network (this article) 
• Premises-based Splunk, via the internet (Part 5)

 In this, Part 4, I’ll go into Premises-based Splunk using a private network. There are a couple of variations.

This is the only configuration that supports the use of an on prem proxy server.   

In the first variation, the Splunk Forwarder VM is just Splunk Enterprise with some features switched off. It receives and indexes the incoming events then passes them along to the centralized indexer (VM or cluster). With ExpressRoute (or a site-to-site VPN) configured to encapsulate the VNET where the Splunk VM resides, there is a clear channel to the Splunk Enterprise box on prem. So this configuration is really exactly the same as the one in Part 2 of this series, with the added Splunk instance on prem.

The Express Route isn’t an absolute requirement in this variation since the Splunk Forwarder VM could be a single instance with a static IP. The proxy server could be programmed to allow the Splunk server on prem to talk to it directly.

In the second variation, the on prem Splunk server can see the Azure Function via the private network. In this case, the Azure Function must exist in the context of an App Service Environment in order to have control over its network configuration.

 There are several scenarios that must be addressed when thinking about getting telemetry from Azure resources to Splunk. What if Splunk is on premises? What if you’re using Splunk Cloud? You could be using a private network connection to Azure, or not. The Splunk add-on approach isn’t suited to all of these. In this blog and others in this series, I’ll introduce some new architectural elements and go into detail on each.

 In this article, “the add-on” refers to this.

 These specific scenarios will be dealt with:

• Cloud-based Splunk, using the add-on (Part 2) 
• Cloud-based Splunk, using the HTTP Event Collector (this article) 
• Premises-based Splunk, using private network (Part 4) 
• Premises-based Splunk, via the internet (Part 5)

 In this, Part 3, I’ll go into Cloud-based Splunk using the HTTP Event Collector (HEC).

Splunk HEC is a “fast and efficient way to send data to Splunk Enterprise and Splunk Cloud.” Here is an Introduction to Splunk HTTP Event Collector. For the purposes of this article, think of HEC as an authenticated Splunk endpoint to which we can send events for indexing.

There are a couple of reasons that I know of that you might want to use HEC:

• You use Splunk Cloud. The add-on won’t install there.
• You don’t want to install the add-on into your Splunk Enterprise system for whatever reason.

To satisfy these requirements we need another mechanism to get the data from event hubs and send it along to Splunk. An Azure Function is a great solution for that. The function that has been built for this purpose is here. Use it with the HEC output binding.

Azure Monitor metrics are available via event hub, the same as diagnostic and activity logs. If the volume of messages increases (or decreases), the Azure Function App will autoscale to meet demand.

While I said initially “Cloud-based Splunk”, that’s just the typical scenario. In reality, as long as the HEC endpoint is reachable by the Azure Function, it’s a perfectly good approach. For example, you have an ExpressRoute connection from Azure to your premises and your Function App leverages an App Service Environment, this connectivity becomes an option. See this article for some of the details.

There are several scenarios that must be addressed when thinking about getting telemetry from Azure resources to Splunk. What if Splunk is on premises? What if you’re using Splunk Cloud? You could be using a private network connection to Azure, or not. The Splunk add-on approach isn’t suited to all of these. In this blog and others in this series, I’ll introduce some new architectural elements and go into detail on each.  

In this article, “the add-on” refers to this. 

These specific scenarios will be dealt with: 

• Cloud-based Splunk, using the add-on (this article)
• Cloud-based Splunk, using the HTTP Event Collector (Part 3) 
• Premises-based Splunk, using private network (Part 4) 
• Premises-based Splunk, via the internet (Part 5)

 In this, Part 2, I’ll go into the already familiar Cloud-based Splunk using the add-on.

In this scenario, all resources reside in Azure. The Splunk VM is Splunk Enterprise and may be a cluster rather than a single box. Setting up the VM’s firewall and the network’s Network Security Group to allow outbound traffic on 443 is sufficient.  

The simplest implementation has all monitored resources and monitoring resources in the same subscription and region, but this is seldom the case. Much more typical is many subscriptions with assets in multiple regions. While there is normally just one Splunk indexer (whether single VM or cluster), you may ingest all telemetry directly into that instance from multiple regional Event Hubs and REST APIs. Alternatively, you may use a Splunk Heavy Forwarder in the remote regions, each with the add-on loaded and configured to ingest that region’s data. Each of the forwarders send the aggregated data along to the main index. In either case, the data never leaves Microsoft’s network. 

The decision to use a Heavy Forwarder in the regions vs sending raw data to the indexer isn’t trivial. Here’s an article by a Splunk consultant on the topic. 

Another consideration is the Azure AD tenant. For various reasons, it’s important that the resources being monitored and the monitoring resources (event hubs, service principal, etc) all be in the same AAD tenant.