atverma

Deploy SonarQube to Azure Kubernetes Service cluster and integrate with Azure DevOps build pipeline

Atul.Verma — Mon, 26 Nov 2018 06:29:29 GMT

This is the next part of the series on developing and deploying

Angular, ASP.NET Core Web API and SQL Server to Azure Kubernetes Service
Function Apps using Azure Functions 2.0 runtime

In this article, I am going to share steps needed to deploy SonarQube to Azure Kubernetes Service cluster and integrate with Azure DevOps pipeline to setup code analysis for Angular and ASP.NET Core web apps created in previous parts of this series. The previous articles of this series are https://blogs.msdn.microsoft.com/atverma/2018/09/16/azure-kubernetes-service-aks-deploying-angular-asp-net-core-and-sql-server-on-linux/ https://blogs.msdn.microsoft.com/atverma/2018/10/19/asp-net-core-2-1-web-api-load-app-configuration-from-appsettings-json-dockerfile-environment-variables-azure-key-vault-secrets-and-kubernetes-configmaps-secrets/ https://blogs.msdn.microsoft.com/atverma/2018/09/26/azure-functions-2-0-create-debug-and-deploy-to-azure-kubernetes-service-aks/ https://blogs.msdn.microsoft.com/atverma/2018/10/29/azure-functions-2-0-create-function-app-from-docker-image-functions-triggered-by-cosmos-db-blob-storage-event-hub-and-signalr-service-bindings https://blogs.msdn.microsoft.com/atverma/2018/11/02/add-real-time-web-functionality-to-angular-application-using-asp-net-core-signalr-azure-signalr-service-and-azure-signalr-service-bindings-for-azure-functions-2-0/ The tools used to develop these components are Visual Studio for Mac/VS Code/VS 2017, AKS Dashboard, Docker for Desktop and kubectl.

SonarQube

SonarQube provides the capability to not only show health of an application but also to highlight issues newly introduced. I am going to configure SQL Server as backend database for SonarQube.

Create a SQL Server Database

Create a SQL Server database named 'SonarDB' in the existing SQL server instance I had created during the first part of this series. SonarQube requires that database collation must be case-sensitive and accent-sensitive. Either you can specify collation while creating database or update it e.g code snippets to create/change collation to SQL_Latin1_General_CP1_CS_AS are

CREATE DATABASE SonarDB
COLLATE SQL_Latin1_General_CP1_CS_AS;

ALTER DATABASE SonarDB
COLLATE SQL_Latin1_General_CP1_CS_AS;

Create a SQL Login account which SonarQube needs to connect to 'SonarDB' database.

Create a Persistent Volume

Persistent volume claim is needed to store SonarQube data. The yaml snippet to create a 5 GB storage is displayed below. The deployment resource is going to mount files to this storage claim. You can read more about Persistent Volumes.

 apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sonar-data-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
   requests:
    storage: 5Gi

Create a Kubernetes Service

The next step is to create a Kubernetes Service for SonarQube. As you can see in yaml snippet below, port 80/9000 is defined and type is LoadBalancer i.e. this service will have external endpoints. Kubernetes will use to selector 'app: sonar-deployment' to map to the deployment as you are going to see next. You can read more about Services.

 apiVersion: v1
kind: Service
metadata:
  name: sonar-svc
  labels:
    name: sonar-svc
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 9000
  selector:
    app: sonar-deployment

Create a Kubernetes Deployment

The next step is to create a Kubernetes Deployment for SonarQube. The yaml snippet is displayed below and a few pointers are

The docker image being used to create this resource is image: sonarqube
You can change the number of pods by updating replicas: 1
Label app: sonar-deployment has to match the selector defined in the service
Persistent volume claim created above is used for storage i.e. persistentVolumeClaim: claimName:sonar-data-claim.
JDBC username, password and URL are specified which need to be updated based on values specific to your 'SonarDB' database and login user.

 apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sonar-deployment
  labels:
    app: sonar-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sonar-deployment
    spec:
      terminationGracePeriodSeconds: 15
      initContainers:
      - name: fix-the-volume-permission
        image: busybox
        command:
        - sh
        - -c
        - chown -R 1000:1000 /opt/sonarqube/extensions
        securityContext:
          privileged: true
        volumeMounts:
        - name: sonarqube-data
          mountPath: /opt/sonarqube/extensions
      - name: fix-the-volume-permission1
        image: busybox
        command:
        - sh
        - -c
        - chown -R 1000:1000 /opt/sonarqube/data
        securityContext:
          privileged: true
        volumeMounts:
        - name: sonarqube-data
          mountPath: /opt/sonarqube/data
      containers:
      - name: sonarqube
        image: sonarqube
        ports:
        - name: sonarqube
          containerPort: 9000
        env:
        - name: SONARQUBE_JDBC_USERNAME
          value: sonar
        - name: SONARQUBE_JDBC_PASSWORD
          value: SONAR_SQL_LOGIN_PASSWORD
        - name: SONARQUBE_JDBC_URL 
          value: jdbc:sqlserver://mssql-deployment:1433;databaseName=sonardb;user=sonar;password=SONAR_SQL_LOGIN_PASSWORD
        volumeMounts:
         - name: sonarqube-data
           mountPath: /opt/sonarqube/extensions
           subPath: extensions
         - name: sonarqube-data
           mountPath: /opt/sonarqube/data
           subPath: data
      volumes:
      - name: sonarqube-data
        persistentVolumeClaim:
           claimName: sonar-data-claim

After SonarQube resources are deployed to AKS cluster, navigate to the public endpoint and login using default credentials i.e. username and password is 'admin'. You can change the password. For this sample built in users are provisioned for authentication however you can configure external identity providers. You can read more about external identity providers and http headers based authentication.

Enable SonarQube plugin in Azure DevOps

You need to enable SonarQube extension in Azure DevOps. Browse for SonarQube extension in marketplace and enable it.

Generate Token in SonarQube

Azure DevOps will need token to connect to SonarQube instance. You need to generate token in SonarQube portal by navigating to SonarQube > Security > Generate Tokens. Keep a note of the token.

Create Projects in SonarQube

The next step is to create projects for Angular App and ASP.NET Core Web API in SonarQube portal. The project keys will be needed in Azure DevOps build pipeline thus keep a note of these values.

Create a new Service Connection in Azure DevOps

In order to connect Azure DevOps to SonarQube, you will need to create a new service connection in Azure DevOps. The main pointers are

Specify a connection name
Specify the Server Url as endpoint address of SonarQube
Specify Token value which is generated in previous step

Setting up Azure DevOps build pipeline for Angular App

The three steps needed for SonarQube integration barring npm install and npm build are

Prepare Analysis Configuration
- This step needs to be run before build
- Select SonarQube server endpoint
- Specify Project Key and Project Name for Angular App
- Specify Sources directory root i.e. path to root directory containing source files
Run Code Analysis
- This step needs to be run after build
Publish Quality Gate Result

Setting up Azure DevOps build pipeline for ASP.NET Core Web API App

The three steps needed for SonarQube integration barring Restore and Build are

Prepare Analysis Configuration
- This step needs to be run before build
- Select SonarQube server endpoint
- Specify Project Key for ASP.NET Core Web API App
- Select integrate with MSBuild option
Run Code Analysis
- This step needs to be run after build
Publish Quality Gate Result

Queue Angular and ASP.NET Core Web API Builds

Queue Angular and ASP.NET Core Web API builds on you can see detailed analysis in SonarQube portal. This completes this article on deploying SonarQube to Azure Kubernetes Service cluster and integration with Azure DevOps build pipeline. The Kubernetes resources needed to deploy SonarQube to AKS cluster can be downloaded from GitHub.

Add real-time web functionality to Angular application using ASP.NET Core SignalR, Azure SignalR service and Azure SignalR Service bindings for Azure Functions 2.0

Atul.Verma — Fri, 02 Nov 2018 21:51:08 GMT

This is the next part of the series on developing and deploying

Angular, ASP.NET Core Web API and SQL Server to Azure Kubernetes Service
Function Apps using Azure Functions 2.0 runtime

In this article I am going to go through steps needed to add real-time web functionality to Angular App using ASP.NET Core SignalR and Azure SignalR Service bindings for Azure Functions 2.0. The specific topics which this article is going to cover are

Add ASP.NET Core SignalR to ASP.NET Core 2.1 Web API
- ASP.NET Core SignalR
- ASP.NET Core SignalR scale out using
  - Azure SignalR Service backplane
  - Redis Cache backplane
Publish/Subscribe messages to SignalR Hub from Angular App
Publish/Subscribe messages to SignalR Hub using Azure SignalR Service bindings for Azure Functions 2.0 from Angular App
Build Docker images and deploy to Azure Kubernetes Service

The previous articles of this series are https://blogs.msdn.microsoft.com/atverma/2018/09/16/azure-kubernetes-service-aks-deploying-angular-asp-net-core-and-sql-server-on-linux/ https://blogs.msdn.microsoft.com/atverma/2018/10/19/asp-net-core-2-1-web-api-load-app-configuration-from-appsettings-json-dockerfile-environment-variables-azure-key-vault-secrets-and-kubernetes-configmaps-secrets/ https://blogs.msdn.microsoft.com/atverma/2018/09/26/azure-functions-2-0-create-debug-and-deploy-to-azure-kubernetes-service-aks/ https://blogs.msdn.microsoft.com/atverma/2018/10/29/azure-functions-2-0-create-function-app-from-docker-image-functions-triggered-by-cosmos-db-blob-storage-event-hub-and-signalr-service-bindings The tools used to develop these components are Visual Studio for Mac/VS Code/VS 2017, AKS Dashboard, Docker for Desktop and kubectl.

Add ASP.NET Core SignalR to ASP.NET Core 2.1 Web API

ASP.NET Core SignalR is an open-source library that simplifies adding real-time web functionality to apps. Real-time web functionality enables server-side code to push content to clients instantly. The next sections are going to go through changes needed to add SignalR to ASP.NET Core 2.1 Web API along with SignalR scale out using Azure SignalR Service and Redis backplanes.

ASP.NET Core SignalR

The main pointers about component diagram displayed below are

On navigating to Angular App (Browser), user will be authenticated and Access Token will be retrieved from Azure AD using MSAL
Angular App will pass the bearer token (JWT) in requests and ASP.NET Core 2.1 Web API will validate the Access Token
Angular App will establish connection to ASP.NET Core SignalR Hub (Web API) and publish/subscribe Messages
Angular App will establish connection to Azure SignalR Service using Function App endpoint and publish/subscribe Messages
Angular App, ASP.NET Core Web API and SQL Server are hosted in Azure Kubernetes Service. I have color coded the Pods assuming two Node configuration.

The steps needed to enable ASP.NET Core SignalR in ASP.NET Core 2.1 Web API are

Configure SignalR Hub: The SignalR Hubs API enables you to call methods on connected clients from the server. In the server code, you define methods that are called by client. In the client code, you define methods that are called from the server. SignalR takes care of everything behind the scenes that makes real-time client-to-server and server-to-client communications possible. Add INotificationHub interface andNotificationHub class which are needed to define methods that are called by client.

     public interface INotificationHub
    {
        Task MessageNotification(string message);
        Task PublishMessageAck(string value);
    }

     public class NotificationHub : Hub<INotificationHub>
    {
        public async Task PublishMessage(string message)
        {
            await this.Clients.AllExcept(this.Context.ConnectionId).MessageNotification($"Broadcast: {message}");
            await this.Clients.Caller.PublishMessageAck($"Broadcast Ack: {message}");
        }
        public override async Task OnConnectedAsync()
        {
            await Groups.AddToGroupAsync(Context.ConnectionId, "Users");
            await base.OnConnectedAsync();
        }
        public override async Task OnDisconnectedAsync(Exception exception)
        {
            await Groups.RemoveFromGroupAsync(Context.ConnectionId, "Users");
            await base.OnDisconnectedAsync(exception);
        }
    }

Specify SignalRConfigMode SingalRConfigMode = SignalRConfigMode.ASPNetCoreSignalR in Startup.cs class
Update ConfigureServices method and add SignalR service.

            services.AddSignalR(options =>
           {
                options.EnableDetailedErrors = true;
           });

- Update Configure method and add SignalR to IApplicationBuilder request execution pipeline. The SignalRHub variable value I have specified in sample is /api/SignalRHub.

          app.UseSignalR((options) =>
          {
              options.MapHub<NotificationHub>(SignalRHub);
          });

Allow the JWT authentication handler to read the access token from the query string when a WebSocket or server sent events request comes in Service.AddAuthentication(...).AddJwtBearer(...) method.

          OnMessageReceived = context =>
         {
              var accessToken = context.Request.Query["access_token"];
              var path = context.HttpContext.Request.Path;
              if (!string.IsNullOrEmpty(accessToken) && (path.StartsWithSegments(SignalRHub)))
              {
                  context.Token = accessToken;
              }
              return Task.CompletedTask;
          },

Claims of the authorized user can be accessed by calling Context.User.Claims in NotificationHub class.

In a single node/server scenario this works fine however when ASP.NET Core 2.1 Web API is scaled out e.g. when running more than one Pods, clients connected to SignalR Hub on one Pod won't receive message published on SignalR Hub on another Pod. ASP.NET Core SignalR supports Redis Cache and Azure SignalR Service scale out.

ASP.NET Core SignalR scale out using Redis Cache backplane

Redis cache can be used as backplane for ASP.NET Core SignalR scale out. Redis cache is used as Pub/Sub to forward messages to other servers. This option needs sticky sessions to be enabled. The only change in component diagram from previous implementation is that Web API Pods are communicating with Redis Cache service. The code changes needed to use Redis backplane are

Install Microsoft.AspNetCore.SignalR.Redis Nuget package
Specify SignalRConfigMode SingalRConfigMode = SignalRConfigMode.Redis in Startup.cs class
Update ConfigureServices method and add AddRedis

             services.AddSignalR(options =>
            {
               options.EnableDetailedErrors = true;
            }).AddRedis(RedisConnectionString);

Specify value of "RedisConnectionString": "REDIS_CONNECTION_STRING" in appsettings.json

ASP.NET Core SignalR scale out using Azure SignalR Service backplane

Azure SignalR Service can be used as backplane for ASP.NET Core SignalR scale out. This option doesn't need sticky sessions to be enabled.One of the key reasons to use the Azure SignalR Service is simplicity. With Azure SignalR Service, you don't need to handle problems like performance, scalability, availability. The only change in component diagram from previous implementation is Web API Pods are communicating with Azure SignalR Service. The code changes needed to use Azure SignalR service are

Install Microsoft.Azure.SignalR Nuget package
Specify SignalRConfigMode SingalRConfigMode = SignalRConfigMode.AzureSignalRService in Startup.cs class
Update ConfigureServices method and add AddAzureSignalR

            services.AddSignalR(options =>
           {
                options.EnableDetailedErrors = true;
           }).AddAzureSignalR(configure =>
           {
                configure.ConnectionString = AzureSignalRConnectionString;
           });

Update Configure method and add UseAzureSignalR

          app.UseAzureSignalR((options) =>
          {
              options.MapHub<NotificationHub>(SignalRHub);
          })

Specify value of "AzureSignalRConnectionString": "AZURE_SIGNALR_CONNECTION_STRING" in appsettings.json

Publish/Subscribe messages to SignalR Hub from Angular App

Angular App sample is updated to publish/subscribe to SignalR Hub. The main pointers about code snippet listed below are

Install aspnet/signalr which provides support for JavaScript and TypeScript clients for SignalR for ASP.NET Core
SignalR Hub URL is specified in ${this.config.get().API_URL}${this.config.get().SIGNALR_HUB}
Access Token is needed to authorize requests to SignalR Hub and is specified in { accessTokenFactory: () => this.authHelper.getAccessTokenFromCache() }
Retry connection interval is 10000
App subscribes to MessageNotification and PublishMessageAck messages of SignalR Hub
App sends messages to SignalR Hub using PublishMessage
Specify 'API_URL': 'WEB_API_URL' in 'environment.ts'

 import { EventEmitter, Injectable, OnDestroy, OnInit } from '@angular/core';
import { HubConnection, HubConnectionBuilder } from '@aspnet/signalr';
import * as signalR from '@aspnet/signalr';
import { AuthHelperService, AccessTokenInfo } from './auth-helper.service';
import { Config } from './config/config';
@Injectable({
  providedIn: 'root',
})
export class SignalRService {
  messageReceived = new EventEmitter<string>();
  hubConnection: HubConnection;
  constructor(private authHelper: AuthHelperService, private config: Config) {
  }
  init() {
    this.createConnection();
    this.startConnection();
  }
  private createConnection() {
    this.hubConnection = new HubConnectionBuilder()
      .withUrl(`${this.config.get().API_URL}${this.config.get().SIGNALR_HUB}`,
        { accessTokenFactory: () => this.authHelper.getAccessTokenFromCache() })
      .configureLogging(signalR.LogLevel.Information)
      .build();
    this.hubConnection.onclose(err => {
      console.log('SignalR hub connection closed.');
      this.stopHubAndunSubscribeToServerEvents();
      this.restartConnection(err);
    });
  }
  private restartConnection(err: Error): void {
    console.log(`Error ${err}`);
    console.log('Retrying connection to SignalR Hub ...');
    setTimeout(() => {
      this.startConnection();
    }, 10000);
  }
  private startConnection(): void {
    this.hubConnection
      .start()
      .then(() => {
        console.log('SignalR Hub connection started');
        this.subscribeToServerEvents();
      })
      .catch(err => {
        this.restartConnection(err);
      });
  }
  public publishMessage(message: string) {
    this.hubConnection.invoke('PublishMessage', message);
  }
  private subscribeToServerEvents(): void {
    this.hubConnection.on('MessageNotification', (data: any) => {
      this.messageReceived.emit('MessageNotification:' + data);
    });
    this.hubConnection.on('PublishMessageAck', (data: any) => {
      this.messageReceived.emit('MessageNotification - Ack :' + data);
    });
  }
  private stopHubAndunSubscribeToServerEvents(): void {
    this.hubConnection.off('MessageNotification');
    this.hubConnection.off('PublishMessageAck');
    this.hubConnection.stop().then(() => console.log('Hub connection stopped'));
  }
}

Publish/Subscribe messages to SignalR Hub using Azure SignalR Service bindings for Azure Functions 2.0 from Angular App

The previous blog of this series describes steps needed to develop a Function App with Azure SignalR Service bindings for Azure Functions 2.0. Angular App sample is updated to publish/subscribe to Azure SignalR Service Hub through Azure Functions (you don't need to host a web application at all). The main pointers about code snippet listed below are

Install aspnet/signalr which provides support for JavaScript and TypeScript clients for SignalR for ASP.NET Core
${this.config.get().FUNCTION_APP_URL}negotiate post request will return SignalRConnectionInfo object having url and accessToken.
SignalRConnectionInfo is needed to connect to Azure SignalR Service Hub by specifying accessTokenFactory: () => info.accessToken and url in withUrl(url, options)
Retry connection interval is 10000
App sends messages to SignalR Hub by sending a post request to ${this.config.get().FUNCTION_APP_URL}sendmessage
App subscribes to SendMessage i.e. this.hubConnection.on('sendMessage', (data: any) of SignalR Hub
Specify 'FUNCTION_APP_URL': 'FUNCTION_APP_URLFUNCTION_APP_URL' in environment.ts

 import { Injectable, EventEmitter, OnDestroy } from '@angular/core';
import { HttpClient, HttpErrorResponse } from '@angular/common/http';
import { HubConnection, IHttpConnectionOptions } from '@aspnet/signalr';
import * as signalR from '@aspnet/signalr';
import { Observable, throwError } from 'rxjs';
import { SignalRConnectionInfo } from './signalRConnectionInfo';
import { Config } from './config/config';
@Injectable({
    providedIn: 'root'
})
export class SignalRFuncService {
    hubConnection: HubConnection;
    messageReceived = new EventEmitter<string>();
    constructor(private http: HttpClient, private config: Config) {
    }
    private getConnectionInfo(): Observable<SignalRConnectionInfo> {
        const requestUrl = `${this.config.get().FUNCTION_APP_URL}negotiate`;
        return this.http.post<SignalRConnectionInfo>(requestUrl, null);
    }
    init() {
        this.getConnectionInfo().subscribe((info: SignalRConnectionInfo) => {
            const options = {
                accessTokenFactory: () => info.accessToken
            };
            this.createConnection(info.url, options);
            this.startConnection();
        },
        error => {
            console.error(`An error occurred during init: ${error}`);
            console.log('Retrying connection to Azure Function - SignalR Hub ...');
            setTimeout(() => {
                this.init();
            }, 10000);
        });
    }    private createConnection(url: string, options: IHttpConnectionOptions) {
        this.hubConnection = new signalR.HubConnectionBuilder()
            .withUrl(url, options)
            .configureLogging(signalR.LogLevel.Information)
            .build();
        this.hubConnection.onclose(err => {
            console.log('Azure Function - SignalR Hub connection closed.');
            this.stopHubAndunSubscribeToServerEvents();
            this.restartConnection(err);
        });
    }
    private startConnection(): void {
        this.hubConnection
            .start()
            .then(() => {
                console.log('Azure Function - SignalR Hub connection started.');
                this.subscribeToServerEvents();
            })
            .catch(err => {
                this.restartConnection(err);
            });
    }
    private restartConnection(err: Error): void {
        console.log(`Error ${err}`);
        console.log('Retrying connection to Azure Function - SignalR Hub ...');
        setTimeout(() => {
            this.startConnection();
        }, 10000);
    }
    send(message: string) {
        const requestUrl = `${this.config.get().FUNCTION_APP_URL}sendmessage`;
        this.http.post(requestUrl, message).subscribe(
            (data: any) => console.log(`Func Hub sendMessage: ${message}`),
            error => console.error(`An error occurred in sendMessage: ${error}`)
        );
    }
    private subscribeToServerEvents(): void {
        this.hubConnection.on('sendMessage', (data: any) => {
            this.messageReceived.emit('MessageNotification - Function: ' + data);
        });
    }
    private stopHubAndunSubscribeToServerEvents(): void {
        this.hubConnection.off('sendMessage');
        this.hubConnection.stop().then(() => console.log('Hub connection stopped'));
    }
}

 export class SignalRConnectionInfo {
    url: string;
    accessToken: string;
}

Build Docker Images and Deploy to Azure Kubernetes Service

Build docker images for ASP.NET Core Web API and Angular App and deploy these docker images to Azure Kubernetes Service cluster. After deployment, browse to Angular App and send message either using ASP.NET Core SignalR or Azure Functions 2.0 endpoint.

Summary

This completes the article on add real-time web functionality to Angular App using ASP.NET Core SignalR and Azure SignalR Service bindings for Azure Functions 2.0. The complete source code can be downloaded from GitHub-AzureFunctions and GitHub-AKS.

Azure Functions 2.0: Create Function App from Docker Image (Functions triggered by Cosmos DB, Blob storage, Event Hub and SignalR service bindings)

Atul.Verma — Mon, 29 Oct 2018 23:16:22 GMT

This is second part of the series on developing and deploying Azure Functions 2.0 where I will

Create a function triggered by Azure Cosmos DB
Create Azure SignalR Service bindings for Azure Functions 2.0
Publish Docker Image to Docker Hub
Create Function App from Docker Image in Azure Portal
Deploy functions to Azure Kubernetes Service from VS Code

The first part of the series provides details on creating functions triggered by Azure Blob storage and Event hub in Visual Studio Code along with deploying Azure Functions to Azure Kubernetes Service (AKS). https://blogs.msdn.microsoft.com/atverma/2018/09/26/azure-functions-2-0-create-debug-and-deploy-to-azure-kubernetes-service-aks/ Dev tools used to develop these components are Visual Studio Code for macOS and Docker. The complete source code for this article can be downloaded from GitHub.

Create a function triggered by Azure Cosmos DB

The next steps will explain how to create a function triggered by data insert/change in Cosmos DB.

Create Azure Cosmos DB

I have specified following values for function

Name of database with the collection to be monitored as 'SampleDB'
Name of collection to be monitored as 'SampleCollection'
Name of collection used to store leases as 'leases'

Keep a note of the connection string as this needs to be specified in function.

Create Azure Cosmos DB trigger

Run command func new to create a new function and select CosmosDBTrigger option and specify function name. I have specified databaseName: "SampleDB", collectionName: "SampleCollection" and LeaseCollectionName = "leases" based on values I had specified in previous step.

[FunctionName("TestCosmosDBTrigger")]
public static void Run([CosmosDBTrigger(
    databaseName: "SampleDB",
    collectionName: "SampleCollection",
    ConnectionStringSetting = "TestCosmonDBConnection",
    LeaseCollectionName = "leases")]IReadOnlyList<Document> input, ILogger log)
{
    if (input != null && input.Count > 0)
    {
        log.LogInformation("Documents modified " + input.Count);
        log.LogInformation("First document Id " + input[0].Id);
    }
}

Update 'local.settings.json' and specify value for "TestCosmonDBConnection":"{COSMOS_DB_CONNECTION_STRING}" as this is needed to run host locally. You can start host by running command func start --build --debug VSCode --cors "*" On mojave macOS, VS Code was not loading this function and the error message was 'Error reading Darwin Kernel Version. System.Private.CoreLib: The given key 'osx.10.14-x64' was not present in the dictionary'. Adding "DOTNET_RUNTIME_ID" : "osx-x64" in 'local.settings.json' fixed that error for me.

Create Azure SignalR Service bindings for Azure Functions

The next steps will explain how to authenticate and send real-time messages to clients connected to Azure SignalR Service by using SignalR Service bindings in Azure Functions. The SignalR Service bindings are provided in the Microsoft.Azure.WebJobs.Extensions.SignalRService NuGet package. The command to install package in VS Code is func extensions install -p Microsoft.Azure.WebJobs.Extensions.SignalRService -v 1.0.0-preview1-10002

Create Azure SignalR Service

Specify the resource name and resource group and keep a note of the connection string since this is needed in HTTP triggers which will invoke a function with a HTTP request.

Create HttpTrigger to retrieve the service endpoint URL and valid access token of Azure SignalR Service

Before a client can connect to Azure SignalR Service, it must retrieve the service endpoint URL and a valid access token. The SignalRConnectionInfo input binding produces the SignalR Service endpoint URL and a valid token that are used to connect to the service. I have specified HubName = "testazuresignalrhub" as hub name. The client app is going to send http post request to 'https://{FUNCTION_URL}/api/negotiate'.

 [FunctionName("negotiate")]
public static IActionResult GetSignalRInfo(
    [HttpTrigger(AuthorizationLevel.Anonymous, "post", "options")]HttpRequest req,
    [SignalRConnectionInfo(HubName = "testazuresignalrhub")]SignalRConnectionInfo connectionInfo, ILogger log)
{
    try
    {
        if (!req.HttpContext.Response.Headers.ContainsKey("Access-Control-Allow-Credentials"))
        {
            req.HttpContext.Response.Headers.Add("Access-Control-Allow-Credentials", "true");
        }
        if (req.Headers.ContainsKey("Origin") && !req.HttpContext.Response.Headers.ContainsKey("Access-Control-Allow-Origin"))
        {
            req.HttpContext.Response.Headers.Add("Access-Control-Allow-Origin", req.Headers["Origin"][0]);
        }
        if (req.Headers.ContainsKey("Access-Control-Request-Headers"))
        {
            req.HttpContext.Response.Headers.Add("Access-Control-Allow-Headers", req.Headers["access-control-request-headers"][0]);
        }
        log.LogInformation("negotiate API succeeded.");
        if (connectionInfo == null)
        {
            return new NotFoundObjectResult("Azure SignalR not found.");
        }
        return new OkObjectResult(connectionInfo);
    }
    catch
    {
        return new BadRequestResult();
    }
}

Create HttpTrigger to send message using Azure SignalR Service

The client will use URL and access token from SignalRConnectionInfo retrieved in previous step to establish SignalR connection. It will then use SignalR output binding to send one or more messages using Azure SignalR Service. The client app is going to send http 'post' request to 'https://{FUNCTION_URL}/api/sendmessage' and provide string payload. This API sends message to all clients who have subscribed to sendMessage.

 [FunctionName("sendmessage")]
public static async Task<IActionResult> SendMessage(
    [HttpTrigger(AuthorizationLevel.Anonymous, "post", "options")]HttpRequest req,
    [SignalR(HubName = "testazuresignalrhub")]IAsyncCollector<SignalRMessage> signalRMessages, ILogger log)
{
    try
    {
        if (!req.HttpContext.Response.Headers.ContainsKey("Access-Control-Allow-Credentials"))
        {
            req.HttpContext.Response.Headers.Add("Access-Control-Allow-Credentials", "true");
        }
        if (req.Headers.ContainsKey("Origin") && !req.HttpContext.Response.Headers.ContainsKey("Access-Control-Allow-Origin"))
        {
            req.HttpContext.Response.Headers.Add("Access-Control-Allow-Origin", req.Headers["Origin"][0]);
        }
        if (req.Headers.ContainsKey("Access-Control-Request-Headers"))
        {
            req.HttpContext.Response.Headers.Add("Access-Control-Allow-Headers", req.Headers["access-control-request-headers"][0]);
        }
        // Commenting since not passing json payload. If passing json payload uncomment this line
        // var message = new JsonSerializer().Deserialize(new JsonTextReader(new StreamReader(req.Body)));
        var message = new StreamReader(req.Body).ReadToEnd();
        await signalRMessages.AddAsync(
            new SignalRMessage
            {
                Target = "sendMessage",
                Arguments = new[] { message }
            });
        log.LogInformation("sendMessage API succeeded.");
        return new OkResult();
    }
    catch
    {
        return new BadRequestResult();
    }
}

Now that Azure SignalR Service bindings are defined, update 'local.settings.json' and specify value for "AzureSignalRConnectionString":"{AZURE_SIGNALR_CONNECTION_STRING}" as this is needed to run host locally. You can start host by running command func start --build --debug VSCode --cors "*".

Publish Docker Image to DockerHub

The next step is to create a Docker Hub repository. Update 'Dockerfile' environment variables based on your connection strings

ENV AzureWebJobsStorage="{BLOB_STORAGE_CONNECTION_STRING}"
ENV AzureBlobStorage="{BLOB_STORAGE_CONNECTION_STRING_OF_CONTAINER_HAVING_TRIGGER}"
ENV AzureEventHubConnectionString="{EVENT_HUB_CONNECTION_STRING}"
ENV TestCosmonDBConnection="{COSMOS_DB_CONNECTION_STRING}"
ENV AzureSignalRConnectionString="{AZURE_SIGNALR_CONNECTION_STRING}"

The commands to build Docker Image are listed below

Build Docker Image: docker build -t samplefunc .
Tag Docker Image: docker tag samplefunc {YOUR_DOCKER_REPO}
Publish Image to Docker Hub: docker push {YOUR_DOCKER_REPO}

Create Function App from Docker Image in Azure Portal

Create a new Function App in Azure Portal and specify

App name
Resource group
Select OS as 'Linux (Preview)'
Select Publish as 'Docker Image'

Docker image published to Docker Hub in previous step will be used to configure container. Open 'Configure container' and select 'Docker Hub' tab and specify values for image and option tag along with repository access. After function App is created, you will see list of Functions. Navigate to the Overview tab and keep a note of the URL. The function URL will be used by SignalR client to connect and send/receive events. If you want to connect to this function app from a browser based client, you will need to allow the origin to make cross-origin calls in Platform Features > API > CORS e.g. I have allowed 'https://localhost:4200' and 'https://localhost:4200/home' origins.

Deploy functions to Azure Kubernetes Service from VS Code

You can also deploy Azure functions 2.0 to AKS and command is func deploy --platform kubernetes --name {DockerHubRepositoryName} --registry {DockerHubRegistry}. You need to update placeholders with your Docker hub repository name and account. Once deployment is complete, you can open the endpoint to verify the deployment. You can also open AKS dashboard to view deployed resources under namespace 'azure-functions'.

Summary

In the next article of this series I will create a few client apps (Console/Angular) to integrate with Azure SignalR Service, ASP.NET SignalR and Azure SignalR Service bindings for Azure functions. This completes this article and complete source code can be downloaded from GitHub.

ASP.NET Core 2.1 Web API - Load App Configuration from appsettings.json, Dockerfile environment variables, Azure Key Vault Secrets and Kubernetes ConfigMaps/Secrets

Atul.Verma — Fri, 19 Oct 2018 22:25:06 GMT

This article is second part of the series on Deploying Angular, ASP.NET Core and SQL Server on Linux to Azure Kubernetes Service (AKS) cluster. The first part, describes steps needed to deploy these components to AKS. App configuration in ASP.NET Core is based on key-value pairs established by configuration providers. Configuration providers read configuration data into key-value pairs from a variety of configuration sources. In this article I am going to share multiple ways to load App configuration in ASP.net Core Web API

Hosting Environment specific appsettings.json
Dockerfile Environment Variables
Kubernetes
- Container Environment variables with data from ConfigMap/Secret
- Populate Volume (Config file) with data stored in a ConfigMap/Secret
Azure Key Vault Secrets

The tools used to develop these components are Visual Studio for Mac/VS Code/VS 2017, AKS Dashboard, Docker for Desktop and kubectl. The formatting of code snippets in this article may get distorted (especially yaml), thus please refer to GitHub repository for complete source code for this article. I have extended sample solution of first part of the series by adding new files i.e. ConfigController, Kubernetes_ConfigMap.yaml and Kubernetes_Deployment_V2.yaml

Hosting environment specific appsettings.json

Hosting environment specific versions of appsettings.json is one of the ways to define App configuration in ASP.net core. In ConfigureAppConfiguration method, based on Hosting environment, App Config will be loaded as displayed below

 config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
                        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: false, reloadOnChange: true)

I have defined a few configuration settings in appsettings.json


 {
    "AppConfiguration": {
        "DatabaseConnectionString": "Server=mssql-sample-service,1433;Database=UsersDB;User Id=sa;Password=P@ssword1$;",
        "IsVaultEnabled": false,
        "Vault":  "AZURE_KEY_VAULT_DNS_NAME",
        "ClientId": "APPLICATION_ID",
        "ClientSecret": "APPLICATION_KEY",
        "DatabaseConnectionStringFromAppsettings": "DatabaseConnectionStringFromAppsettingsValue"
    }
}

The Production environment file i.e. appsettings.Production.json also defines DatabaseConnectionStringFromAppsettings setting, thus during Production Environment build, configuration from this file will get applied based on precedence.

{
  "AppConfiguration": {
    "DatabaseConnectionStringFromAppsettings": "Prod:DatabaseConnectionStringFromAppsettingsValue"
  }
}

Dockerfile Environment Variable

Environment variables can be defined in Dockerfile and args can be passed during building Docker image. In ConfigureAppConfiguration method, config.AddEnvironmentVariables() loads environment variables. In DockerFile displayed below, argument ARG dbConnectionString is defined along with environment variable ENV DatabaseConnectionString. This ARG value can then be specified when building Docker image and the command is docker build --build-arg dbConnectionString=DatabaseConnectionStringFromDockerEnvVariable -t testappwebapi .


 FROM microsoft/dotnet:2.1-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80
ARG dbConnectionString
ENV DatabaseConnectionStringFromDockerEnvVariable $dbConnectionString
FROM microsoft/dotnet:2.1-sdk AS build
WORKDIR /src
COPY ["SampleWebApp/SampleWebApp.csproj", "."]
RUN dotnet restore "SampleWebApp.csproj"
COPY . .
RUN dotnet build "SampleWebApp.csproj" -c Release -o /app
FROM build AS publish
RUN dotnet publish "SampleWebApp.csproj" -c Release -o /app
FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "SampleWebApp.dll"]

ConfigMaps and Secrets in Kubernetes

ConfigMaps and Secret allows you to decouple configuration artifacts from image content to keep containerized applications portable. For this article, I will show how to use ConfigMap to define app configuration.

Container Environment variables with data from ConfigMap/Secret

The ConfigMap resource displayed below defines DatabaseConnectionStringFromKubernetesEnvVariable setting.

apiVersion: v1
kind: ConfigMap
metadata:
  name: samplewebapp-configmap-1
  namespace: default
data:
  DatabaseConnectionStringFromKubernetesEnvVariable: DatabaseConnectionStringFromKubernetesEnvVariableValue

Container environment variables can be set from Secret using env[].valueFrom.secretKeyRef and from ConfigMap using env[].valueFrom.configMapKeyRef. Container environment variable defined in deployment resource(Kubernetes_Deployment_V2.yaml) will be set from ConfigMap and the partial yaml snippet is

env:
- name: DatabaseConnectionStringFromKubernetesEnvVariable
  valueFrom:
    configMapKeyRef:
       name: samplewebapp-configmap-1
       key: DatabaseConnectionStringFromKubernetesEnvVariable

Populate Volume (Config file) with data stored in a ConfigMap/Secret

This is similar to previous step however in this case, data stored in ConfigMap will be mounted to a json file. ASP.net core web api will load App Configuration from this json file. The ConfigMap resource displayed below defines data for AppConfig.json:

 apiVersion: v1
kind: ConfigMap
metadata:
  name: samplewebapp-configmap-2
  namespace: default
data:
  AppConfig.json: |-
    {
        "AppConfiguration": {
            " DatabaseConnectionStringFromKubernetesMountedFile": "DatabaseConnectionStringFromKubernetesMountedFileValue"
        }
    }

AppConfig.json file will be mounted from ConfigMap in deployment resource(Kubernetes_Deployment_V2.yaml). The partial yaml snippets where AppConfig.json is mounted to path mountPath: /app/AppConfig.json from samplewebapp-configmap-2 ConfigMap is

 volumeMounts:
- name: samplewebapp-configmap-2
  mountPath: /app/AppConfig.json
  subPath: AppConfig.json

 volumes:
- name: samplewebapp-configmap-2
  configMap:
    name: samplewebapp-configmap-2

In ConfigureAppConfiguration method, App configuration is loaded from AppConfig.json config.AddJsonFile("/app/AppConfig.json", optional: false, reloadOnChange: true

Azure Key Vault Secrets

Azure Key Vault can be used to securely store and tightly control access to tokens, passwords, certificates, API keys, and other secrets. Centralizing storage of application secrets in Azure Key Vault allows you to control their distribution. Key Vault greatly reduces the chances that secrets may be accidentally leaked. I will provide the steps needed to load App Configuration from Azure Key Vault secrets.

Create Azure AD Application

For Service-to-Azure-Service authentication, this approach involves creating an Azure AD application and associated credential, and using that credential to get a token. This approach does has short comings i.e. application credentials need to be specified in code and credentials expiry which Managed Service Identity fixes. Azure Kubernetes Service(AKS) is not currently natively integrated with Azure Key Vault however, the Azure Key Vault FlexVolume for Kubernetes project enables direct integration from Kubernetes pods to KeyVault secrets. Navigate to Azure AD > App Registrations > New Application Registration and select Application Type as Web app/API and specify Name and Sign-on URL After Azure AD Application is created, navigate to the resource and take note of Application ID which is needed to be specified for ClientID in ASP.net Core Web API appsettings.json. Open Settings and create a new Key. Take note of the value (you can only copy after saving) which is needed to be specified for ClientSecret in ASP.net Core Web API appsettings.json.

Create Azure Key Vault

Create a key vault resource using Azure CLI or Portal You need to specify Access policies. Select principal as Application you created in previous step and grant permissions e.g. I have granted Read and List Secret permissions. Create a Secret and specify Name and Value. The value for this secret is going to be loaded in App Configuration. Keep note of the Key Vault's DNS name which is needed to be specified for Vault in ASP.net Core Web API appsettings.json.

ASP.net Core Web API

Add Azure Key Vault Configuration provider nuget package Microsoft.Extensions.Configuration.AzureKeyVault. In ConfigureAppConfiguration method snippet displayed below, App configuration is loaded from Azure Key Vault. As described above, you need to specify values for Vault, ClientId and ClientSecret in appsettings.json. Along with this you need to set IsVaultEnabled to true in appsettings.json.

if (Convert.ToBoolean(configInProgress["AppConfiguration:IsVaultEnabled"]))
{
  config.AddAzureKeyVault(configInProgress["AppConfiguration:Vault"],
                          configInProgress["AppConfiguration:ClientId"],
                          configInProgress["AppConfiguration:ClientSecret"]);
 config.Build();
}

In ConfigureServices method of Startup class, Configuration instance is registered with AppConfiguration class

 services.Configure<AppConfiguration>(Configuration.GetSection("AppConfiguration"));
services.Configure<AppConfiguration>(Configuration);

 public class AppConfiguration
{
    public string DatabaseConnectionString
    {
        get;
        set;
    }
    public bool IsVaultEnabled
    {
        get;
        set;
    }
    public string Vault
    {
        get;
        set;
    }
    public string ClientId
    {
        get;
        set;
    }
    public string ClientSecret
    {
        get;
        set;
    }
    public string DatabaseConnectionStringFromAppsettings
    {
        get;
        set;
    }
    public string DatabaseConnectionStringFromDockerEnvVariable
    {
        get;
        set;
    }
    public string DatabaseConnectionStringFromKubernetesEnvVariable
    {
        get;
        set;
    }
    public string DatabaseConnectionStringFromKubernetesMountedFile
    {
        get;
        set;
    }
    public string DatabaseConnectionStringFromAzureKeyVault
    {
        get;
        set;
    }
}

The code snippet from ConfigController displays App Configuration values from all the sources

 [HttpGet]
public IEnumerable<string> Get()
{
     return new List<string>()
            {
                $"DatabaseConnectionStringFromAppsettings: {_appSettings.Value.DatabaseConnectionStringFromAppsettings}",
                $"DatabaseConnectionStringFromDockerEnvVariable: {_appSettings.Value.DatabaseConnectionStringFromDockerEnvVariable}",
                $"DatabaseConnectionStringFromKubernetesEnvVariable: {_appSettings.Value.DatabaseConnectionStringFromKubernetesEnvVariable}",
                $"DatabaseConnectionStringFromKubernetesMountedFile: {_appSettings.Value.DatabaseConnectionStringFromKubernetesMountedFile}",
                $"DatabaseConnectionStringFromAzureKeyVault: {_appSettings.Value.DatabaseConnectionStringFromAzureKeyVault}"
            };
}

Build the docker image and deploy the resources to Azure Kubernetes Service. Browse to "https://YOUR_HOST/api/Config" to see list of App Configurations with Values populated from these sources. The source code for this article can be downloaded from GitHub repository

Ingest Azure Redis Cache messages into Elasticsearch, Logstash and Kibana cluster deployed in Azure Kubernetes Service (AKS)

Atul.Verma — Sun, 14 Oct 2018 22:43:14 GMT

This is third article on the series on deploying Elasticsearch, Logstash and Kibana (ELK) in Azure Kubernetes Service (AKS) cluster. The first article covered deploying non-SSL ELK to AKS and consuming messages from Azure Event Hub. The second article described how to secure communications in ELK and use Azure AD SAML based SSO for Kibana and Elasticsearch. In this article I am going to share steps needed to ingest Azure Redis Cache messages into Elasticsearch using Logstash's Redis plugin. Azure Redis Cache is based on the popular open-source Redis cache. It is typically used as a cache to improve the performance and scalability of systems that rely heavily on backend data-stores. Logstash's Redis plugin will read events from Redis instance. I will create a Logstash event processing pipeline where I will define Redis as input and Elasticsearch as output. The component diagram has been updated to add Azure Redis Cache integration. The dev tools used to develop these components are Visual Studio for Mac/VS Code, AKS Dashboard, kubectl, bash and openssl. The code snippets in this article are mostly yaml snippets and are included for reference only as formatting may get distorted thus please refer to GitHub repository for formatted resources.

Create Azure Redis Cache

Create Azure Redis Cache using Portal or Azure CLI command az redis create Navigate to this resource and keep note of the settings listed below as you are going to need them in subsequent sections

Host Name: You will need to specify Host name is Logstash pipeline.
Secondary Access Key: You will need to specify password in Logstash pipeline.
StackExchange.Redis connection string: You will need to specify connection string in AzureRedisCacheSample so as to publish messages to Redis.
Port: Default non-SSL port is 6379. By default, non-SSL port is disabled thus the port to which Logstash pipeline will connect is 6380.

Logstash Redis input plugin

As mentioned previously, I will create a Logstash pipeline which will use Logstash Redis input plugin. This input will read events from a Redis instance; it supports both Redis channels and lists. You can read more about Redis input plugin. You can get installed plugins list by following steps listed below

Run command kubectl exec -ti {Logstash_Pod_Name} bash to connect to Logstash POD.
Run command bin/logstash-plugin list to see installed plugins

Deploy Logstash to Azure Kubernetes Service

Logstash is data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch. Logstash will use Azure Event Hub plugin and Redis input plugin to ingest data into Elasticsearch. In this article, I am going to share main pointers about changes needed to Logstash resources i.e. ConfigMap and Deployment in order to subscribe to Azure Redis Cache only. You can refer to first two parts of this series for more details. The steps needed to deploy Logstash to AKS are listed below

Create a Kubernetes ConfigMap

Create a new pipeline in Logstash for Azure Redis cache integration and main pointers about changes needed to this resource as compared to earlier parts of this series are

A new pipeline has been defined i.e. azureredis is the pipeline id and azureredis.cfg is the path of the configuration file for Redis cache integration. azureredis.cfg file will be mounted from ConfigMap. The Logstash event processing pipeline has three stages: inputs → filters → outputs. This file defines the logstash pipeline for Redis Cache.
- Specify host => "{YOUR_REDIS_HOST_NAME}" based on your Redis instance host name
- The sample Redis client is publishing messages to channel, thus data_type is channel data_type => "channel"
- The channel name I have specified is 'messages'. Update channel name key => "messages" based on your specified channel name.
- Specify password as per your Redis Cache secondary key value password => "{YOUR_REDIS_SECONDARY_KEY}"
- Since HTTPS is enabled, I have specified port as 6380 port => 6380. The default setting is non-SSL port i.e. 6379.
- SSL is enabled ssl => true
- Output is Elasticsearch and Index name is defined as index => "azureredis-%{+YYYY.MM.dd}

The yaml snippet to create this resource is displayed below

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-logstash-configmap
  namespace: default
data:
  logstash.yml: |
    xpack.monitoring.elasticsearch.url: https://sample-elasticsearch:9200
    dead_letter_queue.enable: true
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: logstash_system 
    xpack.monitoring.elasticsearch.password: Password1$
    xpack.monitoring.elasticsearch.ssl.ca: "/usr/share/logstash/config/elastic-stack-ca.pem"
  pipelines.yml: |
    - pipeline.id: azureeventhubs
      path.config: "/usr/share/logstash/azureeventhubs.cfg"
    - pipeline.id: azureredis
      path.config: "/usr/share/logstash/azureredis.cfg"
  azureeventhubs.cfg: |
    input {
      azure_event_hubs {
        event_hub_connections => ["{AZURE_EVENT_HUB_CONNECTION_STRING};EntityPath=logstash"]
        threads => 2
        decorate_events => true
        consumer_group => "$Default"
        storage_connection => "{STORAGE_ACCOUNT_CONNECTION_STRING}"
        storage_container => "logstash"
        }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => ["sample-elasticsearch:9200" ]
        user => "elastic"
        password => "Password1$"
        index => "azureeventhub-%{+YYYY.MM.dd}"
        ssl => true
        cacert => "/usr/share/logstash/config/elastic-stack-ca.pem"
      }
    }
  azureredis.cfg: |
    input {
      redis {
        host => "{YOUR_REDIS_HOST_NAME}"
        key => "messages"
        data_type => "channel"
        password => "{YOUR_REDIS_SECONDARY_KEY}"
        port => 6380
        ssl => true
     }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => ["sample-elasticsearch:9200" ]
        user => "elastic"
        password => "Password1$"
        index => "azureredis-%{+YYYY.MM.dd}"
        ssl => true
        cacert => "/usr/share/logstash/config/elastic-stack-ca.pem"
      }
    }
  logstash.conf: |

Create a Kubernetes Deployment

The only change needed to Logstash_Deployment.yaml is to mount the new pipeline configuration for Redis i.e. azureredis.cfg. You can refer to previous articles of this series for further details about this resource.

volumeMounts:
- name: sample-logstash-configmap
mountPath: /usr/share/logstash/azureredis.cfg
subPath: azureredis.cfg

Deploy Logstash resources to AKS

Deploy Logstash resources to AKS. Once deployed run AzureRedisCacheSample to publish messages to Redis. This is a sample client (.NET Core 2.1) to send messages to Redis channel. This sample also subscribes to Redis channel locally and prints the messages. You need to update StackExchange.Redis connection string and channel name. After sending a few messages, navigate to Kibana endpoint and you will see events received and events emitted in monitoring section You can also create an Index filter in Kibana and messages sent to Redis will be displayed This completes the article on ingesting Azure Redis Cache messages into Elasticsearch using Logstash's Redis plugin. The sample code for this article can be downloaded from GitHub.

Azure Kubernetes Service (AKS): Azure AD SAML based Single Sign on to secure Elasticsearch and Kibana and securing communications in ELK

Atul.Verma — Tue, 09 Oct 2018 05:52:34 GMT

This is second part of the series on deploying Elasticsearch, Logstash and Kibana (ELK) to Azure Kubernetes Service cluster. In this article I am going to share steps needed to enable Azure AD SAML based single sign on to secure Elasticsearch and Kibana hosted in AKS. I will also go through steps needed to secure communications in ELK cluster. The first part describes steps needed to deploy ELK to AKS and consume messages from Azure Event Hub https://blogs.msdn.microsoft.com/atverma/2018/09/24/azure-kubernetes-service-aks-deploying-elasticsearch-logstash-and-kibana-elk-and-consume-messages-from-azure-event-hub/ The third part describes steps needed to ingest Azure Redis Cache messages into Elasticsearch using Logstash's Redis plugin. https://blogs.msdn.microsoft.com/atverma/2018/10/14/ingest-azure-redis-cache-messages-into-elasticsearch-logstash-and-kibana-cluster-deployed-in-azure-kubernetes-service-aks Using SAML SSO for Elasticsearch with AAD means that Elasticsearch does not need to be seeded with any user accounts from the directory. Instead, Elasticsearch is able to rely on the claims sent within a SAML token in response to successful authentication to determine identity and privileges. I have referred to this article to enable SAML based single sign on for Elasticsearch. Kibana, as the user facing component, interacts with the user’s browser and receives all the SAML messages that the Azure AD sends to the Elastic Stack Service Provider. Elasticsearch implements most of the functionality a SAML Service Provider needs. It holds all SAML related configuration in the form of an authentication realm and it also generates all SAML messages required and passes them to Kibana to be relayed to the user’s browser. It finally consumes all SAML Responses that Kibana relays to it, verifies them, extracts the necessary authentication information and creates the internal authentication tokens based on that. The component diagram has been updated to add Azure AD SAML based SSO integration. The dev tools used to develop these components are Visual Studio for Mac/VS Code, AKS Dashboard, kubectl, bash and openssl. The code snippets in this article are mostly yaml snippets and are included for reference only as formatting may get distorted thus please refer to GitHub repository for formatted resources.

Azure AD - Create an Enterprise Application

Before any messages can be exchanged between an Identity Provider and a Service Provider, one needs to know the configuration details and capabilities of the other. This configuration and the capabilities are encoded in an XML document, that is the SAML Metadata of the SAML entity. Thus we need to create non-gallery enterprise application within Azure AD. Please refer to these documents at msdn and elastic for more details as I am going to share main pointers as listed below

In Azure Portal, navigate to Azure Active Directory > Enterprise Applications > New application > Non-gallery application and create an Application.
Configure Single Sign on and specify Identifier and Reply URL. You need to update KIBANA_URL placeholder with DNS of Kibana as displayed below based on your Kibana endpoint. Please note that Reply URL needs to be an HTTPS endpoint. I will also cover securing ELK cluster in this article where I am going to share steps needed to enable TLS for Kibana.
Add a new User Claim by specifying 'Name' and 'Namespace' and select 'user.assignedroles' as source attribute. Role mappings will be defined in Elasticsearch for this claim based on which Elasticsearch roles are assigned to each user. I have specified name as role and namespace as https://schemas.microsoft.com/ws/2008/06/identity/claims/role. Based on values you specify you will need to Elasticsearch.yml.
Application Roles: The application roles are defined in the application's registration manifest. When a user signs into the application, Azure AD emits a roles claim for each role that the user has been granted individually to the user and from their group membership. In order to add a role, you will need to edit the manifest for this App by navigating to Azure Active Directory > App Registrations > Select Application and Edit Manifest. Please refer to this document about creating App Roles in Azure AD. Create a few roles by editing manifest as displayed below and save the manifest.
Assign Role to users: In order to assign roles to users, add a few users. Navigate to Azure Active Directory > Enterprise Applications > Select application > Users and Groups > Add User and assign roles you created in previous step to these users. These roles are going to be mapped to Elasticsearch roles using role_mapping API in subsequent sections.
Finally, download Federation Metadata Xml and take note of Azure AD Identifier URL as these are going to be needed later for setting up SAML based SSO in Elasticsearch. Since I have specified Elasticsearch as Application name, the name of this file is Elasticsearch.xml.

Encrypting communications in ELK Cluster

X-Pack security enables you to encrypt traffic to, from, and within your Elasticsearch cluster. Connections are secured using Transport Layer Security (TLS), which is commonly referred to as "SSL". TLS requires X.509 certificates to perform encryption and authentication of the application that is being communicated with. In order for the communication between nodes to be truly secure, the certificates must be validated. The recommended approach for validating certificate authenticity in a Elasticsearch cluster is to trust the certificate authority (CA) that signed the certificate. By doing this, as nodes are added to your cluster they just need to use a certificate signed by the same CA and the node is automatically allowed to join the cluster. Additionally, it is recommended that the certificates contain subject alternative names (SAN) that correspond to the node’s IP address and DNS name so that hostname verification can be performed. However, I will be generating self signed certificates using elasticsearch-certutil command line tool. You can read more about setting up TLS on a Cluster. I already have Elasticsearch cluster running which I had created during first part of this series and I am going to harness the same to create self signed certificates. If you have certificates, you need to skip steps listed below to generate certificates.

Generate new local CA by running command bin/elasticsearch-certutil ca. The output file name is elastic-stack-ca.p12 and password is Password1$. Change the name and password based on your preferences.
Generate self signed certificate bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --dns sample-elasticsearch. Enter password for CA (elastic-stack-ca.p12). The output file name is elastic-certificates.p12 and password is Password1$. Change the name and password based on your preferences. The --dns parameter value 'sample-elasticsearch' is the Elasticsearch host name and needed for host name verification(especially for Logstash pipeline since hostname verification can't be turned off when ssl is enabled).
Generate a cert to configure Kibana to encrypt communications between the browser and Kibana server bin/elasticsearch-certutil cert --pem -ca elastic-stack-ca.p12 --dns KIBANA_URL. Update the --dns parameter value based on your DNS for Kibana endpoint. This is the same value which was specified during SAML configuration. The output file name is certificate-bundle.zip. If you unzip this folder, you will see instance.crt and instance.key files which we are going to need for setting up TLS in Kibana.

I have generated certs on existing Elasticsearch container, thus the next step is to copy these certs from AKS container to local machine. I will use kubectl command line tool to copy these files from AKS container to certs folder on my machine as displayed below kubectl cp default/sample-elasticsearch-0:elastic-stack-ca.p12 /Users/username/certs/elastic-stack-ca.p12


kubectl cp default/sample-elasticsearch-0:elastic-certificates.p12 /Users/username/certs/elastic-certificates.p12


kubectl cp default/sample-elasticsearch-0:certificate-bundle.zip /Users/username/certs/certificate-bundle.zip

After copying these files to local machine, I used OpenSSL utility to create elastic-stack-ca.pem file which Kibana and Logstash will need to communicate with TLS enabled Elasticsearch

openssl pkcs12 -in /Users/username/certs/elastic-stack-ca.p12 -clcerts -nokeys -chain -out /Users/username/certs/elastic-stack-ca.pem

Now that certificates needed to enable TLS are ready, the next step is to create Elasticsearch, Logstash and Kibana resources. I will share couple of ways in which certificates and IDP federation metadata xml files can be deployed to ELK containers. In the first option, I am going to create custom docker images for Elasticsearch, Logstash and Kibana by bundling files needed to enable TLS and setup SAML based SSO. In the second option, I will create Secrets in Kubernetes for these files and mount these files instead of custom docker images. The second option is an afterthought after I posted this blog since I didn't like the idea of bundling certificates and files into docker image. The second option uses Secrets and Command concepts which are explained in subsequent sections. Thus instead of composing a new article, I have decided to update the existing article. The headers of steps which are specific to option A or option B have been updated. The solution structure of source code which you can download from GitHub is displayed below and you can see that I have copied cert files created in steps above and Elasticsearch.xml (Federation Metadata Xml). You will also see Dockerfiles per folder which I will need to create custom images (needed for first option only as mentioned above). I haven't committed certificates and IDP federation metadata xml thus you need to copy files you have created to corresponding folders before building docker images. The next step is to create Elasticsearch, Logstash and Kibana resources in AKS. Part 1 of this series provides detailed overview of these resources thus in this article I will only cover differences and changes needed to enable SSO and TLS as compared to Part 1 files.

Elasticsearch

Copy elastic-stack-ca.p12, elastic-certificates.p12 and Elasticsearch.xml to Elasticsearch folder. No changes are needed in ES_PersistentVolumeClaim.yml and ES_Service.yml files. The complete yaml files needed to enable TLS and deploy Elasticsearch to AKS can be downloaded from GitHub.

Elasticsearch Dockerfile (this step is specific to first option)

Elasticsearch DockerFile copies certs and federation metadata xml to config folder of Elasticsearch and sets keystore and truststore passwords. You need to update file names and passwords based on your specified values in previous steps.

 FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1
COPY --chown=elasticsearch:elasticsearch elastic-stack-ca.p12 /usr/share/elasticsearch/config/
COPY --chown=elasticsearch:elasticsearch elastic-certificates.p12 /usr/share/elasticsearch/config/
COPY --chown=elasticsearch:elasticsearch Elasticsearch.xml /usr/share/elasticsearch/config/
RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password  -xf"
RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password  -xf"
RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password  -xf"
RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password  -xf;"

The commands to build and publish docker images to Docker Hub are

Build Docker Image: docker build -t testelasticsearch .
Tag Docker Image: docker tag testelasticsearch {YOUR_DOCKER_REPO}
Publish Image to Docker Hub: docker push {YOUR_DOCKER_REPO}

Please update {YOUR_DOCKER_REPO} placeholder with your docker repository.

Create A Kubernetes Secret (this step is specific to second option)

Since in second option, no custom docker files are needed, I am going to create secret for certificates and federation xml files. Kubernetes will save secret in base64 format. The command to create a secret from multiple files is

kubectl create secret generic elasticsearchsecret --from-file=./Elasticsearch/elastic-certificates.p12 --from-file=./Elasticsearch/Elasticsearch.xml --from-file=./Elasticsearch/elastic-stack-ca.p12

Create a Kubernetes ConfigMap

The main pointers about changes needed to this resource as compared to first part of this series are

SSL is enabled for Elasticsearch: xpack.security.transport.ssl.enabled: true, xpack.security.http.ssl.enabled: true
Verification of certificate is set xpack.security.transport.ssl.verification_mode: certificate
Keystore and Truststore type isPKCS12, path is location of cert bundled in image elastic-certificates.p12 and password is Password1$. Update these values based on your specified values.
Cross origin resource sharing is enabled http.cors.enabled: true
Two realms are specified i.e. native and saml underxpack: security: authc: realms:. Order of native realm is 0 so it will take precedence over saml realm.
idp.metadata.path is path of the Federation metadata xml which was downloaded from Azure AD SAML configuration.
idp.entity_id is the Azure AD identifier URL which was copied from Azure AD SAML configuration.
sp.entity_id is the Kibana endpoint
sp.acs is the Kibana reply URL
sp.logout is the Kibana logout URL
attributes.principal is mapped to namespace of the name user claim configured in Azure AD
attributes.groups is mapped to namespace of the role user claim configured in Azure AD

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-elasticsearch-configmap
  namespace: default
data:
  elasticsearch.yml: |
    cluster.name: "sample-elasticsearch-cluster"
    network.host: 0.0.0.0
    discovery.zen.minimum_master_nodes: 1
    # Update max_local_storage_nodes value based on number of nodes
    node.max_local_storage_nodes: 1
    xpack.security.enabled: true
    xpack.monitoring.collection.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.license.self_generated.type: trial
    xpack.ssl.keystore.type: PKCS12
    xpack.ssl.keystore.path: "elastic-certificates.p12"
    xpack.ssl.keystore.password: Password1$
    xpack.ssl.truststore.type: PKCS12
    xpack.ssl.truststore.path: "elastic-certificates.p12"
    xpack.ssl.truststore.password: Password1$
    xpack.security.transport.ssl.keystore.path: "elastic-certificates.p12"
    xpack.security.transport.ssl.truststore.path: "elastic-certificates.p12"
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: "elastic-certificates.p12"
    xpack.security.http.ssl.truststore.path: "elastic-certificates.p12"
    xpack.ssl.verification_mode: certificate
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    http.max_header_size: 16kb
    xpack:
      security:
        authc:
          realms:
            native1:
                type: native
                order: 0
            saml1:
              type: saml
              order: 2
              idp.metadata.path: "/usr/share/elasticsearch/config/Elasticsearch.xml"
              idp.entity_id: "AZURE_AD_IDENTIFIER_URL"
              sp.entity_id:  "https://KIBANA_URL:5601/"
              sp.acs: "https://KIBANA_URL:5601/api/security/v1/saml"
              sp.logout: "https://KIBANA_URL:5601/logout"
              attributes.principal: "https://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"
              attributes.groups: "https://schemas.microsoft.com/ws/2008/06/identity/claims/role"
  role_mapping.yml: |

Create a Kubernetes StatefulSet (this step is specific to first option)

Update ES_Statefulset.yaml file to point to Docker image image: YOUR_ELASTICSEARCH_IMAGE

Create a Kubernetes StatefulSet (this step is specific to second option)

I have created ES_Statefulset_V2.yaml file and main pointers are

securityContext: runAsUser: 2000 will run Elasticsearch as a non-root user.
command:["/bin/bash","-c", ...] is going to execute several commands piped together when POD is started
- is going to assign keystore and truststore passwords
- set bootstrap.password
- start ElasticSearch
- you can also use shell script
Environment variable - name: PASSWORD is created which is used in commands. I have specified value in yaml however a better option is to mount this from secret e.g env[].valueFrom.secretKeyRef
Added - name: sample-elasticsearch-elasticsearchsecret volume and mounting elastic-stack-ca.p12, elastic-certificates.p12 and Elasticsearch.xml from secret.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sample-elasticsearch
  labels:
    service: sample-elasticsearch
spec:
  serviceName: sample-elasticsearch
  # Number of PODS
  replicas: 1
  selector:
    matchLabels:
      service: sample-elasticsearch
  template:
    metadata:
      labels:
        service: sample-elasticsearch
    spec:
      terminationGracePeriodSeconds: 300
      initContainers:
      # https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
      - name: increase-the-vm-max-map-count
        image: busybox
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      containers:
      - name: sample-elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1
        securityContext:
          runAsUser: 2000
        resources:
          limits:
            cpu: 2000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: tcp
        command: ["/bin/bash","-c", "echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password  -xf; \
                echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password  -xf; \
                echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password  -xf ; \
                echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password  -xf ; \
                echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add bootstrap.password  -xf; \
                /usr/share/elasticsearch/bin/elasticsearch ;"]
        env:
          - name: cluster.name
            value: "sample-elasticsearch-cluster"
          - name: "xpack.security.enabled"
            value: "true"
          - name: "TAKE_FILE_OWNERSHIP"
            value: “1”
          - name: PASSWORD
            value: Password1$
        volumeMounts:
        - name: sample-elasticsearch-data-claim
          mountPath: /usr/share/elasticsearch/data
        - name: sample-elasticsearch-configmap
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml
        - name: sample-elasticsearch-configmap
          mountPath: /usr/share/elasticsearch/config/role_mapping.yml
          subPath: role_mapping.yml
        - name: sample-elasticsearch-elasticsearchsecret
          mountPath: /usr/share/elasticsearch/config/elastic-stack-ca.p12
          subPath: elastic-stack-ca.p12
        - name: sample-elasticsearch-elasticsearchsecret
          mountPath: /usr/share/elasticsearch/config/elastic-certificates.p12
          subPath: elastic-certificates.p12
        - name: sample-elasticsearch-elasticsearchsecret
          mountPath: /usr/share/elasticsearch/config/Elasticsearch.xml
          subPath: Elasticsearch.xml
      volumes:
      - name: sample-elasticsearch-data-claim
        persistentVolumeClaim:
           claimName: sample-elasticsearch-data-claim
      - name: sample-elasticsearch-configmap
        configMap:
            name: sample-elasticsearch-configmap
      - name: sample-elasticsearch-elasticsearchsecret
        secret:
            secretName: elasticsearchsecret

Kibana

Copy instance.crt,instance.key andelastic-stack-ca.pem to Kibana folder. The complete yaml files needed to deploy Kibana to AKS can be downloaded from GitHub.

Kibana Dockerfile (this step is specific to first option)

Kibana Dockerfile copies instance.crt and instance.key needed to enable TLS for Kibana along with elastic-stack-ca.pem so that Elasticsearch can trust Kibana.

FROM docker.elastic.co/kibana/kibana:6.4.1
COPY instance.crt /usr/share/kibana/config/
COPY instance.key /usr/share/kibana/config/
COPY elastic-stack-ca.pem /usr/share/kibana/config/

Create A Kubernetes Secret (this step is specific to second option)

Since in second option no custom docker files are needed, I am going to create secret for certificates needed for Kibana. I will update the Deployment.yaml to mount these files from secret. The command to create a secret from multiple files is

kubectl create secret generic kibanasecret --from-file=./Kibana/elastic-stack-ca.pem --from-file=./Kibana/instance.crt --from-file=./Kibana/instance.key

Create a Kubernetes ConfigMap

The main pointers about changes needed to this resource as compared to first part of this series are

Elasticsearch is TLS enabled elasticsearch.url: https://sample-elasticsearch:9200
elasticsearch.ssl.certificateAuthorities is set to path of elastic-stack-ca.pem
SSL is enabled server.ssl.enabled: true
server.ssl.key is set to path of instance.key
server.ssl.certificate is set to path of instance.crt
xpack.monitoring.elasticsearch.ssl.verificationMode is set to certificate
elasticsearch.ssl.verificationMode is set to certificate
xpack.security.authProviders is set to [saml, basic]. The order is important here as Kibana will use saml realm unless basic authorization header is passed in the request and in that case it will give precedence to native realm.
xpack.security.public is specified so that if Kibana instance is behind a proxy, it will tell Kibana how to form its public URL

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-kibana-configmap
  namespace: default
data:
  kibana.yml: |
    server.name: sample-kibana
    server.host: "0.0.0.0"
    elasticsearch.url: https://sample-elasticsearch:9200
    xpack.monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch.username: kibana
    elasticsearch.password: Password1$
    xpack.security.encryptionKey: "something_at_least_32_characters"
    elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/elastic-stack-ca.pem"
    server.ssl.enabled: true
    server.ssl.key: "/usr/share/kibana/config/instance.key"
    server.ssl.certificate: "/usr/share/kibana/config/instance.crt"
    xpack.monitoring.elasticsearch.ssl.verificationMode: certificate
    elasticsearch.ssl.verificationMode: certificate
    xpack.security.authProviders: [saml, basic]
    server.xsrf.whitelist: [/api/security/v1/saml]
    xpack.security.public:
      protocol: https
      hostname: KIBANA_URL

Create a Kubernetes Deployment (this step is specific to first option)

Update Kibana_Deployment.yaml file to point to Docker image image: YOUR_KIBANA_IMAGE

Create a Kubernetes Deployment (this step is specific to second option)

I have created Kibana_Deployment_V2.yaml file and main pointers are

Added - name: sample-kibana-kibanasecret volume and mounting instance.crt, instance.key and elastic-stack-ca.pem files from secret.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: sample-kibana
  labels:
    component: sample-kibana
spec:
  replicas: 1
  selector:
    matchLabels:
     component: sample-kibana
  template:
    metadata:
      labels:
        component: sample-kibana
    spec:
      containers:
      - name: sample-kibana
        image: docker.elastic.co/kibana/kibana:6.4.1
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: http
        volumeMounts:
        - name: sample-kibana-configmap
          mountPath: /usr/share/kibana/config/kibana.yml
          subPath: kibana.yml
        - name: sample-kibana-kibanasecret
          mountPath: /usr/share/kibana/config/elastic-stack-ca.pem
          subPath: elastic-stack-ca.pem
        - name: sample-kibana-kibanasecret
          mountPath: /usr/share/kibana/config/instance.crt
          subPath: instance.crt
        - name: sample-kibana-kibanasecret
          mountPath: /usr/share/kibana/config/instance.key
          subPath: instance.key
        resources: {}
        terminationMessagePath: "/dev/termination-log"
        terminationMessagePolicy: File
        imagePullPolicy: Always
        securityContext:
          privileged: false
      volumes:
      - name: sample-kibana-configmap
        configMap:
            name: sample-kibana-configmap
      - name: sample-kibana-kibanasecret
        secret:
            secretName: kibanasecret
      restartPolicy: Always
      terminationGracePeriodSeconds: 5
      dnsPolicy: ClusterFirst
      securityContext:
        privileged: true
      schedulerName: default-scheduler

Create a Kubernetes Service

Update Kibana_Service.yml file to use 5601 port since we have specified 5601 port for Kibana during SAML configuration and Elasticsearch.yml.

apiVersion: v1
kind: Service
metadata:
  name: sample-kibana
  labels:
    component: sample-kibana
spec:
  type: LoadBalancer
  selector:
    component: sample-kibana
  ports:
  - name: http
    port: 5601
    targetPort: http

Logstash

Copy elastic-stack-ca.pem to Elasticsearch folder. The yaml files needed to deploy Logstash to AKS can be downloaded from GitHub.

Logstash Dockerfile (this step is specific to option)

Logstash DockerFile copies certs needed to enable Logstash to communicate with Elasticsearch.

FROM docker.elastic.co/logstash/logstash:6.4.1
COPY elastic-stack-ca.pem /usr/share/logstash/config/

Create A Kubernetes Secret (this step is specific to second option)

Since in second option, no custom docker files are needed, I am going to create secret for certificates needed for Logstash. The command to create a secret from file is kubectl create secret generic logstashsecret --from-file=./Logstash/elastic-stack-ca.pem

Create a Logstash ConfigMap

The main pointers about changes needed to this resource as compared to first part of this series are

Elasticsearch is TLS enabled xpack.monitoring.elasticsearch.url: https://sample-elasticsearch:9200
xpack.monitoring.elasticsearch.ssl.ca is set to point to elastic-stack-ca.pem
Pipeline config for azureeventhubs.cfg is updated to
- Enable ssl ssl => true
- Set cacert to point to elastic-stack-ca.pem

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-logstash-configmap
  namespace: default
data:
  logstash.yml: |
    xpack.monitoring.elasticsearch.url: https://sample-elasticsearch:9200
    dead_letter_queue.enable: true
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: logstash_system 
    xpack.monitoring.elasticsearch.password: Password1$
    xpack.monitoring.elasticsearch.ssl.ca: "/usr/share/logstash/config/elastic-stack-ca.pem"
  pipelines.yml: |
    - pipeline.id: azureeventhubs
      path.config: "/usr/share/logstash/azureeventhubs.cfg"
  azureeventhubs.cfg: |
    input {
      azure_event_hubs {
        event_hub_connections => ["{AZURE_EVENT_HUB_CONNECTION_STRING};EntityPath=logstash"]
        threads => 2
        decorate_events => true
        consumer_group => "$Default"
        storage_connection => "{STORAGE_ACCOUNT_CONNECTION_STRING}"
        storage_container => "logstash"
        }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => ["sample-elasticsearch:9200" ]
        user => "elastic"
        password => "Password1$"
        index => "azureeventhub-%{+YYYY.MM.dd}"
        ssl => true
        cacert => "/usr/share/logstash/config/elastic-stack-ca.pem"
      }
    }
  logstash.conf: |

Create a Logstash Deployment (this step is specific to first option)

Update Logstash_Deployment.yaml file to point to Docker image image: YOUR_LOGSTASH_IMAGE

Create a Logstash Deployment (this step is specific to second option)

I have created Logstash_Deployment_V2.yaml file and main pointers are

Added - name: sample-logstash-logstashsecret volume and mounting elastic-stack-ca.pem from secret.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: sample-logstash
  labels:
    component: sample-logstash
spec:
  replicas: 1
  selector:
    matchLabels:
     component: sample-logstash
  template:
    metadata:
      labels:
        component: sample-logstash
    spec:
      containers:
      - name: sample-logstash
        image: YOUR_LOGSTASH_IMAGE
        volumeMounts:
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/config/logstash.yml
          subPath: logstash.yml
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/pipeline/logstash.conf
          subPath: logstash.conf
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/azureeventhubs.cfg
          subPath: azureeventhubs.cfg
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/azureredis.cfg
          subPath: azureredis.cfg
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/config/pipelines.yml
          subPath: pipelines.yml
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: http
      volumes:
        - name: sample-logstash-configmap
          configMap:
            name: sample-logstash-configmap

Deploy resources to AKS

Deploy Elasticsearch, Logstash and Kibana resources to AKS. Once external endpoint is created for Kibana service which takes a while proceed to next step to update UP with DNS name of Kibana.

Update a public IP resource with a DNS name

The IP address of the Kibana Service external endpoint is updated with DNS name using commands displayed below. You need to update IP and DNSNAME before running these commands based on your values.

bash-3.2$ IP="KIBANA_SERVICE_EXTERNAL_ENDPOINT_IP"
bash-3.2$ DNSNAME="KIBANA_URL"
bash-3.2$ PUBLICIPID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$IP')].[id]" --output tsv)
bash-3.2$ az network public-ip update --ids $PUBLICIPID --dns-name $DNSNAME

Map Users to Elasticsearch Roles using role_mapping API

Elasticsearch's role_mapping API is used to map user roles to Elasticsearch roles. Mapping between groups and role user claim along with mapping between principal and name user claim was defined in Elasticsearch.yml attributes.principal: "https://schemas.xmlsoap.org/ws/2005/05/identity/claims/name" attributes.groups:"https://schemas.microsoft.com/ws/2008/06/identity/claims/role" A few examples of using role_mapping API to map Roles are

This maps all the users of saml1 realm to kibana_user Elasticsearch role PUT /_xpack/security/role_mapping/aad-saml-allusers { "roles": [ "kibana_user" ], "enabled": true, "rules": { "field": { "realm.name": "saml1" } } }
This maps users of saml1 realm having groups field value as superuser to map to superuser Elasticsearch rolePUT /_xpack/security/role_mapping/aad-saml-superuser { "roles": [ "superuser" ], "enabled": true, "rules": { "all": [ { "field": { "realm.name": "saml1" } }, { "field": { "groups": "superuser" } } ] } }
This maps users of saml1 realm having groups field value as kibanauser to map to kibana_user Elasticsearch rolePUT /_xpack/security/role_mapping/aad-saml-kibanauser { "roles": [ "kibana_user" ], "enabled": true, "rules": { "all": [ { "field": { "realm.name": "saml1" } }, { "field": { "groups": "kibanauser" } } ] } }

This completes all the changes needed to enabled SAML based SSO and securing communications in ELK. Browse to https://KIBANA_URL:5601 and Elasticsearch will redirect to Azure AD and after successful login you will be redirected back to Kibana. Browsers will warn since self signed certs are used and after granting access, Kibana Dashboard will launch Browse to https://KIBANA_URL:5601/api/security/v1/me after authentication it will display mapped Elasticsearch roles and saml claims of the user. On sending messages to Azure Event Hub using the utility I had shared in first part, messages are going to get ingested into Elasticsearch through Logstash pipeline. The source code of this utility can be downloaded from GitHub. You can download source code for this article from GitHub repository

Azure Functions 2.0: Create, debug and deploy to Azure Kubernetes Service (AKS)

Atul.Verma — Wed, 26 Sep 2018 16:08:38 GMT

Azure Functions is a serverless compute service that enables you to run code on-demand without having to explicitly provision or manage infrastructure. You can read about Azure Functions 2.0 general availability @ Introducing Azure Functions 2.0. Runtime 2.0 runs on .NET Core 2, which means it can run on all platforms supported by .NET Core, including macOS and Linux. This enables cross-platform development and hosting scenarios. In this article I am going to

Create Azure Functions triggered by Azure Blob storage and Event hub in Visual Studio Code
Debug locally in Visual Studio Code
Deploy Azure Functions to Azure Kubernetes Service

Dev tools used to develop these components are Visual Studio Code for macOS, Docker, AKS Dashboard and kubectl.

Development environment

The steps needed to configure development environment (Azure Function 2.0 on macOS) are

Install Visual Studio Code, if it's not already installed
Install 'Azure Functions for Visual Studio Code' extension
Use Homebrew to install the Core Tools on macOS
- Install .NET Core 2.1 for macOS, if it's not already installed
- Install Homebrew, if it's not already installed.
- Install the Core Tools package:
  - Run command brew tap azure/functions
- - Before running the next command to install core tools, grant write access by running command sudo chown -R $(whoami) /usr/local/share/man/man5 /usr/local/share/man/man7
  - Run command brew install azure-functions-core-tools

Azure Kubernetes Service (AKS) Cluster

In case you don’t have AKS cluster up and running, please go through this article to Create AKS Cluster.

Azure Blob Storage

AzureWebJobsStorage: The Azure Functions runtime uses storage account connection string for all functions except for HTTP triggered functions. The storage account must be a general-purpose one that supports blobs, queues, and tables. Create a storage account which is going to be used by Azure Functions runtime. Please follow steps listed in this article to create a storage account.

AzureBlobStorage: Instead of creating a blob container in previously created storage account, I created a separate storage account and created a blob container namely 'testblob'. If you specify a different container name, you will need to update blob container name in TestBlobStorageFunc class. Container name and storage connection string will be specified in Azure function triggered by blog storage as we are going to see shortly.

Keep a note of Storage connection strings for both accounts as these need to be updated in 'local.settings.json' and 'DockerFile' files.

Event Hub

I created an Event hub and Event Hub name I have specified in sample solution is 'testeventhub'. If you select a different name, update event hub name in TestEventHubFunc class. Please follow steps listed in this article to create an Event Hub. Keep a note of Event hub connection string as this needs to be updated in 'local.settings.json' and 'DockerFile' files.

Create Azure Functions 2.0

The next step is to create Azure Functions project in Visual Studio Code. Open the folder in integrated terminal of Visual Studio Code and

Run command func init --docker to create Function 2.0 project and docker file as displayed below
If you open host.json file you will see "version": "2.0" which is required for a function app that targets V2 runtime.
Run command func new to create a new function and select BlobTrigger option and provide function name
Run command func new to create a new function and select EventHubTrigger and provide function name

Debug Azure Functions 2.0 locally in Visual Studio Code:

Before debugging locally, 'local.settings.json' file needs to be updated with connection strings of Storage accounts and Event Hub.


{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "{BLOB_STORAGE_CONNECTION_STRING}",
"AzureBlobStorage": "{BLOB_STORAGE_CONNECTION_STRING_OF_CONTAINER_HAVING_TRIGGER}",
"AzureEventHubConnectionString":"{EVENT_HUB_CONNECTION_STRING}",
"FUNCTIONS_WORKER_RUNTIME": "dotnet"
}
}

Update BlobTrigger function and specify container name and connection e.g.

public static void Run([BlobTrigger("testblob/{name}", Connection = "AzureBlobStorage")]Stream myBlob, string name, ILogger log)

Update EventHubTrigger function and specify event hub name and connection e.g.

public static void Run([EventHubTrigger("testeventhub", Connection = "AzureEventHubConnectionString")]string myEventHubMessage, ILogger log)

Start debugging in Visual Studio Code and Functions will get triggered once you send a few messages to Event hub and upload a few blobs.

Deploy Azure Functions 2.0 to AKS

The Values defined in 'local.settings.json' are used only for debugging locally. Thus, we need to specify Values before deploying to AKS. In order to do that DockerFile needs to be updated so that we can pass connection strings through environment variables. Add environment variables listed below to DockerFile and update placeholders with your connection strings.

ENV AzureWebJobsStorage="{BLOB_STORAGE_CONNECTION_STRING}"
ENV AzureBlobStorage="{BLOB_STORAGE_CONNECTION_STRING_OF_CONTAINER_HAVING_TRIGGER}"
ENV AzureEventHubConnectionString="{EVENT_HUB_CONNECTION_STRING}"

Finally, it's time to deploy functions to AKS and command is func deploy --platform kubernetes --name {DockerHubRepositoryName} --registry {DockerHubRegistry}. Update placeholders with your Docker hub repository name and account. Once deployment is complete, you can open the endpoint to verify the deployment. You can also open AKS dashboard to view deployed resources under namespace 'azure-functions'. You will find that Horizontal Pod Autoscaler resource is created which automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization. Send a few messages to Event Hub and upload a few blobs. You can view logs of functions deployed to AKS by running command func logs --name {DockerHubRepositoryName} --platform kubernetes. Update DockerHubRepositoryName placeholder with the name you specified in previous step.

This completes the article about deploying Functions 2.0 to Kubernetes. You can download the source code for this article from GitHub.

Azure Kubernetes Service (AKS): Deploying Elasticsearch, Logstash and Kibana (ELK) and consume messages from Azure Event Hub

Atul.Verma — Mon, 24 Sep 2018 04:15:09 GMT

This article is part of the series of blogs on Azure Kubernetes Service (AKS). In this article I am going to share my experience of setting up Elasticsearch, Logstash and Kibana cluster in Azure Kubernetes Service (AKS) and consume messages from Event Hub. After the end of this article, we are going to have fully functional ELK stack with Azure Event Hub integration. A sample client App (e.g. IOT device) will be publishing messages to Event Hub and these messages will be ingested into Elasticsearch using 'Azure Event Hub' plugin of Logstash. This article needs x-pack features of Elasticsearch thus I will show steps needed to activate trial license. The second part of this series goes through steps needed to enable Azure AD SAML based single sign on to secure Elasticsearch and Kibana hosted in AKS. The third part of this series goes through steps needed to ingest Azure Redis Cache messages into Elasticsearch using Logstash's Redis plugin. The dev tools used to develop these components are Visual Studio for Mac/Visual Studio 2017, AKS Dashboard as well as kubectl commands are used to create/manager Kubernetes resources in AKS.

Azure Kubernetes Service (AKS) Cluster

In case you don’t have AKS cluster up and running, please go through this article to Create AKS Cluster. Once AKS cluster is up and running, you can go through the rest of the article. The code snippets in this article are mostly yaml snippets and are included for reference only as formatting may get distorted thus please refer to GitHub repository for formatted resources.

Event Hub

Event hub messages will be ingested into Elasticsearch using Logstash pipeline, thus first resource I am going to create is Event Hub. Please follow steps listed in this article to create an Event Hub. The main pointers are

Event Hub name I have selected for this sample solution is 'logstash'. If you select a different name, update event hub name in the source code.
Keep a note of Event Hub connection string as this needs to be updated in Logstash pipeline and Event Hub messages publisher client.
Source code uses '$Default' Resource_Group. Update this value if you create a Resource_Group

Azure Blob Storage

The next resource which you will need to create is Azure Blob Storage. Please follow steps listed in this article to create a storage account. Once storage account is created, create Blob Service-> Blobs. The main pointers are

Blob container name I have specified in source code is 'logstash'. If you select a different name, update storage container name in the source code.
Keep a note of Storage connection string as this needs to be updated in Logstash pipeline.

Client App to send messages to Event Hub

AzureEventHubSample project is client app to send messages to Event hub. You will need to update connectionString variable with Event Hub connection string and name of the hub. You can download the source code of this publisher client from Github.

Deploy Elasticsearch to Azure Kubernetes Service

Elasticsearch is a near real time search platform. The steps needed to deploy Elasticsearch to AKS cluster are listed below

Create a Persistent Volume

Persistent volume claim is needed to store Elasticsearch data. The yaml snippet to create a 5 GB storage is displayed below. The StatefulSet resource is going to mount files to this storage claim. You can read more about Persistent Volumes.

 apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sample-elasticsearch-data-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
   requests:
    storage: 5Gi

Create a Kubernetes ConfigMap

ConfigMaps allow you to decouple configuration aspects from image and a few pointers about the yaml snippet displayed below are

elasticsearch.yml and role_mapping.yml files will be mounted from ConfigMap
xpack-security is enabled
Update max_local_storage_nodes based on number of Pods
role_mapping.yml is intentionally left blank as it will be discussed in detail in next blogs of this series

 apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-elasticsearch-configmap
  namespace: default
data:
  elasticsearch.yml: |
    cluster.name: "sample-elasticsearch-cluster"
    network.host: 0.0.0.0
    discovery.zen.minimum_master_nodes: 1
    #Update max_local_storage_nodes value based on number of nodes
    node.max_local_storage_nodes: 1
    xpack.security.enabled: true
    xpack.monitoring.collection.enabled: true
  role_mapping.yml: |

You can read more about ConfigMap.

Create a Kubernetes Service

The next step is to create a Kubernetes Service for Elasticsearch. As you can see in yaml snippet below, port 9200/9300 is defined and type is ClusterIP i.e. this service doesn't has external endpoints. Kubernetes will use to selector 'service: sample-elasticsearch' to map to the deployment as you are going to see next. You can read more about Services.

 apiVersion: v1               
kind: Service                
metadata:                    
  name: sample-elasticsearch        
  labels:                    
    service: sample-elasticsearch
spec:                        
  type: ClusterIP
  selector:                   
    service: sample-elasticsearch
  ports:                     
  - name: http               
    port: 9200               
    targetPort: 9200
    protocol: TCP            
  - name: transport          
    port: 9300               
    targetPort: 9300
    protocol: TCP

Create a Kubernetes StatefulSet

Kubernetes StatefulSet is the workload API used to manage stateful applications. You can read more about StatefulSets. The yaml snippet to create a StatefulSet is displayed below and a few pointers are

service: sample-elasticsearch matches to the selector defined in the service.
I have specified replicas: 1 which means that only one instance of Pod will be created by Kubernetes. You can update this value as needed, however you will need to update ConfigMap value .max_local_storage_nodes: 1 too.
The docker image being used to create this resource is image:docker.elastic.co/elasticsearch/elasticsearch:6.4.1.
By default, Elasticsearch runs inside the container as user elasticsearch using uid:gid 1000:1000. If you are bind-mounting a local directory or file, ensure it is readable by this user, while the data and log dirs additionally require write access. This is the reason I have used environment variable - name: "TAKE_FILE_OWNERSHIP" value: “1”. The alternate option is to grant write access by adding an initcontainer step e.g. command: - sh - -c - chown -R 1000:1000 /usr/share/elasticsearch/data. You can read more at Elasticsearch Docker.
mmap count has been increased vm.max_map_count=262144 by adding an initcontainer step. You can read more about Elasticsearch virtual memory.
ConfigMap defined in previous step is used to mount elasticsearch.yml and role_mapping.yml files i.e. configMap: name: sample-elasticsearch-configmap.
Lastly, persistent volume claim created above is used for storage i.e. persistentVolumeClaim: claimName:sample-elasticsearch-data-claim.
Assign CPU resource as needed by updating section resources:limits:. You can read more about Assign CPU Resources to Containers and Pods

 apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sample-elasticsearch
  labels:
    service: sample-elasticsearch
spec:
  serviceName: sample-elasticsearch
  # Number of PODS
  replicas: 1
  selector:
    matchLabels:
      service: sample-elasticsearch
  template:
    metadata:
      labels:
        service: sample-elasticsearch
    spec:
      terminationGracePeriodSeconds: 15
      initContainers:
      # https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
      - name: increase-the-vm-max-map-count
        image: busybox
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      containers:
      - name: sample-elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1
        resources:
          limits:
            cpu: 2000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: tcp
        env:
          - name: cluster.name
            value: "sample-elasticsearch-cluster"
          - name: "xpack.security.enabled"
            value: "true"
          - name: "TAKE_FILE_OWNERSHIP"
            value: “1”
        volumeMounts:
        - name: sample-elasticsearch-data-claim
          mountPath: /usr/share/elasticsearch/data
        - name: sample-elasticsearch-configmap
          mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          subPath: elasticsearch.yml
        - name: sample-elasticsearch-configmap
          mountPath: /usr/share/elasticsearch/config/role_mapping.yml
          subPath: role_mapping.yml
      volumes:
      - name: sample-elasticsearch-data-claim
        persistentVolumeClaim:
           claimName: sample-elasticsearch-data-claim
      - name: sample-elasticsearch-configmap
        configMap:
            name: sample-elasticsearch-configmap

By default Elasticsearch will be deployed with basic license. After Elasticsearch is deployed, the next step is to activate trail license of Elasticsearch to use x-pack features of Elasticsearch.

Enable Trial Version of Elasticsearch

The steps needed to activate trail license are

Run command kubectl port-forward sample-elasticsearch-0 9200:9200 and now you can access Elasticsearch endpoint at https://localhost:9200. POST https://localhost:9200/_xpack/license/start_trial?acknowledge=true request from any rest client. This is going to activate the trial license.

You can verify if trail version is activated by GET https://localhost:9200/_xpack/license request from any rest client.
Once trial license is activated you can close the terminal as port forwarding isn't needed anymore.

Setup Password for buit-in user accounts of Elasticsearch

x-pack security feature of Elasticsearch is used to secure access thus we now need to setup passwords for built-in user accounts and the steps are

Connect to Elasticsearch POD by running command kubectl exec -ti sample-elasticsearch-0 bash
Run command bin/elasticsearch-setup-passwords interactive to setup built-in user passwords interactively. For this sample I have specified same password i.e. Password1$ for all accounts thus you will need to change password you specified in a few places in source code.

Deploy Kibana to Azure Kubernetes Service

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack. The steps needed to deploy Kibana to AKS are listed below

Create a Kubernetes ConfigMap

The yaml snippet to create this resource is displayed below and a few pointers are

kibana.yml file will be mounted from ConfigMap
Kibana points to Elasticsearch based on elasticsearch.url: https://sample-elasticsearch:9200
Updateelasticsearch.password: Password1$ based on your specified password for kibana (built-in) user

 apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-kibana-configmap
  namespace: default
data:
  kibana.yml: |
    server.name: sample-kibana
    server.host: "0"
    elasticsearch.url: https://sample-elasticsearch:9200
    xpack.monitoring.ui.container.elasticsearch.enabled: true
    elasticsearch.username: kibana
    elasticsearch.password: Password1$

Create a Kubernetes Service

The next step is to create a Kubernetes Service for Kibana. As you can see in yaml snippet below, port 80 is defined and type is LoadBalancer i.e. this service has external endpoints. Kubernetes will use to selector 'component: sample-kibana' to map to the deployment as you are going to see next. The creation of this service is going to take a while and once done you can get the external endpoint of this service either by opening AKS Dashboard or running Kubectl command kubectl describe services sample-kibana.

 apiVersion: v1
kind: Service
metadata:
  name: sample-kibana
  labels:
    component: sample-kibana
spec:
  type: LoadBalancer
  selector:
    component: sample-kibana
  ports:
  - name: http
    port: 80
    targetPort: http

Create a Kubernetes Deployment

The next step is to create a Kubernetes Deployment for Kibana. The yaml snippet is displayed below and a few pointers are

The docker image being used to create this resource is image: docker.elastic.co/kibana/kibana:6.4.1
You can change the number of pods by updating replicas: 1
Labelcomponent: sample-kibana has to match the selector defined in the service
ConfigMap defined in previous step is used to mount kibana.yml file i.e. configMap: name: sample-kibana-configmap

 apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: sample-kibana
  labels:
    component: sample-kibana
spec:
  replicas: 1
  selector:
    matchLabels:
     component: sample-kibana
  template:
    metadata:
      labels:
        component: sample-kibana
    spec:
      containers:
      - name: sample-kibana
        image: docker.elastic.co/kibana/kibana:6.4.1
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: http
        volumeMounts:
        - name: sample-kibana-configmap
          mountPath: /usr/share/kibana/config/kibana.yml
          subPath: kibana.yml
        resources: {}
        terminationMessagePath: "/dev/termination-log"
        terminationMessagePolicy: File
        imagePullPolicy: Always
        securityContext:
          privileged: false
      volumes:
      - name: sample-kibana-configmap
        configMap:
            name: sample-kibana-configmap
      restartPolicy: Always
      terminationGracePeriodSeconds: 5
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler

Open the external endpoint of Kibana service once deployment is completed. Since x-pack security is enabled, Kibana will prompt for credentials.

Deploy Logstash to Azure Kubernetes Service

Create a Kubernetes ConfigMap

The yaml snippet to create ConfigMap is displayed below and a few pointers are

logstash.yml file will be mounted from ConfigMap
pipelines.yml file will be mounted from ConfigMap. Multiple pipelines can be defined in this file e.g. you can see that AzureEventHubs pipeline is defined.
azureeventhub.cfg file will be mounted from ConfigMap. The Logstash event processing pipeline has three stages: inputs → filters → outputs. This file defines the logstash pipeline for Azure Event Hub.
- Update {AZURE_EVENT_HUB_CONNECTION_STRING} and {STORAGE_ACCOUNT_CONNECTION_STRING} values based on your Event Hub and Storage Account values.
- Update EntityPath in case your event hub is named differently then logstash.
- Update storage_container in case your storage container is named differently than logstash.
- Update consumer_group in case your consumer group is different than $Default. Specifying $Default isn't an ideal option.
- Index name is defined as index => "azureeventhub-%{+YYYY.MM.dd}
logstash.conf file will be mounted from ConfigMap
Elasticsearch URL is defined in xpack.monitoring.elasticsearch.url: https://sample-elasticsearch:9200
Updatexpack.monitoring.elasticsearch.password: Password1$ based on your specified password for logstash_system (built-in) user.
Elasticsearch endpoint is defined in hosts => [ "sample-elasticsearch:9200" ]
Update password => "Password1$" to the specified password of your elastic (built-in) user

 apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-logstash-configmap
  namespace: default
data:
  logstash.yml: |
    xpack.monitoring.elasticsearch.url: https://sample-elasticsearch:9200
    dead_letter_queue.enable: true
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: logstash_system 
    xpack.monitoring.elasticsearch.password: Password1$
  pipelines.yml: |
    - pipeline.id: azureeventhubs
      path.config: "/usr/share/logstash/azureeventhubs.cfg"
  azureeventhubs.cfg: |
    input {
      azure_event_hubs {
        event_hub_connections => ["{AZURE_EVENT_HUB_CONNECTION_STRING};EntityPath=logstash"]
        threads => 2
        decorate_events => true
        consumer_group => "$Default"
        storage_connection => "{STORAGE_ACCOUNT_CONNECTION_STRING}"
        storage_container => "logstash"
        }
    }
    filter {
    }
    output {
      elasticsearch {
        hosts => [ "sample-elasticsearch:9200" ]
        user => "elastic"
        password => "Password1$"
        index => "azureeventhub-%{+YYYY.MM.dd}"
      }
    }
  logstash.conf: |

Create a Kubernetes Service

The next step is to create a Kubernetes Service for Logstash. As you can see in yaml snippet below, port 80 is defined and type is ClusterIP i.e. this service has no external endpoints. Kubernetes will use to selector 'component: sample-logstash' to map to the deployment as you are going to see next.

 apiVersion: v1
kind: Service
metadata:
  name: sample-logstash
  labels:
    component: sample-logstash
spec:
  type: ClusterIP
  selector:
    component: sample-logstash
  ports:
  - name: http
    port: 80
    targetPort: http

Create a Kubernetes Deployment

The next step is to create a Kubernetes Deployment for Logstash. The yaml snippet is displayed below and a few pointers are

The docker image being used to create this resource is docker.elastic.co/logstash/logstash:6.4.1
You can change the number of pods by updating replicas: 1
Labelcomponent: sample-logstash has to match the selector defined in the service
ConfigMap defined in previous step is used to mount logstash.yml, logstash.conf, pipelines.yml and azureeventhubs.cfg files i.e. configMap: name: sample-logstash-configmap

 apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: sample-logstash
  labels:
    component: sample-logstash
spec:
  replicas: 1
  selector:
    matchLabels:
     component: sample-logstash
  template:
    metadata:
      labels:
        component: sample-logstash
    spec:
      containers:
      - name: sample-logstash
        image: docker.elastic.co/logstash/logstash:6.4.1
        volumeMounts:
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/config/logstash.yml
          subPath: logstash.yml
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/pipeline/logstash.conf
          subPath: logstash.conf
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/azureeventhubs.cfg
          subPath: azureeventhubs.cfg
        - name: sample-logstash-configmap
          mountPath: /usr/share/logstash/config/pipelines.yml
          subPath: pipelines.yml
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 5601
          name: http
      volumes:
        - name: sample-logstash-configmap
          configMap:
            name: sample-logstash-configmap

As mentioned earlier, Logstash will use azure_event_hubs plugin to ingest data into Elasticsearch. You can get installed plugins list by following steps listed below

Run command kubectl exec -ti {Logstash_Pod_Name} bash to connect to Logstash POD.
Run command bin/logstash-plugin list to see installed plugins

Demo

After all resources are deployed to AKS, run client app to send messages to Event hub. Open Kibana and you will see ELK stack statistics in Monitoring section. The number of messages sent by client App to Event hub will be displayed in Logstash's Events received and events emitted statistics. The Discover tab of Kibana will display events ingested into Elasticsearch once you create index filters e.g. I have selected index filter as 'azureeventhub*' and you can see that this index naming convention was defined in azureeventshub.cfg pipelineindex => "azureeventhub-%{+YYYY.MM.dd}. You can download source code for this article from GitHub repository The second part of this series is Azure Kubernetes Service (AKS): Azure AD SAML based Single Sign on to secure Elasticsearch and Kibana and securing communications in ELK

Azure Dev Spaces - Debug Containers directly in Azure Kubernetes Service (AKS) from Visual Studio 2017

Atul.Verma — Wed, 19 Sep 2018 03:34:39 GMT

This is second part of the series on Azure Dev Spaces. The first part of this services describes steps needed to deploy Angular App and ASP.net Core Web API to AKS using Azure Dev Spaces. This article will show ease of debugging containers deployed in AKS from Visual Studio 2017. In the first part of this series, I had created a sample ASP.net Core Web API app and deployed that to AKS using Azure Dev Spaces. The steps needed to debug ASP.net Core Web API application running in AKS are

Install Visual Studio Tools for Kubernetes. Once installation completes, open 'Tools -> Extensions and Updates' in Visual Studio and update this extension for any updates.
Open SampleWebApp solution in Visual Studio. The source code can be downloaded from GitHub.
Select Debug target as Azure Dev Spaces

Add a breakpoint in Get method of UsersController which we are going to debug
Start Debugging and select AKS cluster and Space in Azure Dev Spaces prompt
You can get public URL from Output window by selecting Azure Dev Spaces as displayed below
Open https://{URL}/api/users in browser and request will hit the breakpoint added previously

You can check the status of URL by running command azds list-uris. If URL isn't available for more than 5 minutes you will need to delete PODS as explained in previous article. This completes this article which showed how easy it is to debug containers deployed in AKS from Visual Studio 2017 using Azure Dev Spaces.

Azure Dev Spaces - Deploying Angular and ASP.net Core App to Azure Kubernetes Service (AKS)

Atul.Verma — Tue, 18 Sep 2018 06:07:57 GMT

This article will show the ease of running and deploying containers in Azure Kubernetes Service (AKS) with minimal dev machine setup with Azure Dev Spaces. I am going to reuse sample application components from my previous article on AKS. With Azure Dev Spaces, you can develop on Windows, Mac, or Linux using familiar tools like Visual Studio, Visual Studio Code, or the command line. I have chosen Visual Studio Code on Mac. In this article, I will share the steps needed to deploy sample application I created in Angular and ASP.net Core Web API using Azure Dev Spaces. At time of writing this article Azure Dev Spaces is in preview and is isn't available in all regions.

Setting up Azure Dev Spaces

If you don't have Azure Kubernetes Service (AKS) cluster running, you can go through this article to Create AKS Cluster. Once AKS cluster is ready, the following commands are needed to setup Azure Dev Spaces

For Visual Studio code, I had to install Azure Dev Spaces extension
Run this Command to setup Azure Dev Spaces: az aks use-dev-spaces -g {ResourceGroup} -n {AKSClusterName}
- Update {ResourceGroup} and {AKSClusterName}
- Confirmation and admin privileges will be needed to proceed with installation
- Confirmation to create an Azure Dev Spaces Controller
- Select a dev space or Kubernetes namespace to use as a dev space
- Confirmation to select Kubernetes namespace 'default' as a dev space

This command is going to take a while. After completion if you browse to Azure Portal, Azure Dev Spaces Controller resource will be created. You can go through this article to Setup Azure Dev Spaces for detailed information. Similar to previous article, this sample application consists of Angular app which loads data from ASP.net Core WebAPI and Web API loads data from SQL Server DB.

SQL Server

This article assumes that SQL Server DB is already deployed to AKS. You can go through steps listed in previous article to setup SQL Server. The source code link of yaml snippets needed to deploy SQL Server can be downloaded from GitHub. After creating all the resources needed to configure SQL Server, the next step is to setup ASP.net Core Web API.

ASP.net Core Web API

You can view source code for this solution at GitHub. Please update DatabaseConnectionString setting in 'appsettings.json' file with the SQL Server database connection string based on your 'sa' user password. This Web.API uses EntityFrameworkCore to create database and seed sample data. There is only one method in the Controller to get list of users from database to keep things simple. As you can see, I haven't added DockerFile or scripts needed to deploy web api to AKS yet. The next step is show the magic of Azure Dev Spaces extension which I had installed for VS Code. The steps are

Open SampleWebApp project folder in Terminal (Right click and select Open in Terminal is using VS Code)
Run Command azds prep --public. This command is going to add files needed to deploy this resource to Kubernetes.
Run Command azds up to start deploying this Web API to AKS. This is going to take a while and steps involved are
- Synchronizing Files
- Building Helm Chart
- Building Container Image
- Deploying Image to AKS

As you can see below, ASP.net Web API is up and running and main pointers are

Azure Dev Spaces will create two URLs one for ingress and other as tunneled as you can see in first two lines of this image. The 'http' ingress URL is going to take a while and if it takes more than 5 minutes which happened in my case too, you will need to delete the pods by running commands kubectl delete pod -n kube-system -l app=addon-http-application-routing-external-dns and kubectl delete pod -n kube-system -l app=addon-http-application-routing-nginx-ingress. Please read Troubleshooting for common problems.
The Command to check list of Uris and Status is azds list-uris
Browse the URL to see list of users i.e. https://{YOUR_HTTP_URL}/api/users
The pod name is also displayed and you can run this command to view details kubectl exec -ti {POD_NAME} bash
As you can see in image captured below, you can see info messages which provides further insights
Disclaimer: I tried to debug which is another cool feature however I ran into weird issue in VS Code. You can read futher about debugging at Initialize debug assets with the VS Code extension.

The next component is Angular App. Please take a note of the URL of this Web API as it is needed to be updated in Angular App.

Angular App

This is the final component and before we proceed, we need to update external endpoint value of the Web API Service in Angular App. You can view the source code of Angular App at GitHub. You will need to update environment.ts and environment.prod.ts to update API_URL

export const environment = {
ApplicationConfig: {
'API_URL': 'https://{YOUR_WEB_API_ENDPOINT}/api/'
}
};

As you can see, I haven't added DockerFile or scripts needed to deploy web api to Angular App yet. The differences which this Angular app code has from previous article's Angular app are

addition of 'server.js' file so that this Angular App can be hosted in nodejs without need for nginx. In 'package.json', I have updated "start": "node server.js". The 'server.js' code snippet is
var express = require('express'); var app = express(); app.use(express.static(__dirname + '/dist/sampleangularapp')); app.get('/', function (req, res) { res.sendFile(__dirname + '/dist/sampleangularapp/index.html'); }); app.get('/api', function (req, res) { res.send(__dirname); }); var port = process.env.PORT || 80; var server = app.listen(port, function () { console.log('Listening on port ' + port); });

addition of RUN npm run build -- --prod command after auto-generating Dockerfile (created by running azds prep --public command) since I have used '/dist/sampleangularapp' path in 'server.js'.

Subsequent steps needed to deploy Angular App to AKS using Azure Dev Spaces are similar to ASP.net Web API i.e.

Open sampleangularapp project folder in Terminal (Right click and select Open in Terminal is using VS Code)
Run Command azds prep --public. This command is going to add files needed to deploy this resource to Kubernetes
Run Command azds up to start deploying this Web API to AKS. This is going to take a while.
Open the Http URL once ingress endpoint status is Available.

You can see the Azure Dev Spaces has created ingresses(samplewebapp and sampleangularapp) by opening AKS Dashboard. If you don't want to create ingresses you can set ingress: enabled: false in helm charts values.yaml.

Summary

In this article I have provided insights about Azure Dev Spaces(still in preview). The source code for this complete article can be downloaded from GitHub.