This is a minor update I have been meaning to post for quite a few weeks now.

Series Decompose Forecast has been introduced and in my mind it makes the trend for disk (and other data points) easier to code and to view.

// Dec 2018 option, using Series_decompose_forecast

//

// Project disk capacity forward nn days

let myComputer = "DC02";

let myDriveLetter = "C:";

let startDate = startofday(ago(60d)); // go back in time nn days

let endDate = now(); // what is the date now

let projectTo = now()+30d; // project forward nn days

let projectForward = 30; // must be same as projectTo value

Perf

| where TimeGenerated between (startDate .. endDate )

| where Computer startswith myComputer

| where CounterName == "% Free Space" and InstanceName has myDriveLetter

| make-series DiskFreeSpace = avg(CounterValue) default=0 on TimeGenerated in range(startDate, projectTo, 1h)

| extend DiskFreeSpaceForecast = series_decompose_forecast(DiskFreeSpace, projectForward*24)

| render timechart

You can see, that the blue line is the past 60days (drops to zero for today) and the green line is the prediction from today onwards to 30days.  You can edit the options for how many days to go back and how many to predict forward in the LET statements. 

]]> https://docs.microsoft.com/archive/blogs/ukhybridcloud/azure-log-analytics-service-level-agreement-part-2 Wed, 21 Nov 2018 16:08:41 GMT https://blogs.msdn.microsoft.com/ukhybridcloud/?p=4095

My post yesterday got a little long so, I’ve done a part2 to show how else you might group your solutions.

Part1: https://blogs.msdn.microsoft.com/ukhybridcloud/2018/11/20/azure-log-analytics-service-level-agreement-sla-part-1/

Here is the new query, I kept it simple so removed the JOIN and links to Perf and Memory that were shown in Part 1.

let start_time =startofday(ago(7d));

let end_time =startofday(now());

// Lets define our groups of computers, using various search techniques

let webServers = (Heartbeat

| where Computer startswith "WEB"

| distinct Computer

| project Computer

);

let domainControllers = (Event

| where EventLog == "Directory Service"

| distinct Computer

| project Computer

);

let aksServers = (Heartbeat

| where Computer has "aks"

| distinct Computer

| project Computer

);

let subnetServers = (WireData

| where LocalIP matches regex "10.118.51."

| where Computer !startswith "D"

| distinct Computer

| project Computer

);

Heartbeat

| where TimeGenerated > start_time and TimeGenerated < end_time

// import generated computer list(s)

| where Computer in (webServers, domainControllers, aksServers, subnetServers)

// Main calculation

| summarize heartbeat_per_hour=count() by bin_at(TimeGenerated, 1h, start_time), Computer

| extend available_per_hour=iff(heartbeat_per_hour>0, true, false)

| summarize total_available_hours=countif(available_per_hour==true) by Computer

| extend total_number_of_buckets=round((end_time-start_time)/1h)

| extend availability_rate=total_available_hours*100/total_number_of_buckets

// Create some readable groupnames

| extend GroupName =

case(

Computer in (webServers), GroupName ="Web Servers",

Computer in (domainControllers),GroupName ="Domain Controllers",

Computer in (aksServers), GroupName ="AKS servers",

Computer in (subnetServers), GroupName ="Subnet51",

"Unknown"

)

// Show the results

| summarize

ServerCount=dcount(Computer),

Availability=avg(availability_rate),

AvailabilityFailedOn=countif(availability_rate < 100)

by GroupName

Essentially this query looks at 4 groups of servers:

1. Web Servers

2. AKS Servers (same as in part1)

3. Domain Controllers

4. A subnet

Web Servers and AKS,  I used the same technique as I did in part1.  However there is a subtle use of the ‘has’ for the AKS servers to pick up the text “aks” anywhere in the computer name.

Domain Controllers, with this group I used the Event table, and looked for the “Directory Service” log which should be on each DC.

Subnet, this uses a regex to look for a specific subnet, I also excluded any Computers that had a name that started with “D”.

These are few techniques you can use to group computers or services together.

I also wanted a friendly name for each service, so used the Case function for that:

| extend GroupName =

case(

Computer in (webServers), GroupName ="Web Servers",

Computer in (domainControllers),GroupName ="Domain Controllers",

Computer in (aksServers), GroupName ="AKS servers",

Computer in (subnetServers), GroupName ="Subnet51",

"Unknown"

)

Basically what was required was an SLA view of a service, where the service was made up of a few computers and multiple SLA thresholds were needed to be checked.

Credit to my colleague Christoph for a lot of the code used in this SLA example, his blog is here:https://blog.peterschen.de/

The first part of the SLA solution was looking at the Heartbeat info

Let start_time=startofday(ago(30d));
let end_time=startofday(now());

Heartbeat

| where TimeGenerated > start_time and TimeGenerated < end_time

| summarize heartbeat_per_hour=count() by bin_at(TimeGenerated, 1h, start_time), Computer

| extend available_per_hour=iff(heartbeat_per_hour>0, true, false)

| summarize total_available_hours=countif(available_per_hour==true) by Computer

| extend total_number_of_buckets=round((end_time-start_time)/1h)

| extend availability_rate=total_available_hours*100/total_number_of_buckets

| order by availability_rate desc

Basically this query looks back 30days, and checks the Heartbeat data for any missing info and calculates a % availability from it. for all Computers.

Please note using Heartbeat is only an SLA indicator, there is a potential for the server to still be up and working correctly but the Heartbeat not working, likewise Heartbeat can be sending data when the server workload might not be working correctly.

Requirements

In the case being discussed we had some other requirements to meet:

1. To show CPU data in the SLA

2. To show Memory data in the SLA

3. To alert on specific thresholds for CPU, Memory and availability

4. Only monitor a specific set of servers that made up a “service”

5. To also show an overall SLA health – good or bad

The final query ended up like this, I’ll break it down into sections afterwards:

let start_time=startofday(ago(30d));

let end_time=startofday(now());

// Add my SLA values

let serviceName = "AKScluster";

let AVAILsla = 90;

let CPUsla = 50;

let MEMsla = 90;

// get server list from heartbeat

let hbav=Heartbeat

// find my 3 servers for my service, they happen to start with "AKS"

| where Computer startswith "aks"

| where TimeGenerated > start_time and TimeGenerated < end_time

| summarize heartbeat_per_hour=count() by bin_at(TimeGenerated, 1h, start_time), Computer

| extend available_per_hour=iff(heartbeat_per_hour>0, true, false)

| summarize total_available_hours=countif(available_per_hour==true) by Computer

| extend total_number_of_buckets=round((end_time-start_time)/1h)

| extend availability_rate=total_available_hours*100/total_number_of_buckets

| order by availability_rate desc;

// CPU details

let cpuutil=Perf

| where TimeGenerated > start_time and TimeGenerated < end_time

| where ObjectName == "Processor"

| where CounterName == "% Processor Time"

| where InstanceName == "_Total"

| summarize CpuUtilization=avg(CounterValue) by Computer;

//MEMORY details

let memutil=Perf

| where TimeGenerated > start_time and TimeGenerated < end_time

| where ObjectName == "Memory"

| where CounterName == "% Used Memory"

| summarize MemUtilization=avg(CounterValue) by Computer;

hbav

| join kind= inner (cpuutil) on Computer

| join kind= inner (memutil) on Computer

// show the status of my service

| summarize

Name = serviceName,

avg(availability_rate),

availabilitySLA = iif(avg(availability_rate) < AVAILsla, "Bad","Good"),

avg(CpuUtilization),

cpuSLA = iif(avg(CpuUtilization) > CPUsla, "Bad","Good"),

avg(MemUtilization),

memSLA = iif(avg(MemUtilization) > MEMsla, "Bad","Good"),

ComputerList = makeset(Computer),

dcount(Computer)

How did we achieve the SLA info

At the top of the script I added some variables for the ServiceName and the defined SLA thresholds.

In this case we wanted alerts to show when:

• The overall SLA was below 90%,
• average CPU usage was above 50%
• average Memory was above 90%.

I had 3 servers in an AKS cluster that I used, but you could amend the line to filter on your own service and list of computers.  I used startswith “AKS”, please adapt for your own naming convention.

// Add my SLA values

let serviceName = "AKScluster";

let AVAILsla = 90;

let CPUsla = 50;

let MEMsla = 90;

// get server list from heartbeat

let hbav=Heartbeat

// find my 3 servers for my service, they happen to start with "AKS"

| where Computer startswith "aks"

We then added two sections to get the CPU and Memory info from the Perf Log Analytics table and Join those to theHeartbeat table.
Here are the lines for CPU, you can see it uses the same timespan set earlier and can easily be changed for any other relevant Perf counter.  The Memory info was a copy of this with a few changed values.

// CPU details

let cpuutil=Perf

| where TimeGenerated > start_time and TimeGenerated < end_time

| where ObjectName == "Processor"

| where CounterName == "% Processor Time"

| where InstanceName == "_Total"

| summarize CpuUtilization=avg(CounterValue) by Computer;

The JOIN operator was used to link the data with the Heartbeat – again an amended line was repeated for memory join.

| join kind= inner (cpuutil) on Computer

All we needed then, was to display the data.  You can do this in many ways but I chose the Summarize operator and some IIF functions.

| summarize

Name = serviceName,

avg(availability_rate),

availabilitySLA = iif(avg(availability_rate) < AVAILsla, "Bad","Good"),

avg(CpuUtilization),

cpuSLA = iif(avg(CpuUtilization) > CPUsla, "Bad","Good"),

avg(MemUtilization),

memSLA = iif(avg(MemUtilization) > MEMsla, "Bad","Good"),

ComputerList = makeset(Computer),

dcount(Computer)

In this example I used words to denote when an SLA wasn’t reached i.e. Good or Bad.  If you plan on raising an Alert from this you could change these to a 0 or 1 to enable an alert to trigger on a threshold change.

So in the above example I got the value of the SLA and used IIF to display the words “Bad” or “Good” depending on the value compared to the variables we defined at the beginning on the script with the LET operators.

Note: there is also a count of the computers and a list of the computer names , but I haven't shown that in the above screen clip, to keep it readable on the page

For the final requirement of an overall SLA health

| extend SLAobjectiveMet = iif( availabilitySLA == "Good" and cpuSLA == "Good" and memSLA == "Good" , "SLA ok","SLA Not Ok")

| project serviceName, SLAobjectiveMet

I used an another IIF to link all three results, so if all 3 individual SLAs were ‘good’ - then the overall SLA was ‘SLA ok’.
Again you could use a 0 and 1 for generating Azure Alerts.

**** this post was restored, as the original was deleted by mistake ****

In my last post I wrote about Print and Let - which reminded me of some work I did to manipulate some text recently (it also used a Print operator, which was the thing that reminded me).

The raw data was this (it was from an event log but I don't recall which one):

"Device=\\.\\PHYSICALDRIVE; FirmwareRevision=AAA3; Interface=IDE; MediaType=\\HDD\\; Model=123456; SerialNumber=987654321"

So using Print I was able to manipulate the text, each part was needed in a column and some of the data was also needed to be formatted to removed unwanted text or characters.  Ultimately this isn't a finished query rather an example of a few techniques that you may find useful to manipulate text.

Step1 – I used Print as per the last post, then on the next line create an array using the ";" as a character to split the line at:

print txt = "Device=\\.\\PHYSICALDRIVE; FirmwareRevision=AAA3; Interface=IDE; MediaType=\\HDD\\; Model=123456; SerialNumber=987654321"

| extend txtArray = split(txt, ";")                          

As you can see by the above output we now have 6 rows of data returned in the array.

I wanted the data in a column so I used an Extend, which does that, adding these next two lines as an example

| extend DeviceID = txtArray [0]

, Firmware = txtArray [1]

Again this gets us the information, but now we need to tidy it up a little.

I first added an extra line

Interface = split(txtArray [2], "=",1)

This line splits the 2nd item [2] in the array, at the “=” character and then keeps only the txt after that character; change the 1 to 0 if you want the opposite.  You can see below, the column is now named Interface and the result is “IDE”.

The next line of syntax I added is the most complex

MediaType = trim(@"[^\w]+", tostring(split(txtArray [3],"=",1)))

We are doing a few things to the data here.

1. Using Trim and some regex to remove any non word characters from the beginning or end of the array item.  See Trim for help on that.

2. Then I used Split again to get just the text after the “=”, and this time I added tostring as well.  So txtArray[3] went from \\HDD\\ to HDD in the MediaType column:

So the full script now looks like:

print txt = "Device=\\.\\PHYSICALDRIVE; FirmwareRevision=AAA3; Interface=IDE; MediaType=\\HDD\\; Model=123456; SerialNumber=987654321"

| extend txtArray = split(txt, ";") // split on ";" char

| extend DeviceID = txtArray [0]

, Firmware = txtArray [1]

, Interface = split(txtArray [2], "=",1) // like the line above but use split to show just the result

, MediaType = trim(@"[^\w]+", tostring(split(txtArray [3],"=",1))) // again, split on "=" and then trim all non word chars

, Model = txtArray [4]

// etc...

| project DeviceID , Firmware , Interface , MediaType, Model

So in the above example Column 1 and 4 are essentially unformatted, but columns 2 and 3 have two different formats applied to make them more readable.

Log Analytics shows Usage in two main places

1. Azure Monitor

2.  Log Analytics

1. On the left we can see the break down between Ingestion (data sent to Azure) and retention (if above the 31days default).
2. The top right chart shows the daily ingestion, useful to spot if we have a dramatic rise on a specific day.
3. The last report breaks the costs down by solution, what are the highest ones?
4. Then in the top left corner – you can get more details, press “Usage Details” – from the next screen I immediately toggle to 30days to show a month.
   1. We’d then look at the Data volume over time and Data by Solution details.
   2. The Volume over time is good to spot a few servers that are sending lots of data – like my vCenter one (16GB) in the below example. The entry “-“ is also of interest as that is PaaS or SaaS data.

Planning

Use the Print operator to create some

Use the Let statement with datatable

Event
| where EventID > 7600
| count

let dummyData = datatable(dummyTxt:string)
[
    "This is yet another test"
];
dummyData
| where dummyTxt has "yet" 

let dummyData = datatable(Date:datetime, dummyTxt:string)
[
    datetime(now) , "This is yet another test" ,
    datetime(now) , "This is yet another test string" 
];
dummyData
| where dummyTxt == "This is yet another test string" 

 

Create a new VSTS team project

Create a new build pipeline