gRCP Health Check with Powershell

Question

Hello.
I am checking periodically gRPC health of several microservices, every two seconds, it is working since I got the response they are serving, the output is { "status": "SERVING" }

 while ($true) {grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key" host6:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key" host7:port grpc.health.v1.Health/Check;start-sleep 2}

Now I need a notification if the output is not "SERVING" while the code keeps running in a loop, how can I achieve this?

Please note, one or more service can fail but other can be ok so notification should only about the ones are not answering "SERVING", thanks,

Answer 1

There are ways to make this more efficient, and to also handle those times when grpcurl doesn't return anything in STDOUT (e.g. the host cannot be contacted, authentication fails, etc.) but writes an error message to STDERR, but I'll leave that part up to you,

Give this one a try:

function Serving{
    param(
        $val
    )
    if ($val -isnot [array]){
        return $false
    }
    if ($val.count -ne 3){
        return $false
    }
    Try{
        $p = $val -join "" | ConvertFrom-Json -ErrorAction STOP
        if ($p.status -eq 'serving'){
            $true
        }
        else{
            $false
        }
    }
    Catch{
        Write-Host "'$p' is not JSON"
        $false
    }
}
while ($true) {
    $beep = $false
    [array]$response = grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service1 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service2 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service3 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
    if (-Not (Serving $response)){
        Write-Host "Service4 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service5 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service6 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
    if (-Not (Serving $response)){
        Write-Host "Service7 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    if ($beep){
        [console]::beep(500, 300)
    }
    Start-Sleep 2 
}

Answer 2

Still not working this way.
Each grpcurl excution create this output:

host1: {
host2: "status": "SERVING"
host3: }

Answer 3

Lets use the last code you shared with curl instead, so we can both test it, goal is if the ouput is not 200 OK then send an email(only for the failing curls) and keep checking in a loop. Also I changed the beep to send email ,which works better for the alarming purpose..

 $responses = @()
while ($true) {
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca4.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.goofle.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.googrrle.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.es 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses){
         $r = $response.Trim()
         $ctr++
         if ($r -ne '{ "status": "200" }'){
             $bad += $ctr
         }      
     }
    if ($bad.count -gt 0){
         #we  define the mail object here
$Message = new-object Net.Mail.MailMessage
$Message.Subject = "gRPC Health Microservices Sandbox Notification"
$smtp = new-object Net.Mail.SmtpClient("smtp.gmail.com", 587)
#we declare our email credentials here, which includes our email and the password to the email
$smtp.Credentials = New-Object System.Net.NetworkCredential("xxxx@xxx.net", "xxyyxyx");
$smtp.EnableSsl = $true
#we specify our email here, whereby it's coming from us 
$Message.From = "xxxx@xxx.net"
#we specify who we want to send the message to
$Message.To.Add("xxxx@xxx.net")
#we add the body of our message here
$Message.Body = "response is not SERVING. Check this service!"
#then we send it with the smtp object
$smtp.Send($Message)
         $bad |
             ForEach-Object{
                 Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
             }

     }
     $responses = @()
     $bad = @()
     Start-Sleep 2 
 }

Answer 4

I am getting emails for each execution and the idea is get an email only if is not SERVING and what are the service or services which are not SERVING (in the curl code example if is not 200 )

Answer 5

I removed the e-mail from the code. Also, the test on line 14 of your example using curl is incorrect -- curl only returns a numeric status, not a JSON string.

$responses = @()
while ($true) {
    $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca.com
    $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca4.com
    $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.goofle.com
    $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.googrrle.com
    $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.com
    $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.es 
    $ctr = 0
    [array]$bad = @()
    ForEach ($response in $responses) {
        $r = $response.Trim()
        $ctr++
        if ($r -ne '200') {
            $bad += $ctr
        }      
    }
    if ($bad.count -gt 0) {
        $bad |
            ForEach-Object {
                Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
            }

    }
    $responses = @()
    $bad = @()
    Start-Sleep -s 5 
}

That code returns this:

PS C:\Users\richm> . "c:\Junk\Untitled-22.ps1"
host2: 000
host4: 000
host5: 000
host2: 000
host4: 000
host5: 000
etc.
etc.
etc.

Only the non-200 status values are flagged.