gRCP Health Check with Powershell

FJS 21

Hello.
I am checking periodically gRPC health of several microservices, every two seconds, it is working since I got the response they are serving, the output is { "status": "SERVING" }

 while ($true) {grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key" host6:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key" host7:port grpc.health.v1.Health/Check;start-sleep 2}

Now I need a notification if the output is not "SERVING" while the code keeps running in a loop, how can I achieve this?

Please note, one or more service can fail but other can be ok so notification should only about the ones are not answering "SERVING", thanks,

Rich Matheisen 44,776 Reputation points

2022-05-08T18:37:15.09+00:00

I'm not familiar with grpcurl. It looks like you're running 7 instances of that tool and each (except the last) is appending it's output (STDOUT) to the following instance -- but what does that "following" instance do with that?

Is it only the output from the "host7" instance that you see ('{ status": "SERVING" }'), or do you get back a JSON array of statuses?

Oh, and what sort of notification are you looking for?
FJS 21 Reputation points

2022-05-08T18:42:18.133+00:00

Hi,
It is not appending, is running grpcurls in a secuence once per host then 2 seconds after that starts again, the idea is to monitor the health of each 7 services and get a warn once one or more are not responding "SERVING" any notification should work like a beep, popup, etc...

Accepted answer

Rich Matheisen 44,776

There are ways to make this more efficient, and to also handle those times when grpcurl doesn't return anything in STDOUT (e.g. the host cannot be contacted, authentication fails, etc.) but writes an error message to STDERR, but I'll leave that part up to you,

Give this one a try:

function Serving{
    param(
        $val
    )
    if ($val -isnot [array]){
        return $false
    }
    if ($val.count -ne 3){
        return $false
    }
    Try{
        $p = $val -join "" | ConvertFrom-Json -ErrorAction STOP
        if ($p.status -eq 'serving'){
            $true
        }
        else{
            $false
        }
    }
    Catch{
        Write-Host "'$p' is not JSON"
        $false
    }
}
while ($true) {
    $beep = $false
    [array]$response = grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service1 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service2 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service3 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
    if (-Not (Serving $response)){
        Write-Host "Service4 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service5 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check
    if (-Not (Serving $response)){
        Write-Host "Service6 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    [array]$response = grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
    if (-Not (Serving $response)){
        Write-Host "Service7 isn't serving" -ForegroundColor Yellow
        $beep = $true
    }
    if ($beep){
        [console]::beep(500, 300)
    }
    Start-Sleep 2 
}

FJS 21 Reputation points

2022-05-11T21:16:41.923+00:00

Brillant is exactly what I needed, I got some errors when some services are not SERVING also beep. Wonder if I use my email notification instead beeps work as well, I will try, Many thanks!

14 additional answers

Rich Matheisen 44,776

See if this works:

$responses = @()  
while ($true) {  
    $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check  
    $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check  
    $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check   
    $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check   
    $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check   
    $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check   
    $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check   
    $ctr = 0  
    [array]$bad = @()  
    ForEach ($response in $responses){  
        $ctr++  
        if ($response -ne '{ "status": "SERVING" }'){  
            $bad += $ctr  
        }        
    }  
    if ($bad.count -gt 0){  
        [console]::beep(500,300)  
        $bad |  
            ForEach-Object{  
                Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow  
            }  
  
    }  
    $responses = @()  
    $bad = @()  
    Start-Sleep 2   
}

I don't know what your host names are so the script will probably need some work to be usable.

FJS 21 Reputation points

2022-05-08T20:43:25.243+00:00

It works, but couple of things all are SERVING now and I get a beep at first just when the script is executed and a beep when the 7 checks are done just before start the loop again, the idea of the beep used as a warning is in case the output is not SERVING also how can I change in the Write-Host "host$ part to have the name of each service?
now I am geting this output...
host1: {
host2: "status": "SERVING"
host3: }
host4: {
host5: "status": "SERVING"
host6: }
host7: {
host8: "status": "SERVING"
host9: }
host10: {
host11: "status": "SERVING"
host12: }
host13: {
host14: "status": "SERVING"
host15: }
host16: {
host17: "status": "SERVING"
host18: }
host19: {
host20: "status": "SERVING"
host21: }
host1: {
host2: "status": "SERVING"
host3: }
host4: {
host5: "status": "SERVING"
host6: }
host7: {
host8: "status": "SERVING"
host9: }
host10: {
host11: "status": "SERVING"
host12: }
host13: {
host14: "status": "SERVING"

thanks!
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Rich Matheisen 44,776

I just phonied up a few "responses and commented out the "while" and the execution of the grpcurl tool, and changed the "sleep" time to 5 seconds.

$responses = @(  
    '{ "status": "SERVING" }'  
    '{ "status": "SERVING" }'  
    '{ "status": "is not SERVING" }'  
    '{ "status": "SERVING" }'  
    '{ "status": "is not SERVING" }'  
    '{ "status": "SERVING" }'  
)  
#while ($true) {  
#     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check  
#     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check  
#     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check   
#     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check   
#     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check   
#     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check   
#     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check   
    $ctr = 0  
    [array]$bad = @()  
    ForEach ($response in $responses){  
        $ctr++  
        if ($response -ne '{ "status": "SERVING" }'){  
            $bad += $ctr  
        }        
    }  
    if ($bad.count -gt 0){  
        [console]::beep(500,300)  
        $bad |  
            ForEach-Object{  
                Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow  
            }  
  
    }  
    $responses = @()  
    $bad = @()  
    Start-Sleep -s 5   
#}

The results were a "beep" and this:

PS C:\Users\richm> . "c:\Junk\Untitled-22.ps1"  
host3: { "status": "is not SERVING" }  
host5: { "status": "is not SERVING" }  
PS C:\Users\richm>

I don't any way to run grpcurl so I can't tell you what is being written to STDOUT. Perhaps there are multiple lines being written to STDOUT by each execution? Or each line is ended (or preceded) by a CrLf pair?

Rich Matheisen 44,776

If there are CrLf in the grpcurl responses they'll screw up the response checking.

Try this version. It removes any whitespace at the begging and end of each response from grpcurl:

while ($true) {
     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
    $ctr = 0
    [array]$bad = @()
    ForEach ($response in $responses){
        $r = $response.Trim()
        $ctr++
        if ($r -ne '{ "status": "SERVING" }'){
            $bad += $ctr
        }      
    }
    if ($bad.count -gt 0){
        [console]::beep(500,300)
        $bad |
            ForEach-Object{
                Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
            }

    }
    $responses = @()
    $bad = @()
    Start-Sleep 2 
}