question

FJS avatar image
0 Votes"
FJS asked FJS commented

gRCP Health Check with Powershell

Hello.
I am checking periodically gRPC health of several microservices, every two seconds, it is working since I got the response they are serving, the output is { "status": "SERVING" }

  while ($true) {grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key" host6:port grpc.health.v1.Health/Check>> grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key" host7:port grpc.health.v1.Health/Check;start-sleep 2}

Now I need a notification if the output is not "SERVING" while the code keeps running in a loop, how can I achieve this?

Please note, one or more service can fail but other can be ok so notification should only about the ones are not answering "SERVING", thanks,

windows-server-powershell
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I'm not familiar with grpcurl. It looks like you're running 7 instances of that tool and each (except the last) is appending it's output (STDOUT) to the following instance -- but what does that "following" instance do with that?

Is it only the output from the "host7" instance that you see ('{ status": "SERVING" }'), or do you get back a JSON array of statuses?

Oh, and what sort of notification are you looking for?

0 Votes 0 ·
FJS avatar image FJS RichMatheisen-8856 ·

Hi,
It is not appending, is running grpcurls in a secuence once per host then 2 seconds after that starts again, the idea is to monitor the health of each 7 services and get a warn once one or more are not responding "SERVING" any notification should work like a beep, popup, etc...

0 Votes 0 ·
RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered FJS commented

There are ways to make this more efficient, and to also handle those times when grpcurl doesn't return anything in STDOUT (e.g. the host cannot be contacted, authentication fails, etc.) but writes an error message to STDERR, but I'll leave that part up to you,

Give this one a try:

 function Serving{
     param(
         $val
     )
     if ($val -isnot [array]){
         return $false
     }
     if ($val.count -ne 3){
         return $false
     }
     Try{
         $p = $val -join "" | ConvertFrom-Json -ErrorAction STOP
         if ($p.status -eq 'serving'){
             $true
         }
         else{
             $false
         }
     }
     Catch{
         Write-Host "'$p' is not JSON"
         $false
     }
 }
 while ($true) {
     $beep = $false
     [array]$response = grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service1 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service2 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service3 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
     if (-Not (Serving $response)){
         Write-Host "Service4 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service5 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check
     if (-Not (Serving $response)){
         Write-Host "Service6 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     [array]$response = grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     if (-Not (Serving $response)){
         Write-Host "Service7 isn't serving" -ForegroundColor Yellow
         $beep = $true
     }
     if ($beep){
         [console]::beep(500, 300)
     }
     Start-Sleep 2 
 }
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Brillant is exactly what I needed, I got some errors when some services are not SERVING also beep. Wonder if I use my email notification instead beeps work as well, I will try, Many thanks!

0 Votes 0 ·
RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered

See if this works:

 $responses = @()
 while ($true) {
     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses){
         $ctr++
         if ($response -ne '{ "status": "SERVING" }'){
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0){
         [console]::beep(500,300)
         $bad |
             ForEach-Object{
                 Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow
             }
    
     }
     $responses = @()
     $bad = @()
     Start-Sleep 2 
 }

I don't know what your host names are so the script will probably need some work to be usable.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FJS avatar image
0 Votes"
FJS answered FJS published

It works, but couple of things all are SERVING now and I get a beep at first just when the script is executed and a beep when the 7 checks are done just before start the loop again, the idea of the beep used as a warning is in case the output is not SERVING also how can I change in the Write-Host "host$ part to have the name of each service?
now I am geting this output...
host1: {
host2: "status": "SERVING"
host3: }
host4: {
host5: "status": "SERVING"
host6: }
host7: {
host8: "status": "SERVING"
host9: }
host10: {
host11: "status": "SERVING"
host12: }
host13: {
host14: "status": "SERVING"
host15: }
host16: {
host17: "status": "SERVING"
host18: }
host19: {
host20: "status": "SERVING"
host21: }
host1: {
host2: "status": "SERVING"
host3: }
host4: {
host5: "status": "SERVING"
host6: }
host7: {
host8: "status": "SERVING"
host9: }
host10: {
host11: "status": "SERVING"
host12: }
host13: {
host14: "status": "SERVING"

thanks!

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered

I just phonied up a few "responses and commented out the "while" and the execution of the grpcurl tool, and changed the "sleep" time to 5 seconds.

 $responses = @(
     '{ "status": "SERVING" }'
     '{ "status": "SERVING" }'
     '{ "status": "is not SERVING" }'
     '{ "status": "SERVING" }'
     '{ "status": "is not SERVING" }'
     '{ "status": "SERVING" }'
 )
 #while ($true) {
 #     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
 #     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
 #     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
 #     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses){
         $ctr++
         if ($response -ne '{ "status": "SERVING" }'){
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0){
         [console]::beep(500,300)
         $bad |
             ForEach-Object{
                 Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow
             }
    
     }
     $responses = @()
     $bad = @()
     Start-Sleep -s 5 
 #}

The results were a "beep" and this:

 PS C:\Users\richm> . "c:\Junk\Untitled-22.ps1"
 host3: { "status": "is not SERVING" }
 host5: { "status": "is not SERVING" }
 PS C:\Users\richm> 

I don't any way to run grpcurl so I can't tell you what is being written to STDOUT. Perhaps there are multiple lines being written to STDOUT by each execution? Or each line is ended (or preceded) by a CrLf pair?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered RichMatheisen-8856 edited

If there are CrLf in the grpcurl responses they'll screw up the response checking.

Try this version. It removes any whitespace at the begging and end of each response from grpcurl:

 while ($true) {
      $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
      $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
      $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
      $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses){
         $r = $response.Trim()
         $ctr++
         if ($r -ne '{ "status": "SERVING" }'){
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0){
         [console]::beep(500,300)
         $bad |
             ForEach-Object{
                 Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
             }
    
     }
     $responses = @()
     $bad = @()
     Start-Sleep 2 
 }
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FJS avatar image
0 Votes"
FJS answered RichMatheisen-8856 commented

Still not working this way.
Each grpcurl excution create this output:

host1: {
host2: "status": "SERVING"
host3: }

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Add this line richt below the "ForEach..." on line 11"

 $response | format-hex

I only need to see a couple of the results. Seeing exactly what's in the $response would be enlightening. I'm beginning to suspect that what grpcurl is returning is JSON, and it's on more than one line. It probably looks like this (over 3 lines)

 {
     "status":  "SERVING"
 }
0 Votes 0 ·
FJS avatar image
0 Votes"
FJS answered

Lets use the last code you shared with curl instead, so we can both test it, goal is if the ouput is not 200 OK then send an email(only for the failing curls) and keep checking in a loop. Also I changed the beep to send email ,which works better for the alarming purpose..


  $responses = @()
 while ($true) {
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca4.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.goofle.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.googrrle.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.com
      $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.es 
      $ctr = 0
      [array]$bad = @()
      ForEach ($response in $responses){
          $r = $response.Trim()
          $ctr++
          if ($r -ne '{ "status": "200" }'){
              $bad += $ctr
          }      
      }
     if ($bad.count -gt 0){
          #we  define the mail object here
 $Message = new-object Net.Mail.MailMessage
 $Message.Subject = "gRPC Health Microservices Sandbox Notification"
 $smtp = new-object Net.Mail.SmtpClient("smtp.gmail.com", 587)
 #we declare our email credentials here, which includes our email and the password to the email
 $smtp.Credentials = New-Object System.Net.NetworkCredential("xxxx@xxx.net", "xxyyxyx");
 $smtp.EnableSsl = $true
 #we specify our email here, whereby it's coming from us 
 $Message.From = "xxxx@xxx.net"
 #we specify who we want to send the message to
 $Message.To.Add("xxxx@xxx.net")
 #we add the body of our message here
 $Message.Body = "response is not SERVING. Check this service!"
 #then we send it with the smtp object
 $smtp.Send($Message)
          $bad |
              ForEach-Object{
                  Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
              }
        
      }
      $responses = @()
      $bad = @()
      Start-Sleep 2 
  }
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FJS avatar image
0 Votes"
FJS answered FJS edited

I am getting emails for each execution and the idea is get an email only if is not SERVING and what are the service or services which are not SERVING (in the curl code example if is not 200 )

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered RichMatheisen-8856 edited

I removed the e-mail from the code. Also, the test on line 14 of your example using curl is incorrect -- curl only returns a numeric status, not a JSON string.

 $responses = @()
 while ($true) {
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.marca4.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.goofle.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.googrrle.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.com
     $responses += curl.exe -s -o /dev/null -w "%{http_code}" https://www.larazon.es 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses) {
         $r = $response.Trim()
         $ctr++
         if ($r -ne '200') {
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0) {
         $bad |
             ForEach-Object {
                 Write-Host "host$($_): $($responses[$_ - 1].Trim())" -ForegroundColor Yellow
             }
           
     }
     $responses = @()
     $bad = @()
     Start-Sleep -s 5 
 }

That code returns this:

 PS C:\Users\richm> . "c:\Junk\Untitled-22.ps1"
 host2: 000
 host4: 000
 host5: 000
 host2: 000
 host4: 000
 host5: 000
 etc.
 etc.
 etc.

Only the non-200 status values are flagged.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered

If the response from grpcurl is JSON, then this should get you the correct output:

 while ($true) {
     $responses = @()
     $responses += grpcurl -d '{ \"service1\": \"name1\" }' -H "authorization: key key"  host1:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service2\": \"name2\" }' -H "authorization: key key"  host2:port grpc.health.v1.Health/Check
     $responses += grpcurl -d '{ \"service3\": \"name3\" }' -H "authorization: key key"  host3:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service4\": \"name4\" }' -H "authorization: key key"  host4:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service5\": \"name5\" }' -H "authorization: key key"  host5:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service6\": \"name6\" }' -H "authorization: key key"  host6:port grpc.health.v1.Health/Check 
     $responses += grpcurl -d '{ \"service7\": \"name7\" }' -H "authorization: key key"  host7:port grpc.health.v1.Health/Check 
     $ctr = 0
     [array]$bad = @()
     ForEach ($response in $responses) {
         $ctr++
         $pp = $response | ConvertFrom-Json
         if ($pp.status -ne 'SERVING') {
             $bad += $ctr
         }      
     }
     if ($bad.count -gt 0) {
         [console]::beep(500, 300)
         $bad |
             ForEach-Object {
                 Write-Host "host$($_): $($responses[$_ - 1])" -ForegroundColor Yellow
             }
        
     }
     Start-Sleep 2 
 }
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.