Error on Azure Kubernetes Service on Azure Stack HCI Error Fetching Billing Information

Christopher Low Kin Siong 47 Microsoft Vendor

This is a "brand new" azure stack hci cluster
its 4 node dell emc ax 640

aks version 1.0.1.10628
azure stack hci version 10.0.17784 (should be latest non preview build)

Azure Kubernetes Service on Azure Stack HCI

Fetching billing information for this setup failed. Some dashboard functionalities like 'Add cluster' will be disabled in the meanwhile. Reload page to retry

Message
Getting billing status information failed with the error: Get billing status failed. C:\Program Files\AksHci\kvactl.exe cluster addons billing get-status --kubeconfig C:\ClusterStorage\SGLOCHCIVol01\1.0.1.10628\kubeconfig-mgmt --outputformat=json returned a non zero exit code 1 [Error: Get "https://10.36.146.111:6443/api?timeout=10s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]

An error occurred while fetching clusters. Error: C:\Program Files\AksHci\kubectl.exe --kubeconfig=C:\ClusterStorage\SGLOCHCIVol01\1.0.1.10628\kubeconfig-mgmt get akshciclusters -o json returned a non zero exit code 1 [Unable to connect to the server: dial tcp 10.36.146.111:6443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.]

---
basically, we did a demo of mars/site recovery. this broke the reporting of the cluster to azure as documented in
https://learn.microsoft.com/en-us/answers/questions/436566/azure-stack-registration-query-failed-service-prob.html

I have run the
[System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")
$publish = New-Object System.EnterpriseServices.Internal.Publish
$publish.GacInstall("c:\windows\system32\azshci\NewtonSoft.Json.dll")

which i think is what broke my aks.

any clue how to fix it. (or even reverse what I did above. ) (so that I can toggle between having a usable kubernetes and azure hci cluster blade on azure)

Christopher Low Kin Siong 47 Reputation points Microsoft Vendor

2021-08-10T17:33:31.497+00:00
Get-AksHciBillingStatus
Get billing status failed. C:\Program Files\AksHci\kvactl.exe cluster addons billing get-status --kubeconfig
C:\ClusterStorage\SGLOCHCIVol01\1.0.1.10628\kubeconfig-mgmt --outputformat=json returned a non zero exit code 1
[Error: Get "https://10.36.146.111:6443/api?timeout=10s": context deadline exceeded]

CategoryInfo : OperationStopped: (Get billing sta...dline exceeded]:String) [], RuntimeException

FullyQualifiedErrorId : Get billing status failed. C:\Program Files\AksHci\kvactl.exe cluster addons billing get
-status --kubeconfig C:\ClusterStorage\SGLOCHCIVol01\1.0.1.10628\kubeconfig-mgmt --outputformat=json returned a no
n zero exit code 1 [Error: Get "https://10.36.146.111:6443/api?timeout=10s": context deadline exceeded]

ping 10.36.146.111

Pinging 10.36.146.111 with 32 bytes of data:
Reply from 10.36.146.11: Destination host unreachable.
Reply from 10.36.146.11: Destination host unreachable.
Reply from 10.36.146.11: Destination host unreachable.
Reply from 10.36.146.11: Destination host unreachable.

Ping statistics for 10.36.146.111:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

10.36.146.111 was my vip pool start ip.

vipPoolStart "10.36.146.111

--
I'm beginning to suspect the aks cluster is borked up.
Christopher Low Kin Siong 47 Reputation points Microsoft Vendor

2021-08-10T17:53:42.28+00:00
Uninstall-AksHci -SkipConfigCleanup
IInstall-AksHci -Verbose
VERBOSE: [08/11/2021 01:51:49] [AksHci] Initializing environment
VERBOSE: [08/11/2021 01:51:49] [AksHci] Importing Configuration
VERBOSE: [08/11/2021 01:51:49] [AksHci] Importing Configuration Completed
VERBOSE: [08/11/2021 01:51:51] [AksHci] Saving Configuration for AksHci =>
C:\ClusterStorage\SGLOCHCIVol01.AksHci\psconfig.json
VERBOSE: [08/11/2021 01:52:32] [AksHci] Saving Configuration for AksHci =>
C:\ClusterStorage\SGLOCHCIVol01.AksHci\psconfig.json
GetRelease - error returned by API call: Received unexpected response code: 404
At C:\Program Files\WindowsPowerShell\Modules\DownloadSdk\1.0.0\DownloadSdk.psm1:388 char:9

throw $("$Context - error returned by API call: $errString")

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo : OperationStopped: (GetRelease - er...ponse code: 404:String) [], RuntimeException

FullyQualifiedErrorId : GetRelease - error returned by API call: Received unexpected response code: 404

2 answers

MattMcSpirit-MSFT 561 Reputation points

2021-08-10T18:59:04.237+00:00

Hey - can you confirm that the AzSHCI cluster is running normally, and still lists as registered correctly, syncing with Azure etc?

Have you made any other changes to the AzSHCI Cluster since making the MARS agent change below?

[System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")
$publish = New-Object System.EnterpriseServices.Internal.Publish
$publish.GacInstall("c:\windows\system32\azshci\NewtonSoft.Json.dll")

Thanks,
Matt
Please sign in to rate this answer.
Christopher Low Kin Siong 47 Reputation points Microsoft Vendor

2021-08-11T01:57:06.223+00:00

yes. at that point of time.

i cannot be certain the get-aksbillingstatus initially started when. (it could possible started before the mars thing).

at this point of time I'm just keen to redo the whole aks service
Uninstall-AksHci -SkipConfigCleanup
IInstall-AksHci -Verbose
VERBOSE: [08/11/2021 01:51:49] [AksHci] Initializing environment
VERBOSE: [08/11/2021 01:51:49] [AksHci] Importing Configuration
VERBOSE: [08/11/2021 01:51:49] [AksHci] Importing Configuration Completed
VERBOSE: [08/11/2021 01:51:51] [AksHci] Saving Configuration for AksHci =>
C:\ClusterStorage\SGLOCHCIVol01.AksHci\psconfig.json
VERBOSE: [08/11/2021 01:52:32] [AksHci] Saving Configuration for AksHci =>
C:\ClusterStorage\SGLOCHCIVol01.AksHci\psconfig.json
GetRelease - error returned by API call: Received unexpected response code: 404
At C:\Program Files\WindowsPowerShell\Modules\DownloadSdk\1.0.0\DownloadSdk.psm1:388 char:9

throw $("$Context - error returned by API call: $errString")

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo : OperationStopped: (GetRelease - er...ponse code: 404:String) [], RuntimeException

FullyQualifiedErrorId : GetRelease - error returned by API call: Received unexpected response code: 404

RIGHT NOW though. I'm updating the cluster to the next azure hci version 10.0.20348. I'll let you know if i can cleanly uninstall akshci then reinstall it

Christopher Low Kin Siong 47 Reputation points Microsoft Vendor

2021-08-11T09:17:37.607+00:00

so I've already updated the servers + august critical updates.
unstall

I'm getting some error with Set-AksHciRegistration
note I'm only using powershell because the last time i tried wac/gui create, it failed to do so.

TODAY, powershell fails and
10 minutes into install-akshci on the WAC GUI and it fails as well.

WAC GUI Error
"
[Install-AksHci]:The operation has timed out.

Clean up your host environment and re-start the setup process."

--
Powershell Error

[TESTAX-640-1.local.contoso.net]: PS C:\Users\mylogon\Documents> Set-AksHciRegistration -subscriptionid "mysubscriptionid" -tenantid "mytenantid" -resourcegroupname "myresourcegroup" -Region "SouthEastAsia"
Cannot bind argument to parameter 'version' because it is an empty string.

CategoryInfo : InvalidData: (:) [Set-AksHciRegistration], ParameterBindingValidationException

FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,Set-AksHciRegistration

if i key in using set-akshciregistration, it fails after i key in the resourcegroupname

PS C:\Users\dellazurehci.adm> Set-AksHciRegistration

cmdlet Set-AksHciRegistration at command pipeline position 1
Supply values for the following parameters:
SubscriptionId: xxxx
ResourceGroupName: mysubscriptioname
Set-AksHciRegistration : Cannot bind argument to parameter 'version' because it is an empty string.
At line:1 char:1

Set-AksHciRegistration
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Christopher Low Kin Siong 47 Reputation points Microsoft Vendor

2021-08-11T09:21:08.02+00:00

I think I should close this and restart. i think the issue whether previously was caused by some form of corruption. the new issue is

after Uninstall-AksHci runs on all nodes with no issue, I am still unable to create aks service either on gui or powershell.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Error on Azure Kubernetes Service on Azure Stack HCI Error Fetching Billing Information

2 answers

Your answer