Adding WAD Diagnostic extension to Managed Service Fabric

Question

We have a Managed ServiceFabric cluster we just deployed for a new application. Unfortunately, we did not see any options to deploy the Diagnostic Extension to the Scale Sets in that new Managed Service Fabric setup. Which leads us to this week, where we have been unsuccessfully trying to add WAD, the Diagnostic Extension to the Scale Sets to gather data for Application Insights. I understand that Fabric Observer/Clluster Observer is a Git app that gives us additional information on SF. I have built and managed to install the app to managed SF. But our attempts to get the diagnostic extension installed (I imagine it gives us a different set of data than FO does, correct me if I am wrong) has been a frustrating lack of progress. Has anyone successfully installed the Diagnostic Extension to a managed SF or to the older SF? I'd appreciate any tips or insight. Below is the script I wrote to install the extension to the VMSS nodes, but it keeps failing trying to read the config.

$Settings = ' {  
    "WadCfg": {  
        "DiagnosticMonitorConfiguration": {  
            "overallQuotaInMB": "50000",  
            "SinksConfig" : {   
                "Sink" : {  
                    "name" : "AISink",  
                    "ApplicationInsights" : "f1230e9f-0e43-4812-832d-XXXXXXXXXXXX"  
                }  
              
            },  
            "PerformanceCounters": {  
                "scheduledTransferPeriod": "PT1M",  
                "sinks": "AzureMonitorSink",  
                "PerformanceCounterConfiguration": [  
                    {  
                        "counterSpecifier": "\\Processor(_Total)\\% Processor Time",  
                        "sampleRate": "PT1M",  
                        "unit": "percent"  
                    }  
                ]  
            },  
            "Directories": {  
                "scheduledTransferPeriod": "PT5M",  
                "IISLogs": {  
                    "containerName": "iislogs"  
                },  
                "FailedRequestLogs": {  
                    "containerName": "iisfailed"  
                }  
            },  
            "EtwProviders": {  
                "EtwEventSourceProviderConfiguration": [  
                    {  
                        "provider": "Microsoft-ServiceFabric-Actors",  
                        "scheduledTransferKeywordFilter": "1",  
                        "scheduledTransferPeriod": "PT5M",  
                        "DefaultEvents": {  
                            "eventDestination": "ServiceFabricReliableActorEventTable"  
                        }  
                    },  
                    {  
                        "provider": "Microsoft-ServiceFabric-Services",  
                        "scheduledTransferPeriod": "PT5M",  
                        "DefaultEvents": {  
                            "eventDestination": "ServiceFabricReliableServiceEventTable"  
                        }  
                    }  
                ],  
                "EtwManifestProviderConfiguration": [  
                    {  
                        "provider": "cbd93bc2-71e5-4566-b3a7-XXXXXXXXXXXX",  
                        "scheduledTransferLogLevelFilter": "Information",  
                        "scheduledTransferKeywordFilter": "4611686018427387928",  
                        "scheduledTransferPeriod": "PT5M",  
                        "DefaultEvents": {  
                            "eventDestination": "ServiceFabricSystemEventTable"  
                        }  
                    },  
                    {  
                        "provider": "02d06793-efeb-48c8-8f7f-XXXXXXXXXXXX",  
                        "scheduledTransferLogLevelFilter": "Information",  
                        "scheduledTransferKeywordFilter": "4611686018427387928",  
                        "scheduledTransferPeriod": "PT5M",  
                        "DefaultEvents": {  
                            "eventDestination": "ServiceFabricSystemEventTable"  
                        }  
                    }  
                ],  
                "WindowsEventLog": {  
                    "scheduledTransferPeriod": "PT5M",  
                    "DataSource": [  
                        {  
                            "name": "System!*[System[Provider[@Name:''Microsoft Antimalware'']]]"  
                        },  
                        {  
                            "name": "System!*[System[Provider[@Name:''NTFS''] and (EventID:55)]]"  
                        },  
                        {  
                            "name": "System!*[System[Provider[@Name:''disk''] and (EventID:7 or EventID:52 or EventID:55)]]"  
                        }  
                    ]  
                },  
                "Logs": {  
                    "scheduledTransferPeriod": "PT1M",  
                    "scheduledTransferLogLevelFilter": "Verbose",  
                    "sinks": "ApplicationInsights.AppLogs"  
                }  
      
            }  
        }  
    }  
}'  
  
$rgName = "XXXXXX-px-msfc-usnc1-rg"  
$MsfcResourceGroup = "SFC_b05c8d5c-1a49-4fab-aed0-XXXXXXXXXXXX"  
$clusterName = "XXXXXX-px-msfc"  
$Publisher = "Microsoft.Azure.Diagnostics"  
$ExtName = "IaaSDiagnostics"  
$ExtType = "IaaSDiagnostics"  
$TypeExtVerson = "1.5"  
  
$NodeTypeName1 = "app1"  
$NodeTypeName2 = "seed1"  
$NodeTypeName3 = "proxy1"  
  
Set-AzContext -Subscription "a57c1fc6-297f-47bf-a83f-XXXXXXXXXXXX"  
  
Remove-AzServiceFabricManagedNodeTypeVMExtension -ResourceGroupName $rgName -ClusterName $clusterName -NodeTypeName $NodeTypeName1 -Name ("$ExtName-$NodeTypeName1")  
Update-AzVmss -ResourceGroupName $MsfcResourceGroup -Name $NodeTypeName1 -IdentityType "SystemAssigned"  
Add-AzServiceFabricManagedNodeTypeVMExtension -ResourceGroupName $rgName -TypeHandlerVersion $TypeExtVerson -ClusterName $clusterName -NodeTypeName $NodeTypeName1 -Name ("$ExtName-$NodeTypeName1") -Publisher $Publisher -Type $ExtType -AutoUpgradeMinorVersion -Verbose -setting $Settings  
  
Remove-AzServiceFabricManagedNodeTypeVMExtension -ResourceGroupName $rgName -ClusterName $clusterName -NodeTypeName $NodeTypeName2 -Name ("$ExtName-$NodeTypeName2")   
Update-AzVmss -ResourceGroupName $MsfcResourceGroup -Name $NodeTypeName2 -IdentityType "SystemAssigned"  
Add-AzServiceFabricManagedNodeTypeVMExtension -ResourceGroupName $rgName -TypeHandlerVersion $TypeExtVerson -ClusterName $clusterName -NodeTypeName $NodeTypeName2 -Name ("$ExtName-$NodeTypeName2") -Publisher $Publisher -Type $ExtType -AutoUpgradeMinorVersion -Verbose -setting $Settings  
  
Remove-AzServiceFabricManagedNodeTypeVMExtension -ResourceGroupName $rgName -ClusterName $clusterName -NodeTypeName $NodeTypeName3 -Name ("$ExtName-$NodeTypeName3")  
Update-AzVmss -ResourceGroupName $MsfcResourceGroup -Name $NodeTypeName3 -IdentityType "SystemAssigned"  
Add-AzServiceFabricManagedNodeTypeVMExtension -ResourceGroupName $rgName -TypeHandlerVersion $TypeExtVerson -ClusterName $clusterName -NodeTypeName $NodeTypeName3 -Name ("$ExtName-$NodeTypeName3")-Publisher $Publisher -Type $ExtType -AutoUpgradeMinorVersion -Verbose -setting $Settings  

Additionally, I have tried a hashtable for the settings:

    @{  "WadCfg":   
        @{"DiagnosticMonitorConfiguration":   
            @{ "overallQuotaInMB": "50000"}  
        }  
      }  

And various simplified JSON data versions and using “convert-fromjson”.
We even pulled a config from another unmanaged SF that has Diagnostics installed to no avail:

    $Settings = '{  
        "WadCfg": {  
          "DiagnosticMonitorConfiguration": {  
            "overallQuotaInMB": "50000",  
            "EtwProviders": {  
              "EtwEventSourceProviderConfiguration": [  
                {  
                  "provider": "Microsoft-ServiceFabric-Actors",  
                  "scheduledTransferKeywordFilter": "1",  
                  "scheduledTransferPeriod": "PT5M",  
                  "DefaultEvents": {  
                    "eventDestination": "ServiceFabricReliableActorEventTable"  
                  }  
                },  
                {  
                  "provider": "Microsoft-ServiceFabric-Services",  
                  "scheduledTransferPeriod": "PT5M",  
                  "DefaultEvents": {  
                    "eventDestination": "ServiceFabricReliableServiceEventTable"  
                  }  
                }  
              ],  
              "EtwManifestProviderConfiguration": [  
                {  
                  "provider": "cbd93bc2-71e5-4566-b3a7-XXXXXXXXXXXX",  
                  "scheduledTransferLogLevelFilter": "Information",  
                  "scheduledTransferKeywordFilter": "4611686018427387928",  
                  "scheduledTransferPeriod": "PT5M",  
                  "DefaultEvents": {  
                    "eventDestination": "ServiceFabricSystemEventTable"  
                  }  
                },  
                {  
                  "provider": "02d06793-efeb-48c8-8f7f-XXXXXXXXXXXX",  
                  "scheduledTransferLogLevelFilter": "Information",  
                  "scheduledTransferKeywordFilter": "4611686018427387928",  
                  "scheduledTransferPeriod": "PT5M",  
                  "DefaultEvents": {  
                    "eventDestination": "ServiceFabricSystemEventTable"  
                  }  
                }  
              ]  
            }  
          }  
        }  
      }'  

This is the error the Add returns (at least this week, last week it was behaving differently):

PS C:\Dev\service-fabric-observer> Add-AzServiceFabricManagedNodeTypeVMExtension -ResourceGroupName $rgName -TypeHandlerVersion $TypeExtVerson -ClusterName $clusterName -NodeTypeName $NodeTypeName1 -Name ("$ExtName-$NodeTypeName1") -Publisher $Publisher -Type $ExtType -AutoUpgradeMinorVersion -Verbose -setting $Settings

VERBOSE: Performing the operation "Add Extenions IaaSDiagnostics-app1 with type IaaSDiagnostics to node type app1" on target "IaaSDiagnostics-app1".

Add-AzServiceFabricManagedNodeTypeVMExtension: Code: UnhandledException, Message: Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 14.

Add-AzServiceFabricManagedNodeTypeVMExtension: {"error":{"code":"UnhandledException","message":"Error reading JObject from JsonReader. Current JsonReader item is not an object: String. Path '', line 1, position 14."}}
Add-AzServiceFabricManagedNodeTypeVMExtension: Operation returned an invalid status code 'InternalServerError'