How to retrive APIM eventhub log from policy : log-to-eventhub
Hi, we've followed this post to add request/response log to eventhub then capture to storage account, it works.
Related content can be found in *.avro from storage account, but the format is not user-friendly so we try to leverage existing Apache Nifi to directly read data from eventhub.
It's weird that we can read Gatewaylogs, metrics by Nifi , but no APIM request/response in it, so we try to inspect log analytics.
Since in policy setting we log request in partition 0 and response in partition 1, we use following script to check, but nothing in 0 and 1 partition.
SELECT
*
INTO
[OutputAlias]
FROM
[evh-xxx]
WHERE PartitionId = 1
Really confused how the request/response log captured in storage account, we cannot read them from log analytics and Nifi.
Here's the config about event-hub and its namespace
//Event-hub namespace
{
"sku": {
"name": "Standard",
"tier": "Standard",
"capacity": 5
},
"id": "/subscriptions/12345678/resourceGroups/RG_xxx/providers/Microsoft.EventHub/namespaces/evhns-xxx",
"name": "evhns-xxx",
"type": "Microsoft.EventHub/Namespaces",
"location": "West US 2",
"tags": {},
"properties": {
"isAutoInflateEnabled": true,
"maximumThroughputUnits": 20,
"kafkaEnabled": true,
"provisioningState": "Succeeded",
"metricId": "12345678-3976-49e0-b333-389a5fdb7f7b:evhns-xxx",
"createdAt": "2021-01-07T10:31:38.057Z",
"updatedAt": "2021-01-07T10:32:21.077Z",
"serviceBusEndpoint": "https://evhns-xxx.servicebus.windows.net:443/",
"status": "Active"
}
}
-------------------
//Event-hub
{
"id": "/subscriptions/12345678/resourceGroups/RG_xxx/providers/Microsoft.EventHub/namespaces/evhns-xxx/eventhubs/evh-xxx",
"name": "evh-xxx",
"type": "Microsoft.EventHub/Namespaces/EventHubs",
"location": "West US 2",
"properties": {
"messageRetentionInDays": 7,
"partitionCount": 20,
"status": "Active",
"createdAt": "2021-01-07T10:49:15.263",
"updatedAt": "2021-01-18T08:28:26.973",
"partitionIds": [
"0",
"1",
"2",
"3",
"4",
"5",
"6",
"7",
"8",
"9",
"10",
"11",
"12",
"13",
"14",
"15",
"16",
"17",
"18",
"19"
],
"captureDescription": {
"enabled": true,
"encoding": "Avro",
"destination": {
"name": "EventHubArchive.AzureBlockBlob",
"properties": {
"storageAccountResourceId": "/subscriptions/12345678/resourceGroups/RG_xxx/providers/Microsoft.Storage/storageAccounts/storagexxx",
"blobContainer": "evh-xxx",
"archiveNameFormat": "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}"
}
},
"intervalInSeconds": 300,
"sizeLimitInBytes": 314572800
}
}
}