讀取流量記錄
在本文中,您將了解如何使用 PowerShell 選擇性地讀取 Azure 網路監看員流量記錄的一些部分,而不必剖析整個記錄。 流量記錄會以區塊 Blob 的形式儲存在儲存體帳戶中。 每個記錄都是每小時產生的個別區塊 Blob,每隔幾分鐘就會以最新的資料進行更新。 使用本文中提供的指令碼,您可以從流量記錄讀取最新的資料,而不需下載整個記錄。
本文中討論的概念不限於 PowerShell,而且適用於 Azure 儲存體 API 支援的所有語言。
必要條件
具有有效訂用帳戶的 Azure 帳戶。 免費建立帳戶。
在您的機器上安裝 PowerShell。 如需詳細資訊,請參閱在 Windows、Linux 和 macOS 上安裝 PowerShell。 本文需要 Az PowerShell 模組。 如需詳細資訊,請參閱安裝 Azure PowerShell (部分機器翻譯)。 若要尋找已安裝的版本,請執行
Get-Module -ListAvailable Az
。區域中的流量記錄或更多。 如需詳細資訊,請參閱建立網路安全性群組流量記錄或建立虛擬網路流量記錄。
流量記錄的訂用帳戶和儲存體帳戶的必要 RBAC 權限。 如需詳細資訊,請參閱網路監看員 RBAC 權限。
擷取封鎖清單
以下的 PowerShell 指令碼會設定查詢網路安全性群組流量記錄 Blob 所需的變數,並列出 CloudBlockBlob 區塊 Blob 內的區塊。 更新指令碼以包含您環境的有效值,特別是 "yourSubscriptionId"、"FLOWLOGSVALIDATIONWESTCENTRALUS"、"V2VALIDATIONVM-NSG"、"yourStorageAccountName"、"ml-rg"、"000D3AF87856"、"11/11/2018 03:00"。 例如,yourSubscriptionId 應取代為您的訂用帳戶識別碼。
function Get-NSGFlowLogCloudBlockBlob {
[CmdletBinding()]
param (
[string] [Parameter(Mandatory=$true)] $subscriptionId,
[string] [Parameter(Mandatory=$true)] $NSGResourceGroupName,
[string] [Parameter(Mandatory=$true)] $NSGName,
[string] [Parameter(Mandatory=$true)] $storageAccountName,
[string] [Parameter(Mandatory=$true)] $storageAccountResourceGroup,
[string] [Parameter(Mandatory=$true)] $macAddress,
[datetime] [Parameter(Mandatory=$true)] $logTime
)
process {
# Retrieve the primary storage account key to access the network security group logs
$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroup -Name $storageAccountName).Value[0]
# Setup a new storage context to be used to query the logs
$ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
# Container name used by network security group flow logs
$ContainerName = "insights-logs-networksecuritygroupflowevent"
# Name of the blob that contains the network security group flow log
$BlobName = "resourceId=/SUBSCRIPTIONS/${subscriptionId}/RESOURCEGROUPS/${NSGResourceGroupName}/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/${NSGName}/y=$($logTime.Year)/m=$(($logTime).ToString("MM"))/d=$(($logTime).ToString("dd"))/h=$(($logTime).ToString("HH"))/m=00/macAddress=$($macAddress)/PT1H.json"
# Gets the storage blog
$Blob = Get-AzStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName
# Gets the block blog of type 'Microsoft.Azure.Storage.Blob.CloudBlob' from the storage blob
$CloudBlockBlob = [Microsoft.Azure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob
#Return the Cloud Block Blob
$CloudBlockBlob
}
}
function Get-NSGFlowLogBlockList {
[CmdletBinding()]
param (
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
)
process {
# Stores the block list in a variable from the block blob.
$blockList = $CloudBlockBlob.DownloadBlockListAsync()
# Return the Block List
$blockList
}
}
$CloudBlockBlob = Get-NSGFlowLogCloudBlockBlob -subscriptionId "yourSubscriptionId" -NSGResourceGroupName "FLOWLOGSVALIDATIONWESTCENTRALUS" -NSGName "V2VALIDATIONVM-NSG" -storageAccountName "yourStorageAccountName" -storageAccountResourceGroup "ml-rg" -macAddress "000D3AF87856" -logTime "11/11/2018 03:00"
$blockList = Get-NSGFlowLogBlockList -CloudBlockBlob $CloudBlockBlob
$blockList
變數會傳回 Blob 中的區塊清單。 每個區塊 Blob 都至少包含兩個區塊。 第一個區塊的長度為 12 個位元組,而且包含 JSON 記錄的左方括號。 另一個區塊是右括號,且長度為 2 個位元組。 下列範例記錄中有七個個別項目。 記錄檔中的所有新項目都會新增到末端,位於最後一個區塊之前。
Name Length Committed
---- ------ ---------
ZDk5MTk5N2FkNGE0MmY5MTk5ZWViYjA0YmZhODRhYzY= 12 True
NzQxNDA5MTRhNDUzMGI2M2Y1MDMyOWZlN2QwNDZiYzQ= 2685 True
ODdjM2UyMWY3NzFhZTU3MmVlMmU5MDNlOWEwNWE3YWY= 2586 True
ZDU2MjA3OGQ2ZDU3MjczMWQ4MTRmYWNhYjAzOGJkMTg= 2688 True
ZmM3ZWJjMGQ0ZDA1ODJlOWMyODhlOWE3MDI1MGJhMTc= 2775 True
ZGVkYTc4MzQzNjEyMzlmZWE5MmRiNjc1OWE5OTc0OTQ= 2676 True
ZmY2MjUzYTIwYWIyOGU1OTA2ZDY1OWYzNmY2NmU4ZTY= 2777 True
Mzk1YzQwM2U0ZWY1ZDRhOWFlMTNhYjQ3OGVhYmUzNjk= 2675 True
ZjAyZTliYWE3OTI1YWZmYjFmMWI0MjJhNzMxZTI4MDM= 2 True
讀取區塊 Blob
在本節中,您會讀取 $blocklist
變數以擷取資料。 在下列範例中,我們會逐一查看區塊清單,從每個區塊讀取位元組,並將它們儲存在陣列中。 使用 DownloadRangeToByteArray 方法來擷取資料。
function Get-NSGFlowLogReadBlock {
[CmdletBinding()]
param (
[System.Array] [Parameter(Mandatory=$true)] $blockList,
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
)
# Set the size of the byte array to the largest block
$maxvalue = ($blocklist | measure Length -Maximum).Maximum
# Create an array to store values in
$valuearray = @()
# Define the starting index to track the current block being read
$index = 0
# Loop through each block in the block list
for($i=0; $i -lt $blocklist.count; $i++)
{
# Create a byte array object to story the bytes from the block
$downloadArray = New-Object -TypeName byte[] -ArgumentList $maxvalue
# Download the data into the ByteArray, starting with the current index, for the number of bytes in the current block. Index is increased by 3 when reading to remove preceding comma.
$CloudBlockBlob.DownloadRangeToByteArray($downloadArray,0,$index, $($blockList[$i].Length)) | Out-Null
# Increment the index by adding the current block length to the previous index
$index = $index + $blockList[$i].Length
# Retrieve the string from the byte array
$value = [System.Text.Encoding]::ASCII.GetString($downloadArray)
# Add the log entry to the value array
$valuearray += $value
}
#Return the Array
$valuearray
}
$valuearray = Get-NSGFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob
$valuearray
陣列現在包含每個區塊的字串值。 若要確認項目,請透過執行 $valuearray[$valuearray.Length-2]
取得陣列的第二個到最後一個值。 您不需要最後一個值,因為它是右方括弧。
這個值的結果會顯示在下列範例中:
{
"records": [
{
"time": "2017-06-16T20:59:43.7340000Z",
"systemId": "abcdef01-2345-6789-0abc-def012345678",
"category": "NetworkSecurityGroupFlowEvent",
"resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/MYRESOURCEGROUP/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/MYNSG",
"operationName": "NetworkSecurityGroupFlowEvents",
"properties": {
"Version": 1,
"flows": [
{
"rule": "DefaultRule_AllowInternetOutBound",
"flows": [
{
"mac": "000D3A18077E",
"flowTuples": [
"1497646722,10.0.0.4,168.62.32.14,44904,443,T,O,A",
"1497646722,10.0.0.4,52.240.48.24,45218,443,T,O,A",
"1497646725,10.0.0.4,168.62.32.14,44910,443,T,O,A",
"1497646725,10.0.0.4,52.240.48.24,45224,443,T,O,A",
"1497646728,10.0.0.4,168.62.32.14,44916,443,T,O,A",
"1497646728,10.0.0.4,52.240.48.24,45230,443,T,O,A",
"1497646732,10.0.0.4,168.62.32.14,44922,443,T,O,A",
"1497646732,10.0.0.4,52.240.48.24,45236,443,T,O,A"
]
}
]
},
{
"rule": "DefaultRule_DenyAllInBound",
"flows": []
},
{
"rule": "UserRule_ssh-rule",
"flows": []
},
{
"rule": "UserRule_web-rule",
"flows": [
{
"mac": "000D3A18077E",
"flowTuples": [
"1497646738,13.82.225.93,10.0.0.4,1180,80,T,I,A",
"1497646750,13.82.225.93,10.0.0.4,1184,80,T,I,A",
"1497646768,13.82.225.93,10.0.0.4,1181,80,T,I,A",
"1497646780,13.82.225.93,10.0.0.4,1336,80,T,I,A"
]
}
]
}
]
}
}
]
}