Packet capture fails every time.

Jakezxz1 40

Hello,

I am trying to run a packet capture from my VM to see all traffic coming in and out.

I have a storage account associated with my VM and I have stepped into networking within the storage account and selected the VNETS and IP address as well as enabled the service endpoint status for those VNETs.

Connectivity wise the troubleshooter says that my storage account is reachable.

I cannot access the storage of the VM so I cannot save it locally - this is because my VM is running as an NVA and cannot access the underlying architecture directly.

Every time I run the pcap it fails right away:

Session status

Stopped

Storage location

packetcapture_20_42_24_349.cap

Start time

2023-05-08T20:42:24.8802769Z

Errors

LocalFileFailed,StorageFailed

Virtual machine

NSv270

Timeout Interval

18000

Subscription

Azure subscription 1

Filters

Any: : (Local) - : (Remote)

ChaitanyaNaykodi-MSFT 23,106 Reputation points Microsoft Employee

2023-05-11T18:26:40.4233333+00:00
@Jakezxz1

Thank you for reaching out on the Microsoft Q&A forum.

If I understand correctly, you are trying to perform a packet capture from your VM which is a NVA so you do not have access to the VM's disk storage so you are trying to store cap file to storage account but you are unable to run the packet capture and it fails immediately.
Based on my understanding above can you please confirm few settings mentioned below also try the troubleshooting steps.

As documented in the pre-requisite sections here. Can please confirm the VM has the following outbound TCP connectivity:

to the chosen storage account over port 443

to 169.254.169.254 over port 80

to 168.63.129.16 over port 8037

Can you please confirm if you have installed AzureNetworkWatcherExtension in the Virtual Machine?

If above requirements check out, Can you please try to PS-ping your storage account from the VM? You can perform PSping connectivity test for your storage account here to determine if there are no NSG/OS firewall rules blocking this connectivity. Sample Command psping -n 300 <accountname>.blob.core.windows.net:443 You can also check the storage firewall configuration as mentioned here.

There is a possibility that this can happen due to version of the Network Watcher extension installed. You can follow the steps mentioned here to determine if you have the latest 1.4.2573.1 version is installed on the VM. If the latest version is not installed. You can follow the AZ CLI steps mentioned here to upgrade the extension.

Looking forward to your reply. Please let me know if you have any questions. Thank you!

Jakezxz1 40

Hi Thanks for your reply,

Within my storage account - I have selected "Enabled from selected virtual networks and IP addresses"

I have selected the 2 Subnets that my VM exists in and enabled service endpoint status.

I have the AzureNetworkWatcherExtension already installed - It is version 1.4 - When I tried the force update I was given an error message:

(VMExtensionProvisioningError) VM has reported a failure when processing extension 'AzureNetworkWatcherExtension'. Error message: "We couldn't read any status for Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux-1.4.2573.1 extension, for the sequence number 1. It failed due to [ExtensionStatusError] [Errno 2] No such file or directory: u'/usr/share/oem/waagent/Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux-1.4.2573.1/status/1.status'
Inner error: [Errno 2] No such file or directory: u'/usr/share/oem/waagent/Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux-1.4.2573.1/status/1.status'"

More information on troubleshooting is available at https://aka.ms/vmextensionlinuxtroubleshoot 
Code: VMExtensionProvisioningError
Message: VM has reported a failure when processing extension 'AzureNetworkWatcherExtension'. Error message: "We couldn't read any status for Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux-1.4.2573.1 extension, for the sequence number 1. It failed due to [ExtensionStatusError] [Errno 2] No such file or directory: u'/usr/share/oem/waagent/Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux-1.4.2573.1/status/1.status'
Inner error: [Errno 2] No such file or directory: u'/usr/share/oem/waagent/Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux-1.4.2573.1/status/1.status'"

More information on troubleshooting is available at https://aka.ms/vmextensionlinuxtroubleshoot

My NSG is fully open - Any/Any - inbound and outbound.

ChaitanyaNaykodi-MSFT 23,106 Reputation points Microsoft Employee

2023-05-12T23:03:02.61+00:00

@Jakezxz1

Thank you for getting back and sharing additional details.

Based on the error received, I wonder if the issue due to access to the VM's disk storage. You can try the troubleshooting steps shared by TP below and see if that resolves the issue.

I also think creating a support ticket in this case will be good option as a support engineer can take a deeper look at the logs and provide additional insights specific to your environment. If you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details. I will help create a one-time free support ticket for you in this case.

Subject : Attn Chaitanya

Thread URL: Link to this thread.

Subscription ID

Please let me know once you have done the same.
Jakezxz1 40 Reputation points

2023-05-13T00:55:26.9566667+00:00

Hi Attn,

Thank you for your detailed and swift support.

As per the response from TP below, I have verified that the version you have recommended is already the present one installed.

Furthermore, I already created a new LRS account and connected my VMs to it with sadly the same effect.

I do not believe this to be an admin / permission issue as I setup the subscription.

My Disk on the VM itself only has an OS Disk - No Data disks - Is this an issue?
Jakezxz1 40 Reputation points

2023-05-13T01:58:24.3433333+00:00

I have completed the actions you have requested:

The error says "The specified container doesn't exist" - But I don't quite know what it means - I cannot specify a container name as far as I can tell - Am I missing something?

Im quite new to azure, but it seems like the pcap is trying to create a storage location, fails or doesn't do it, and when I then go to said location, it doesn't exist.

Is this correct?
Jakezxz1 40 Reputation points

2023-05-19T14:59:49.4733333+00:00

Hello please update
ChaitanyaNaykodi-MSFT 23,106 Reputation points Microsoft Employee

2023-05-22T20:04:32.73+00:00

@Jakezxz1

Apologies for the delayed response here. I think a deeper investigation is required in the case.

As a support engineer can take a deeper look at the logs and provide additional insights specific to your environment. If you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details. I will help create a one-time free support ticket for you in this case.

Subject : Attn Chaitanya

Thread URL: Link to this thread.

Subscription ID

Please let me know once you have done the same.

1 answer

TP 77,076 Reputation points

2023-05-08T23:39:40.4733333+00:00

Hi,

When I've seen that error it was because there was a problem saving to the storage account due to networking.

What I would recommend is to create a new general purpose v2 lrs storage account in same region, with public access enabled from all networks, and then try a new capture with it set to new storage account.

If the above was useful please click Accept Answer.

Thanks.

-TP
Please sign in to rate this answer.
Jakezxz1 40 Reputation points

2023-05-10T12:38:24.29+00:00

Hi thanks for the reply

I did exactly as you asked and it didn't work:

Session IDbcdffb5554e54cc48220ea92e01516f7

Resource IDNot available

ExtensionMicrosoft_Azure_Storage

ContentBlobPropertiesBladeV2

Error code403

Details

Forbidden

Could you please recommend additional steps?

TP 77,076 Reputation points

2023-05-10T19:25:48.0966667+00:00

@Jakezxz1 From your error message it appears you don't have permission to create a storage account. Has the owner of the subscription restricted your access? You may need to ask them to create the storage account for you and/or grant you additional permissions.

Jakezxz1 40 Reputation points

2023-05-11T08:42:19.4+00:00

This is not the case, I am the owner of the subscription.

TP 77,076 Reputation points

2023-05-12T15:40:59.34+00:00

@Jakezxz1 To be clear, you are saying that if you go to the portal, click Create a resource, choose storage account, it errors out with the message you posted above (which includes "403 Forbidden")? As account owner you should have no difficulty creating a new storage account in the subscription.

Regardless, I see above you mentioned there is an issue with AzureNetworkWatcher VM Extension. Please try uninstalling the extension from the VM, installing it, then test your packet capture.

In portal, navigate to the VM -- Extensions + applications blade, click on AzureNetworkWatcher, then click Uninstall button. After extension has been uninstalled (wait for Azure notification that says "Successfully deleted vm extension") or similar, click the Add button.

After clicking Add button, type network watcher in the box, press enter, click Load more button, then click Network Watcher agent, click Next, etc., to start the install. Wait for the install to finish and then test your packet capture again.

Thanks.

Jakezxz1 40 Reputation points

2023-05-13T01:28:57.67+00:00

I have completed the actions you have requested:

The error says "The specified container doesn't exist" - But I don't quite know what it means - I cannot specify a container name as far as I can tell - Am I missing something?

{ "shellProps": { "sessionId": "872c20300bff464aaa5a284b19699588", "extName": "Microsoft_Azure_Storage", "contentName": "BlobPropertiesBladeV2" }, "error": { "message": "The specified container does not exist.", "code": 404, "details": { "htmlTemplate": "<ul data-bind=\"foreach: causes\"><li> <a target=\"_blank\" data-bind=\"attr:{href:uri},text:text\"></a></li></ul>", "viewModel": { "causes": [ { "details": "NotFound", "link": null } ] } } }}

TP 77,076 Reputation points

2023-05-13T03:21:06.72+00:00

@Jakezxz1 I just did quick test. Created linux VM, had it install network watcher extension during creation, created new storage account, started capture, stopped after few minutes. In storage account the capture file was stored under path similar to below:

Containers/ network-watcher-logs / subscriptions / <subscriptionId> / resourcegroups / <resourceGroup> / providers / microsoft.compute / virtualmachines / <VMName> / 2023 / 05 / 13

The above container, path, and capture blob was created automatically by the capture. I didn't need to create it.

When you created capture, did it succeed and when you refreshed the list of captures in network watcher, did it show as status Running? Did you Stop the capture, navigate to the Storage account in the portal -- Containers, and check if a path similar to above exists?

Jakezxz1 40 Reputation points

2023-05-13T10:15:57.3933333+00:00

Hi friend thanks so much.

I am surprised by your question; this is exactly the issue I’m having :-)

the packet capture stops automatically with the error message I have shared with you - the pcap gets created but immediately stops, when I click open it says the file doesn’t exist
Sign in to comment