Desperate help in fixing broken Profiles.

Question

Desperate help in fixing broken Profiles.

Shaun Williams 1

I have a thread on Reddit here: https://www.reddit.com/r/sysadmin/comments/o0rzt3/in_desperate_need_of_help_with_a_problem_with/

Here is a copy of my issue.

Having major issues.

We have AD on Server 2019, and we also run Remote App with Session Hosts etc. Basic run of the mill setup. Clients are all Windows 10 21H1. We also run WSUS (which is helping my problem be manageable). We also run FSLogix for roaming profiles. For those that don't know, FSLogix uses a VHD file for mounting effectively the AppData folder for the user.

The problem:

We are getting a few updates that are causing a problem. The problem is that the VHD mounts, but you cannot assign a drive letter to it to be able to read it. FSLogix doesn't add a drive letter but seems to mount the VHD (I see the file getting read) but it keeps saying it's already mounted, seems it just can't read it.

The VHD files are on a Synology NAS. The problem occurs on both VHD or VHDX files. I have played with the SMB settings and I have tested on my home PC, that has Synology NAS and WIndows 10, just no domain, but I don't seem to have the issue at home.

The Updates are as follows (for both Windows Server (Session Hosts) and Windows 10:

KB5001337

KB5001330

KB5001406

KB5003173

KB5003171

KB5003637

KB5003646

There can't be the exact same problem with every single update. I am starting to think that something these updates do, is a symptom and the actual problem is something on our network, maybe even a configuration issue. The ONLY thing that I can think of that is glaringly different on my works network as opposed to home, is that here at work we have Multicasting disabled on the network. But I also can't imagine that being the problem.

I really need help working this out. WSUS lets me manage these updates but I am now concerned we will be open to other issues these updates solve. I really need to get to the bottom of this issue. Any help in further diagnosing, and/or suggestions on what to look at next would be much appreciated.

Shaun Williams 1 Reputation point

2021-06-23T02:12:14.617+00:00

OK so an update.

After many hours of testing different scenarios:

First the storage location of the FSLogix VHD file: Works 100% fine when using a Share from the AD Server. Not sure if it's specifically the AD server or just a Windows Share. 100% fails when using the NAS.

I tried letting FSLogix create the VHD from scratch on both, tried combos of copying a working VHD from AD Share to NAS and vice versa, and even tried twice on all situations to make sure there wasn't an issue with creating etc.

The first weird situation was I created a VHD with FSLogix (by login on new profile) and then copied the NAS version to the AD share. When I then tried to log in, it worked, but it renamed the original file as corrupted-{sid}-{username}.vhd, so It failed and only worked cause it created a new VHD next to the renamed version.
Shaun Williams 1 Reputation point

2021-06-23T02:12:39.353+00:00

I even tried to manually mount under a local profile, AD Share version worked, but the NAS share version, mounted the VHD, but I couldn't set a drive letter. It seems to be unable to read the partition in the VHD file.

Conclusions (so far):

Its NOT the NAS specifically. I tried all the above on another NAS. Both Synology (all I have), but both different models, and 1 of them is 3 updates behind. So not a firmware issue either. I have tried different SMB settings too.

Specifically, the issue seems that FSLogix, can create the VHD on the NAS, it creates the SID subfolder and everything. It then can't read or mount the partition so it can then save files inside it. I conclude this when I created the new VHD files, the AD version was around 500Mb the NAS version was 140Mb, both should have been exactly the same.
Shaun Williams 1 Reputation point

2021-06-23T02:12:56.557+00:00

I also copied both AD and NAS created VHD's to the NAS Profile share and tried manually mounting. I mounted both VHD's fine, it even auto-assigned a drive letter and was able to browse.

I have tried the NAS version inside the SID subfolder and next to it. It always mounts and reads fine outside of SID folder, but inside it will work sometimes, and not others.

At this stage, it looks like a permission issue, but 2 things make me question that.

I can read and write to the folder fine, I have ownership and FSLogix creates the folders and permissions.
Shaun Williams 1 Reputation point

2021-06-23T02:13:03.497+00:00

The VHD works sometimes in the SID folder, pointing that it cant be permissions, I haven't changed any other settings other than moving the VHD and remounting it.

How would folder permissions affect reading partitions from the VHD? I can understand if I couldn't mount the VHD at all etc.

So in conclusion, I know a lot more about the problem, but still not enough to know exactly what it is and possibly fix it. So I know much much more but still know nothing.

Shaun Williams 1 Reputation point

2021-06-23T02:12:14.617+00:00

OK so an update.

After many hours of testing different scenarios:

First the storage location of the FSLogix VHD file: Works 100% fine when using a Share from the AD Server. Not sure if it's specifically the AD server or just a Windows Share. 100% fails when using the NAS.

I tried letting FSLogix create the VHD from scratch on both, tried combos of copying a working VHD from AD Share to NAS and vice versa, and even tried twice on all situations to make sure there wasn't an issue with creating etc.

The first weird situation was I created a VHD with FSLogix (by login on new profile) and then copied the NAS version to the AD share. When I then tried to log in, it worked, but it renamed the original file as corrupted-{sid}-{username}.vhd, so It failed and only worked cause it created a new VHD next to the renamed version.
Shaun Williams 1 Reputation point

2021-06-23T02:12:39.353+00:00

I even tried to manually mount under a local profile, AD Share version worked, but the NAS share version, mounted the VHD, but I couldn't set a drive letter. It seems to be unable to read the partition in the VHD file.

Conclusions (so far):

Its NOT the NAS specifically. I tried all the above on another NAS. Both Synology (all I have), but both different models, and 1 of them is 3 updates behind. So not a firmware issue either. I have tried different SMB settings too.

Specifically, the issue seems that FSLogix, can create the VHD on the NAS, it creates the SID subfolder and everything. It then can't read or mount the partition so it can then save files inside it. I conclude this when I created the new VHD files, the AD version was around 500Mb the NAS version was 140Mb, both should have been exactly the same.
Shaun Williams 1 Reputation point

2021-06-23T02:12:56.557+00:00

I also copied both AD and NAS created VHD's to the NAS Profile share and tried manually mounting. I mounted both VHD's fine, it even auto-assigned a drive letter and was able to browse.

I have tried the NAS version inside the SID subfolder and next to it. It always mounts and reads fine outside of SID folder, but inside it will work sometimes, and not others.

At this stage, it looks like a permission issue, but 2 things make me question that.

I can read and write to the folder fine, I have ownership and FSLogix creates the folders and permissions.
Shaun Williams 1 Reputation point

2021-06-23T02:13:03.497+00:00

The VHD works sometimes in the SID folder, pointing that it cant be permissions, I haven't changed any other settings other than moving the VHD and remounting it.

How would folder permissions affect reading partitions from the VHD? I can understand if I couldn't mount the VHD at all etc.

So in conclusion, I know a lot more about the problem, but still not enough to know exactly what it is and possibly fix it. So I know much much more but still know nothing.

Answer 1

Hi,

Thanks for posting in Q&A platform.

Please understand, due to the environment limitation, it is hard for us to reproduce and analyze this behavior in our lab. Some necessary network traffics should also be collected for further analyzing this issue. However, analysis of log is beyond our forum support level and due to forum security policy, we have no such channel to collect user log information. If this issue is urgent, I would suggest you contact Microsoft Customer Support and Services where more in-depth investigation can be done so that you would get a more satisfying explanation and solution to this issue.

You may find phone number for your region accordingly from the link below:

https://support.microsoft.com/en-us/help/4051701/global-customer-service-phone-numbers

Best Regards,
Sunny

----------

If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 2

Hello @Shaun Williams ,

I am not familiar with your type of set-up (I have been retired for a few years now) but, if it makes sense to you, you could try the following on a client:

Issue the command: logman start why -ets -p Microsoft-Windows-SMBClient Smb_Info -o why.etl.
Try to connect to and use the profile.
Issue the command: logman stop why –ets.
Issue the command: wevtutil qe /f:text /lf:true /q:"*[System[EventID = 30406 or EventID = 30407]]" why.etl | findstr "SMB.send SMB.receive".
Review the output of the process and verify that it matches your expectations; post here if there are things that seem odd to you.

Gary

Answer 3

Julian Southern 1

Hi @Shaun Williams

Did you ever find a slution to your problem? We are experiencing the exact same issue, we have a call rasied with Microsoft but we are not getting anywhere fast. The issue has been occuring since we installed the May (I've not confirmed if it happened with the April update).

I've created the logs that @Gary Nebbett has suggested and passed it over to our storage provider. I'll be honest, I'm not sure if it will provide any isight.

Regards

Gary Nebbett 4,256 Reputation points

2021-08-20T10:44:17.843+00:00

Hello @Julian Southern ,

If you make your logs available then I will take a look at them. Depending on what the logs contain, there may be other tracing steps that provide more targeted information about the possible causes of the problem.

Gary
Julian Southern 1 Reputation point

2021-08-20T10:47:23.623+00:00

Thanks Gary, it's really appreciated.

I have attached the log. 125025-smb.txt
Shaun Williams 1 Reputation point

2021-08-20T10:52:09.17+00:00

No have not found a solution as yet, other than moving our share for fslogix vhd files to be on server 2019.

I have confirmed that this issue is happening when trying to mount the vhd profile for fslogix if the vhd is on a NAS shared via SMB.

I have tried all sorts of things to try to find the exact issue, but so far it isn't path name/length, it isn't SMB version, tested from 1 to 4.1, it isn't authentication either. It also isn't an issue with browsing, as I can still browse the Nas and we even have folder redirection pointed to the Nas over SMB. All files can be read and written. It is very specifically an issue with mounting a vhd from the smb Nas location. I move the same vhd to a local drive, and it mounta fine.

I to am desperate for an answer, 2e are not willing to decommission, $20,000 worth of storage just because of one bug.

This only started happening after the updates I mentioned. Remove those updates and it works perfectly again.
Gary Nebbett 4,256 Reputation points

2021-08-20T11:59:30.783+00:00

Hello @Julian Southern ,

Your trace data was not consistent with my first "guess" about the possible cause of the problem.

I have a clear idea about what tracing I would perform next, but it is unlikely that you would be able to interpret the (binary) trace data; furthermore, it is very likely that the trace data would contain information that you would not choose to share.

I would include the following ETW providers in the "next step" trace:

Microsoft-Windows-SMBClient
Microsoft-Windows-VHDMP
Microsoft-Windows-MountMgr
{467c1914-37f0-4c7d-b6db-5cd7dfe7bd5e} # WPP_ThisDir_CTLGUID_Mountmgr
{3c70c3b0-2fae-41d3-b68d-8f7fcaf79adb} # WPP_ThisDir_CTLGUID_VhdmpTraceGuid
{e4ad554c-63b2-441b-9f86-fe66d8084963} # WPP_ThisDir_CTLGUID_Smb20Log

One way of starting such a trace would be to issue the command:

netsh trace start provider=Microsoft-Windows-SMBClient provider=Microsoft-Windows-VHDMP provider=Microsoft-Windows-MountMgr provider={467c1914-37f0-4c7d-b6db-5cd7dfe7bd5e} keywords=0xFFFFFFFF level=255 provider={3c70c3b0-2fae-41d3-b68d-8f7fcaf79adb} keywords=0xFFFFFFFF level=255 provider={e4ad554c-63b2-441b-9f86-fe66d8084963} keywords=0xFFFFFFFF level=255 report=disabled tracefile=why.etl

The trace can be stopped with the command netsh trace stop.

In the unlikely event that you choose to make the data available, I would take a look at it.

Gary
Julian Southern 1 Reputation point

2021-08-20T14:13:24.633+00:00

Hi Gary

Thank you so much for your help with this, it's really good of you. We are just reviewing the contents of the data internally.

If we were to share I would have to privately send you an expiting link so you can be download it. Is it possible to PM in these forums?

Many Thanks
Gary Nebbett 4,256 Reputation points

2021-08-20T16:07:45.213+00:00
Hello @Julian Southern ,

There are a variety of possibilities to make one-to-one contact:

My e-mail address is in my avatar image.

I am the only Gary Nebbett on LinkedIn.

You could provide a link that requires the reader to request access.

...

As with any troubleshooting problem, it is possible that we would invest time in transferring the trace data only to find that it does not help to solve the problem.

Gary
Julian Southern 1 Reputation point

2021-08-24T12:20:15.487+00:00

Hi @Shaun Williams

Just wanted to let you know what we have discovered whilst trying to figure out a workaround that lets us keep using the storage and update the servers. If we add SYSTEM to the top level NTFS permissions and apply to 'This folder, subfolder and files', it appears to 'fix' the issue whilst retaining the correct CREATOR OWNER permissions.

IMaybe worth a shot your end?

Regards
Julian Southern 1 Reputation point

2021-08-24T12:26:18.19+00:00

I've also shared these findings with @Gary Nebbett who has been incredibly helpful in trying to figure out what is going on here.
Gobi Siva 1 Reputation point

2021-08-30T08:56:57.77+00:00

Hi Julian

I can confirm adding the system NTFS permission indeed fix the issue.
My environment is Windows 2016 VDI with Synology FSLogix share.

Answer 4

Hello @Shaun Williams ,

Julian took the risk and shared some data with me. My current suspicion is that an update to vdhmp.sys is causing the problem; vdhmp.sys seems to be included in the updates that you mentioned (at least, those for which I could find a list of affected files).

The problem seems to be use of the file system control code FSCTL_VIRTUAL_STORAGE_QUERY_PROPERTY issued by the routine VhdmpiInitializeScsiStateForSurface, which fails with error code STATUS_NOT_SUPPORTED when sent to some NAS servers.

Anything that you can add to firm up or refute this suspicion would be appreciated.

If the suspicion is founded, then there is probably no workaround to enable continued use of the NAS but the information could help in discussions with the NAS manufacturer.

Gary

Desperate help in fixing broken Profiles.

4 answers

Activity