RamblingsFromTheField

Make friends and work closely with your networking team: Skype audio and video

Mack McMillan — Wed, 29 Mar 2017 14:10:20 GMT

Imagine that you are a car manufacturer. Of course I am talking about Ford. You have a meeting with the county and they determine that they will do no more road maintenance because they cannot afford to pay inspectors to review the integrity of the roads. Unless you’re a real man and drive a Ford Pickup Truck this is horrible news. Fairly soon the road will have so many pot holes that driving a car will become very difficult and unpleasant. Your users will no longer enjoy taking drives in their cars. Car sales will fall off and you will soon be laying off all of your employees. Users will go back to using your competitor’s products; horse and buggy, motorcycle, walking, etc. You have two options:

Explain to your users that it is the county’s fault for not monitoring their roads.
Now imagine that you make a deal with the county. Since your employees travel down the county roads every day, you will have your employees provide feedback to the county when they find a pot hole. Then the county can dispatch a maintenance crew to repair that specific area of the road. You do not need to shut down your factories and lay everybody off. Users are happy again.

It’s a win-win situation. Hopefully you can see where I am going with this. You need to use the Skype monitoring reports, especially the location report, to let the networking team know what the issue is and the IP addresses of the end points that experienced the issues. Skype is the canary in the coal mine. (Shout out to our President for putting our coal miners back to work.) Real time Audio and Video will find every little issue with the network. Providing this information to the networking team will help everybody involved. Key network issues that affect Skype: Packet loss: I have seen the following cause this

Quality of Service (QoS). This is very common when QoS is configured properly on Skype and the OS but not properly configured on the internal network or the ISP’s network. I have found with Verizon that they have a service called Gold Car where they expedite all traffic in the EF queue. However, if the customer’s network equipment is putting Audio packets in the EF queue but Gold Car is not configured on the Verizon circuit, all of the audio packets are dropped.
Congestion on the network. There just simply is not enough bandwidth to handle all of the demands. Packets will be dropped.
Maxim Transmission Unit (MTU) size mismatch. The maximum size of the packets is set on the computer and the network equipment. If the computer is using a larger packet size than the network equipment then those packets must be fragmented, often leading to Packet drops. A common issue is when the MTU size is set differently at remote locations than it is at the datacenter.

Jitter: I have seen the following cause this:

Congestion on the network.
QoS
MTU size mismatch
Jitter buffer http://searchunifiedcommunications.techtarget.com/definition/jitter-buffer

Latency: When it takes the packets too long to get from point A to point B the quality of the audio and video will be significantly degraded. http://sourcedaddy.com/windows-7/what-causes-latency-how-measure-it-and-how-to-control-it.html

When dealing with international organizations the distances between locations can be too great to get the packets to their destination in a timely manner.
Routing. I have seen many cases where routing was not properly configured to use the most efficient path for the packets. You can provide evidence of this to the networking team by doing a trace route between the end points and providing the screen capture of the results.

I have saved screen shots of trace routes to my key locations, when all is well, so that I can compare them to new trace routes when I suspect a routing issue. It is not uncommon for the route to change as network equipment goes offline. Also Performance based routing can cause a route change. The maximum latency that Microsoft considers acceptable for Skype Audio and Video is 150ms per direction. Meaning that for an A/V conference there should be no more than 150ms from the client to the server and 150ms from the server back to the client. I have built a good relationship with my networking teams at most of my customers. There was one that no matter how nice I was they hated me because my application was using their bandwidth and I wouldn’t allow them to perform deep packet inspection on traffic to and from the Edge servers. At my current customer I have almost daily meetings with the network team to explain where there are possible issues. This makes things much better for the users, rather than me complaining about the network and doing nothing of value. Working with the closely with the networking team we have reduced the poor call quality from 40% to below 1%.

Performance Based Routing: not necessarily good for real time audio and video

Mack McMillan — Tue, 28 Mar 2017 11:21:33 GMT

At first read performance based routing sounds great. The network automatically choosing the best route based on the performance of the routes and being able to make automatic adjustments without admin intervention. However if the network doesn’t converge quickly after the route is changed it can adversely affect real time audio and video. At my customer we use both Skype and ACANO/CISCO VTC. The network is stretched across the globe but mostly across the greatest country in the world, you guessed it “the United States of America”. Several of our field sites were having issues when using real time audio and video. Their Lync clients would disconnect constantly during A/V conferences and peer to peer sessions with users at other locations, the Video would freeze and the Audio drop. Needless to say the users at these locations were very unhappy with Microsoft and ACANO. After many times of providing metrics, pulled from the Skype monitoring reports I was finally able to get the networking team to take notice and monitor the network for one of the remote locations for more than a 30 second snap shot. What they found was that every few minutes routing loops would form and it would take about 30 seconds before things went back to normal. Another indicator was that performing tracert’s you could see the routes changing frequently. The network engineer, we’ll call him Bob, determined that this was one of the remote locations where Performance Based Routing had been Piloted. For some reason the route was constantly changing and when it did it would take the network up to 30 seconds to converge. 30 seconds is a life time for Skype. After removing the performance based routing we held a 30 minute A/V conference with several sites, including the affected site and the experience was as expected. The users at the location were extremely happy. Of course they all asked me what I had fixed in Skype and the ACANO. Luckily for me the network engineer is a standup guy and explained in detail about the issue. Of course now we are fighting the battle with the manager who championed performance based routing as the magical thing that would fix all of our networking issues. No one wants to hear that their baby is ugly. Performance Based Routing: http://www.cisco.com/c/en/us/products/ios-nx-os-software/performance-routing-pfr/index.html http://docwiki.cisco.com/wiki/Performance_Routing_FAQs

Office 2013 Lag – Slow Launch on a Closed Loop Network

Flurb — Mon, 06 Mar 2017 02:37:05 GMT

Day 1: Today my customer is experiencing an issue with multiple lag problems running Office 2013 on a Closed Loop Network. What is a Closed Loop Network? Closed Loop Network (which I will reference as CLN from now on) is a network that has NO ability to connect the Internet. Let’s breakdown the scenario: 1) Running Windows 7 w/SP-1. 2) Running Office 2013 w/SP-1. 3) Running McAfee antivirus. 4) Running Internet Explorer as the default browser. 5) Bitlocker is being used for hard drive encryption 6) All workstations are domain joined. 7) All Domain Controllers are running Server 2012R2 (DFL/FFL is also at 2012R2). 8) Home Drives (H-Drives) are mapped to NetApp appliances. 9) Folder Redirection for My Documents also points to the same NetApp appliance. 10) No roaming profiles. Here is what ALL customers are experiencing: 1) Customer Logs in. 2) Customer launches Word 2013 (I just picked this application as an example). 3) At the Word 2013 “splash screen”, you see one of the following plug-ins load and sit there for up to 30 seconds (per plug-in): a. OneNote 2013. b. Adobe Acrobat XI. c. Dragon Naturally Speaking. 4) Once all the plug-ins finish, then the customer can utilize Word 2013. a. If they were to close Word and re-open Word, the behavior happens all over again. 5) This is happening to all main Office 2013 applications (Outlook, Word, Excel & PowerPoint, but not Access). 6) Obviously not everyone has all 3 of the above “problem” plug-ins installed but for those that do, they are experiencing up to a 90 second lag every time they launch an Office application. Now I just love spoilers, so I am going to tell you what the culprit/root cause is: CRL Checking. The how/what/why/who/when/where I will get in to on Day 2, which I will post tomorrow. ………… to be continued. Day 2 (sorry for the delay): Let me start off by saying that 2 of my most commonly used tools are ProcMon (part of the SysInternals Suite) and NetMon (but this has been replaced by Microsoft Message Analyzer). ProcMon (Process Monitor) is great at giving you a SUPER detailed report of every process that is running on your computer and what it is try do (Query/Write/Open a file or registry key). NetMon (Network Monitor) is great at giving you a SUPER detailed report of every process that is running that is accessing a Port and/or Protocol. The reports that these utilities create can be overwhelming. I ran both of these tools (not going in to detail about the use of these great utilities), one at a time, and started digging through the results. These are the "key" items that I observed:

When I launched Word (with Dragon Naturally Speaking installed) I saw DNS traffic looking for crl.verisgn.com.
When I launched Word (with OneNote 2013 previous launched, this is KEY) I saw DNS traffic looking for crl.microsoft.com (NOTE: I later found out that this also happened with Visual Studio installed)
When I launched Word (with Adobe Acrobat XI installed) I saw DNS traffic looking for crl.symcb.com & crl.symdb.com & ocsp.symantec.com.
1. NOTE: symcb and symdb are simply variants of Symantec.

Well on a CLN (Closed Loop Network), this type of traffic is NEVER going to resolve and the websites will never be reached and therefore the servers at the other end of these request will never respond. The most important part of these URLs are the CRL (Certificate Revocation List) and OCSP (Online Certificate Status Protocol) section. Without going in to SUPER detail, these products are trying to contact a server to find out if the certificates that were used to sign their perspective company's files are valid or not. Since there is no server to talk to, they wait XXX number of seconds, then timeout. XXX number of seconds = LAG! Each plugin that is installed is that much more LAG experienced. If each plug-in take 30 seconds and you have all 3 products installed, it could take Word over 90 seconds to launch before the customer can type his/her first letter. Well this is highly unacceptable. So now that we know that the Certificate checking is causing the lag there are 3 more things we need to figure out.

What files are causing this Certificate checking?
1. Well I left my list of DLL/MSI/EXE files for each product at work, so I can't provide that this today but I will have then added to this BLOG by the end of the week.
Are there any other websites/URLs that these plug-ins are trying to reach that my captures may have missed?
1. HECK YES!
2. Between the 3 plug-ins there were almost 20 different URLs that these products were trying to access (that I captured).
How do eliminate/reduce the LAG time of the Certificate checking that will NEVER work?
1. This is the most important question on this entire web page.
2. There are over 8 different ways to attack this, but I am only going to focus on the 2 that I used

Option #1 (AKA: The down and dirty, I washed my hands for hours after doing this and never felt clean): NOTE: I highly recommend to NEVER DO THIS long term and only use this method as a Proof Of Concept/testing. Also NEVER do this on an Internet connected computer!

HOSTS. file
1. Using the HOSTS. file (no extension), located at the C:\Windows\System32\Drivers\ETC folder, I added the following "DNS" entries:
  1. 127.0.0.1 crl.microsoft.com
  2. 127.0.0.1 crl.verisign.com
  3. 127.0.0.1 crl.symcb.com
  4. 127.0.0.1 crl.symdb.com
  5. 127.0.0.1 ocsp.symantec.com
2. I then saved the HOSTS. file (by default it is read only to a "normal" user).
3. I re-launched Word and the launch time, with all 3 plug-ins installed, was between 5-10 seconds (before it was over a minute long).
  1. I tested this with Excel, PowerPoint and Outlook and they all had the same speed improvement.

So what did I do/accomplish by modifying my HOSTS. file? I told the software that when you go looking for these EXACT (notice there are no wildcards being used here) websites that you will "call home". 127.0.0.1 is every computer home address, no matter what its true IP address is. So the Certificate checking was looking for crl.microsoft.com, and the HOSTS file said to look locally ("call home") and since there was a computer with a "heart beat", the timeout took less than 1 second. The fact that your computer CAN NOT perform Certificate checking on these plug-ins and their files is irrelevant. There are some big downsides of doing this:

This is a manual process to create and maintain these entries.
There is a good chance that these websites addresses will change with future releases of their products (I know that one of Dragon Naturally Speaking crl sites changed between v10 and v12)).
Since I can't use wildcards there is a good chance that I will miss one of the Certificate sites that the plug-in may reference later on in the usage of the application.
OH YEA, this is just a horrible idea (did I mention that I felt dirty even trying this?)!

So, what is a better way to handle this? Remember the modification of the HOSTS. file should only be used for testing or a temporary solution. Also remember this is a CLN (Closed Loop Network), any traffic destined for ANYTHING.Microsoft.com or ANYTHING.Symantec.com or ANYTHING.Verisgn.com will NEVER make it. Time to break out the BIG GUNS and start creating zones in DNS. However I will cover what we tested and finally did in DNS tomorrow. ………… to be continued.

Services on Lync Edge server will not start and 14611, 42012 errors are in the logs

Mack McMillan — Fri, 14 Nov 2014 13:55:22 GMT

I wrote this blog after the customer asked me to look into
why one of their Edge server’s services would not start and would not
replicate.

My customer has Two Lync Edge servers in its topology.

Issue: One edge server is functioning fine. The services on
the second Lync edge server would not start. The Lync Management Console on the
Front End server showed that only the healthy Edge server was replicating.

Lync Event logs on the troublesome Edge server shows 14611
and 42012 errors.

“Current topology configuration either does not include this
machine or the definition for this service.”

A quick Bing search showed several blogs with the issue
being an error with the topology and importing it into the Edge server.

At my customer this was not the issue.

The customer had forgotten to add a primary DNS suffix to
the one of their Edge servers. So the Edge server had a short name but did not
have an FQDN. As you know Lync is all about FQDNs.

I opened the computer properties, clicked on Advance System
Settings, clicked on Computer Name, clicked Change, clicked More, entered the
domain name into the Primary DNS Suffix of this computer window, clicked my way
out of the properties and rebooted the server.

After the reboot the services were started and the
replication showed green on the Lync management console.

Turns out that the Event log error, “Current topology
configuration either does not include this machine or the definition for this
service.”, was spot on. The topology
listed an FQDN for the Edge servers, as it should. That FQDN did not exist on
the server so when the services tried to start there was no match for the FQDN
listed in the topology.

Lync Edge services do not start-14611-42012-errors.docx

Using SEFautil to manage users delegates

Mack McMillan — Fri, 14 Nov 2014 13:41:00 GMT

I wrote this blog when my customer asked me to handle the
following issue.

UserA had made UserB a delegate. However, now UserA is
archived and I wasn’t allowed to remove them from Lync. UserB was tired of
getting Lync calls for UserA. Apparently UserA was quite popular.

The SEFAutil resource kit utility was not easy to use.
However once you get the syntax down it does the job nicely.

Using the SEFAutil Resource Kit tool

Before you can use the SEFAutil tool to manage delegates and
calling features for users you need to configure the Lync environment to trust
the tool.

Environment:

Domain: Mackzilla.com

SIPDomain: Songcha.com

Site = Site1

PoolFQDN = Lyncpool.MackZilla.com

FrontEnd Servers: HQ-LYFE-401, HQ-LYFE-402,
HQ-LYFE-403

Create variable for site:

$site=Get-CsSite –Identity “”

Create Trusted Application Pool

New-CsTrustedApplicationPool –Identity
-Registrar -Site $site.SiteID
–ComputerFQDN

Create Trusted Application

New-CsTrustedApplication –ApplicationID sefautil
–TrustedApplicationPoolFQDN -Port 7489

Enable the Topology

Enable-CsTopology

Note: Allow the port through the firewall of the server your
running the SEFAutil command. In my case TCP:7489

Example

$site=Get-CsSite –Identity “site1”

New-CsTrustedApplicationPool –Identity Lyncpool.MackZilla.com
–Registrar LyncPool.MackZilla.com –Site $site.SiteID –ComputerFQDN HQ-LYFE-401

New-CsTrustedApplication –ApplicationID sefautil
–TrustedApplicationPoolFQDN Lyncpool.MackZilla.com –Port 7489

Enable-CsTopology

To add additional Computers to the pool:

New-CsTrustedApplicationComputer -Identity -Pool

Example:

New-CsTrustedApplicationComputer –Identity
HQ-LYFE-402.MackZilla.com –Pool Lyncpool.MackZilla.com

Using SEFAutil

The Lync Resource Kit needs to be installed on the server.
In my case HQ-LYFE-401, HQ-LYFE-402 and HQ-LYFE-403.

Open a command prompt and navigate to the ResKit folder:

In my environment C:\Program Files\Microsoft Lync Server
2010\ResKit\

Remove a delegate from a user

Type: sefautil.exe /server:
/RemoveDelegate:

Example: sefautil.exe /server:hq-lyfe-401.MackZilla.com userAlpha@songcha.com /RemoveDelegate:USERBRAVO@songcha.com

Note: Leave SIP: off of the user’s sipuri.

Note: The delegates sip name is case sensitive. Looking in
the Lync Management Console showed that the users sip name was capitalized. If you
don’t match the case the command will not work. Oddly enough the case of the
actual user you are removing the delegate from does not matter. This command does not provide any errors so
you have to verify that it worked by looking at the final output of the
command:

If the command works the delegate name will not show:

User Aor: sip:userAlpha@songch.com

Display Name: User C.
Alpha

UM Enabled: True (Could
also say false)

Simulring enabled:
false (Could also say true)

CallForwarding Enbaled:
false (Could also say true)

The command did not work if it says:

User Aor: sip:userAlpha@songch.com

Display Name: User C. Alpha

UM Enabled: True (Could
also say false)

Simulring enabled:
false (Could also say true)

Simultaneously Ringing
Delegates: sip:USERBRAVO@songcha.com

If the delegate is
still listed then the command did not perform the task.

Note: This was written for the 2010 version of the tool.
There is a 2013 version as well. I will verify that the syntax is the same and
update this blog.

Kudos to Jens Trier Rasmussen: http://blogs.technet.com/b/jenstr/archive/2010/12/07/how-to-get-sefautil-running.aspx

Kudos to Frederik Lefevre http://www.ucprofessional.com/2011/05/sefautil.html

Using the SEFAutil Resource Kit tool.docx