They have released the Azure batch services addon for Blender:  https://github.com/Azure/azure-batch-apps-blender  and http://dl.windowsazure.com/batchapps/blender/guide_setup.html#blender

This is just a quick rundown on enabling it

CLIENT SIDE

Download the python libs:

· Batch Apps Python Client

· Keyring

· OAuthLib

· Requests-OAuthLib

COPY FILES TO BLENDER:

C:\Users\spat\Downloads\Blender_Addons\azure\CloudAppComponents\Blender\2.73\python\lib\site-packages

Copy the client code to blender addons dir:

C:\Users\spat\AppData\Roaming\Blender Foundation\Blender\2.73\scripts\addons

Activate the blender addon – make sure the addon shows in the animation tab

SERVER SIDE:

1. Enable batch services: https://account.windowsazure.com/PreviewFeatures, You then need to update the service:

2. Download client and servers apps from here: https://github.com/Azure/azure-batch-apps-blender

3. Compile in VS 2013

4. Download imageMagick

· http://www.imagemagick.org/download/binaries/ImageMagick-6.9.0-4-Q16-x86-windows.zip

5. Download blender or zip up your existing install

6. Zip up the assemblies

7. Zip up blender + imagemagick

Create the application and upload

When it is done you have this:

SUBMIT JOB

Wanted to add multiple claims and needed to also have the name specified - not sure where this was documented but here it is anyway - maybe i overlooked it somewhere.

    $claim_PrimarySID = New-AdfsClaimRuleSet -ClaimRule '@RuleName = "Pass through GroupSID" c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/groupsid"] => issue(claim = c);'

    $claim_GroupSid = New-AdfsClaimRuleSet -ClaimRule '@RuleName = "Pass through UPN" c:[Type == "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn"] => issue(claim = c);'

    $All_Claims = New-AdfsClaimRuleSet -ClaimRule  ($claim_GroupSid.ClaimRules + $claim_PrimarySID.ClaimRules)

   Add-ADFSRelyingPartyTrust -Name $name -Identifier $name -IssuanceAuthorizationRules $authorizationrule -issuancetransformrules  $All_Claims.ClaimRulesString etc.. 

It's all built into two commands:

-Set-OrganizationConfig --> set the token signing cert, ADFS issuer and AudienceURIs
-Set-EcpVirtualDirectory and Set-OWAVirtualDirectory -AdfsAuthentication

See http://technet.microsoft.com/en-us/library/dn635116(v=exchg.150).aspx for more details

spat

Ran into an interesting issue the other day where WinRM connections were taking 15-20s per connection to the domain controllers and due to many connections from an automation system, connections would backup and then fail.

The only real hint we had was from the WinRM logs:

logman.exe start winrm -p Microsoft-Windows-Winrm -o winrm.etl –ets
< repro>
logman.exe stop winrm -ets

tracerpt.exe winrmtrace.etl -of XML -o winrmtrace.xml

tracerpt.exe winrmtrace.etl -of CSV -o winrmtrace.csv

18 seconds in between this connection.  After taking a look at some network traces and a debugging the svchost instance with wsmsvc.dll loaded ( the winRM service modeule)

tlist -m wsmsvc.dll

c:\windows\system32\wsmsvc.dll -  856 svchost.exe

We can see a call to LookupAccountName()

In this case, it is looking for the group  “WinRMRemoteWMIUsers__"  See  http://msdn.microsoft.com/en-us/library/aa384295(v=vs.85).aspxfor more information:

Default Group Access

During setup, WinRM creates the local group WinRMRemoteWMIUsers__. WinRM then restricts remote access to any user that is not a member of either the local administration group or the WinRMRemoteWMIUsers__ group. You can add a local user, domain user, or domain group to WinRMRemoteWMIUsers__ by typing net localgroup WinRMRemoteWMIUsers__ /add <domain>\<username> at the command prompt. Optionally, you can use the Group Policy to add a user to the group.

The code here passes an unqualified name “WinRMRemoteWMIUsers”  to advapi32!LookupAccountNameW which will then cause the delay because it fails to find it in the local domain and proceeds to try all the trusted domains – there were about a dozen trusted domains to enumerate and it did find it one eventually – it actually makes two calls, one to determine the amount of memory needed to allocate and another after the memory is allocated/

Resolution: created the WinRMRemoteWMIUsers group as a domain group and the delays went away.

In order to avoid situations like this where some code makes a plea for a name resolution without specifying any domain or machine – customers should implement LsaLookupRestrictIsolatedNameLevel - see http://support.microsoft.com/default.aspx?scid=kb;EN-US;818024

In addition, customers should consider implementing the ‘NeverPing’ value in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters

Theory on how it happened? WinRM was installed on a member server and the local group created – then the member server was promoted and the local group was lost as it became a DC.

 EDIT: Thanks to another engineer I was made aware that this can also cause hi CPU in LSASS.EXE

[EDIT] 8/6/2104:

Another error which you can hit is this one:

System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server SERVERFOO.CONTOSO.CON failed with the following error message : The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic.

This is because it could randomly write another ACE for the SID resolved from WinRMRemoteWMIUsers__ 

The process is something like this:

1. Check for group - get SID
2. Check security descriptor for the SID
1. For WMI provider:  CD to WSMan:\localhost\Plugin\WMI Provider
2. SID not exist -> go write the ACE
3. Next time - repeat since the SID returned is random based on SID lookup outside of the forest it may try to write yet another ACE w/ "new" SID for WinRMRemoteWMIUsers__
4. Buffer allocated for SD is too small and we get back 0xc0000023 - return in the ETL trace shows "The authorization of the user failed with error 122"

[2]01FC.14AC::08/04/2014-17:01:12.065 [Microsoft-Windows-WinRM/Analytic ] User CONTOSO\Administrator authenticated successfully using CredSSP authentication 
[2]01FC.14AC::08/04/2014-17:01:12.065 [Microsoft-Windows-WinRM/Analytic ] Authorizing the user 
[0]01FC.14AC::08/04/2014-17:01:24.246 [Microsoft-Windows-WinRM/Analytic ] An error was encountered while processing an operation. Error Code: 122 Error String:<f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="122" Machine="FOOSERVER.CONTOSO.COM"><f:Message>The data area passed to a system call is too small. </f:Message></f:WSManFault> EXTRADATA: 122, <f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="122" Machine="FOOSERVER.CONTOSO.COM"><f:Message>The data area passed to a system call is too small. </f:Message></f:WSManFault>, , , , 
[0]01FC.14AC::08/04/2014-17:01:24.246 [Microsoft-Windows-WinRM/Operational ] The authorization of the user failed with error 122 
[0]01FC.14AC::08/04/2014-17:01:24.246 [Microsoft-Windows-WinRM/Analytic ] Sending HTTP error back to the client due to a transport failure. The HTTP status code is 500

keyword:  LookupAccoutName slow NeverPing LsaLookupRestrictIsolatedNameLevel 

Just a quick note: In case you were not aware - netdom.exe cannot create a Forest trust (inbound or outbound). But you can leverage the S.DS namespace to automate this with a little powershell:

$targetForestName = "targetForest.local"

$trustPassword = "PassWord123!23"

$TrustDirection = "Outbound"   # see http://msdn.microsoft.com/en-us/library/system.directoryservices.activedirectory.trustdirection.aspx

$Forest  = [System.DirectoryServices.ActiveDirectory.Forest]::GetCurrentForest()

$Forest.CreateLocalSideOfTrustRelationship($targetForestName,$TrustDirection,$trustPassword)

or both sides:

$DirectoryContextForTargetForest= new-object
System.DirectoryServices.ActiveDirectory.DirectoryContext("Forest",”contoso.local”,"contoso\administrator","Password")

$TargetForest =
[System.DirectoryServices.ActiveDirectory.Forest]::GetForest($DirectoryContextForTargetForest)

 $Forest  =
[System.DirectoryServices.ActiveDirectory.Forest]::GetCurrentForest()

$Forest.CreateTrustRelationship($TargetForest,"Bidirectional")

Hope it helps someone

spat 

This was an interesting one which rolled by recently, and it’s a looong post so I apologize ahead of time.

Let’s start with the end user experience and move on from there: User(s) cannot send mail or retrieve mail from Exchange 2010 server.

Well that’s pretty simple J.

So when we looked at the Exchange servers, we see a whole slew of events from various Exchange roles and servers – I have changed the target DC name, but the DC failing to return in a timely manner also varied.

 At first we wondered if the queries were in fact “bad” ( expensive or inefficient ) queries but after looking at them they were all very targeted queries.

Something like:

“CN=ExchSrv-01 MBX Store 013,CN=Databases,CN=Exchange Administrative Group(BAR),CN=Administrative Groups,CN=XCH,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=DC1,DC=FOO,DC=COM"

This path only had a few objects under it and the DCs should have been able to handle massive amounts of these calls without even breathing hard, and sure enough, when the Perf data was reviewed the CPU for LSASS was not very high ( IIRC something around 20% ) during the timeframe in which the Exchange servers were falling over. 

Which brings us to performance data.  There was no baseline for anything in this environment , so in order to determine what “normal” was, we gathered a day or so’s worth of data

The list below may not be the exact names as I am typing them from memory, but you get the picture (I hope).

·         Overall System CPU

·         LSASS CPU

·         Memory(*)

·         Process LSASS Private bytes

·         NTDS (all)

·         Logical Disk

·         Physical Disk

However, none of this data was captured for a baseline, or from a DC during an outage.

So, I wrote a rudimentary LDAP stress tool which you can spin up X amount of threads in order to hit the DCs with various LDAP searches. We wanted to use this during off hours in order to determine the impact on the DCs, because at the time we did not have any perf data from an actual outage and wanted to get some more data before the next day.

I use Perfmon to trend and analyze high level data – think of it like day to day trending and analysis, and use something like SPA ( In 2k8 it’s in Perfmon under “Data Collector Sets”) see here for more info on it.  I think of SPA more like a microscope, when you want to zoom in on a period of time where you know, or suspect, something is going on. You would never (IMO) want to take a day’s worth of SPA data to plow through.

So, we ran some LDAP stress against the DCs, trying to make them cry uncle and give us a timeout but it did not. Perhaps it just wasn’t generating enough load (even at 800 threads from multiple instances on multiple clients).

What we did see, was interesting. We can see that the stress tool was indeed giving us some perf data increase, and one of the more interesting counters is the ATQ (asynchronous thread queue) counters :

·         \NTDS\ATQ Outstanding Queued Requests

·         \NTDS\ATQ Request Latency

·         \NTDS\ATQ Estimated Queue Dela

NOTE: ‘normal’ data  is on the left ( may not be entirely normal as this is still a Saturday afternoon and lower than expected numbers)

Looking at ATQ Outstanding Queued Requests to see how many requests are queued at the domain controller.

A high level of queuing indicates that requests are arriving at the domain controller faster than they can be processed. This can also lead to a high latency in responding to requests. Delay is the estimated time (in milliseconds) the next request will spend in the queue prior to being serviced by the directory service, 1.265 seconds.

So as the stress load increased we see that the delays in processing goes up – perhaps to the point of failure?

By default there are 4 threads per processor allocated to the LDAP thread pool, we can change that via LDAP policies, specifically MaxPoolThreads:

·         MaxPoolThreads = Maximum number of threads created by the domain controller for query execution (4 per processor)

Theory at this point was that we were clogging up the thread pool and more throughput here would improve performance as I did not believe we were not constrained by Disk\CPU\Memory on the systems. Processor, Disk IO counters, and Database (\Database(lsass)\Database Cache % Hit)  give us some insight to my reasoning here.

We increased the threads to 8 per proc on  a test DC and re-ran the ldap stress.

CPU usage is still within normal ranges ~20%.

We have lowered queue delay and increased ( nearly doubled ) DS search operations ( previous stress  is on left , post 8 threads is on the right , note we also had the VM at one proc previously and this is now two)

At this point we wait for the next morning to hit, with Perfmon ready to go, SPA ready to go, and ready to get some LSASS.EXE dumps via procdump.exe if the issue occurs again.

Waiting…

Waiting…

POW!

Arghh!!

Sure enough, Exchange goes down in spectacular flames… 

How. Disappointing.

We got the SPA data, LSA dumps etc. and in hindsight, some of the DCs had not been rebooted after making the LDAP policy change and so were still at the default 4 threads.

We examine the actual outage perf data and sure enough, we see that the queue delay is quite high, up to ~4 seconds as seen in the perf graph below.

But why? Why can this DC not handle the throughput here? Typically I see CPU increase when a DC in under serious load, and these DCs weren’t breathing hard.

The SPA data also showed a large number of LDAPPing requests which took ~2.2 seconds:

In addition, the raw SPA data showed more information ( client addresses )

ELAPSEDTIME=2555.5163:CPUTIME=0:LDAP

      CLIENT: 10.10.10.87:1666

  ENCRYPTION: None

   TRANSPORT: UDP

        UNK0: 3

      EXITID: 0

         DSE: LDAPPing

      RESULT: Success

        TYPE: 4

    INSTANCE: NTDS

    INFLIGHT: 8

Now this was a bit interesting…Let’s look at the LSASS process dump.

I saw all 8 ATQ  threads and all of them were in this state:

Ahh very interesting. 

Netlogon of course, processes DC locator and DSget* API’s. So, let’s recap what we have here.

·         Exchange servers fall over from LDAP delays

·         DCs have a default of four threads processing data

·         DCs are not CPU constrained and LDAP queries are not expensive or inefficient

·         DCs have some clients doing a DC discovery and sending the standard “LDAPPING” as documented in the [MS-ADTS]: Active Directory Technical Specification, you can see this in a network trace as well

·         DCs threads are all gummed up in waiting to resolve name?

Upon further investigation we see the following in the DCs netlogon logs:

SUMMARY:

At the end of the day, after some additional code review and proving it out in a lab,  the DC is trying to resolve the site to client mapping and if it does not find a subnet for the client, and can not resolve the name in DNS, it will try NetBios which takes ~2.5 seconds to timeout and a the number of calls from clients which fail in this manner, combined with the number of LDAP requests Exchange needed just to handle the morning logons –   plugs the few default ATQ  threads with client LDAP pings working through netbios name resolution.

The workaround in this case was to disable Netbios on the DC but this may not be an option for everyone, so the site subnet mapping or DNS name resolution should also fix this kind of an issue.

KEYWORDS: LDAP search hang delay netbios ATQ  LDAP timeout

So there was an interesting case which floated my way the other day.

The Audit policies in the domain controllers policy was set to the following, and there were no other policies blocking or changing these.

After a policy update the following events were logged:

In addition, auditpol /get /category:* simply would show no auditing after policy update:

So,  where was this crazy thing being overwritten? It wasn’t in the policies, since we checked all of them carefully for inheritance etc..

Looking at where a client actually stores audit policy may give us a clue (C:\Windows\system32\grouppolicy\machine\microsoft\windows nt\audit\audit.csv  and C:\Windows\security)

But there was nothing there of interest.  So, the last place to look was the sysvol data:

M:\SYSVOL\domain\Policies\{CEF3323C-FD89-4C03-9410-18F7A4922E5A}\Machine\microsoft\windows nt\Audit

Aha! Under here was the .CSV file with the headings  - but no configuration data in it!

We removed this file and now audit policies flowed properly to the DCs and audit event were generated.

Odd. It turns out that they had applied the policies via a GPOBackup and perhaps something had occurred prior to the backup.

Anyway – hope it helps someone someday

Spat

spat

OK – time for the random Monday morning blog.

So I was working on some FIM CM SQL API testing and was hitting an error calling 'externalSubmitSuspendRequest'

So – based on some limited past experience I thought “Hrmm no DB access, well I do recall that the clmApp role would fix this..” so I granted him this role.

Sure enough – ran it again and the account could get past that error, only to hit another.

C:\Spat.CLM.App\FIMSQLapis - suspend\FIMSQLapis\bin\Debug>FIMSQLapis.exe
{29a06d65-0157-5113-2891-44153e24ffff}
MSBaseCSP
Active
---------------

Unhandled Exception: System.Data.SqlClient.SqlException: The EXECUTE permission was denied on the object 'externalSubmitSuspendRequest', database 'FIMCertificateManagement', schema 'dbo'.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader()
   at FIMSQLapis.Program.Main(String[] args) in C:\Users\administrator.SPATFIM\Documents\Visual Studio 2008\Projects\FIMSQLapis - suspend\FIMSQLapis\Program.cs:line 168

Now - based on http://msdn.microsoft.com/en-us/library/bb468093(VS.85).aspx   it sounded like all that is needed was clmExternalAPI but it still does not create the external request. Continued errors.

Now sometimes you just need to talk to someone ( sometimes yourself -- out loud ) in order to walk through the logic. This person doesn’t even need to be familiar with the code , or technology – just a sounding board sometimes.

That someone wandered in at just the right time :

Someone: “Whatcha working on , how was your weekend?”
Me: “Weekend was good , working on this access denied error” ( show him SQL traces etc.. )
Someone: “Hrmm interesting. You sure he isn’t denied somewhere else?”
Me: “Huh? Denied no I didn’t den—oh wait. Hrmm… aha!! You sir are a genius!”

Looking at the stored procedure I was calling shows that the clmAp role has an explicit deny on execute. Kapow! Thx Odin.

Posting this here in case someone else is bitten by it.. now off to complete my suspend code….

Spatdsg