Just another module from my kernel.... https://docs.microsoft.com/archive/blogs/paulocondeca/feed.xml Wed, 17 Apr 2019 03:51:53 GMT en-US hourly 1 Leveraging Windows Server 2016 S2D with SQL Server 2016 to build hyper-converged clusters for the "Automated Datacenter" https://docs.microsoft.com/archive/blogs/paulocondeca/leveraging-windows-server-2016-s2d-with-sql-server-2016-to-build-hyper-converged-clusters-for-the-automated-datacenter Wed, 26 Apr 2017 13:37:03 GMT https://blogs.msdn.microsoft.com/paulocondeca/?p=85 Support policy for Microsoft SQL Server products that are running in a hardware virtualization environment > Hello! With the release of Windows Server 2016 and the "great/awesome/fabulous feature" storage spaces direct(S2D) that has the concept of leveraging local storage to build a "virtual-san", with different storage type from SAS/SSD/NVMe to generate different storage tiers/caching that can be both applicable on the cloud and on premises, this is game changer for SQL, both from a deployment perspective(automation) and cost vs performance ! How could one start to know a little bit more about this feature of Windows Server 2016?  Start Here. Architecture: 1

Lab: The first steps towards building our SQL Server Always On Failover Cluster Instance with powershell!

In the cloud (Microsoft Azure): Make sure we have our SQL Server VMs running in the same availability set to handle planned and unplanned maintenance and for the VMs not to run on the same rack/ network switch, meaning each VM will be assigned to an Update Domain (UD) split across Fault Domains (FD) : 2 
New-AzureRmAvailabilitySet -ResourceGroupName PauloCondecaInfraRG01 -Name SQLServerFCIAvailabilitySet01 -Location "westeurope"
After the Availability Set created, it is time to create our storage account with Premium Storage .. 
#PremiumStorageAccount
$newstorageaccountname = "paulo1337storage"
New-AzureRMStorageAccount -StorageAccountName $newstorageaccountname -Location westeurope -Type "Premium_LRS"
Now, its time to install our Cluster Nodes on the top of this storage account: 3 Create Disks (Premium // SSD): With Powershell Remoting, manage both servers from a management/jump box and run the bellow to add the machines to the data Domain, Windows Features : 
# On both servers:
 POWERCFG.EXE /S SCHEME_MIN
 Add-Computer -DomainName data.local -Credential paulo
 Restart-Computer -force 
 #Add Windows Features to our environment 
 Install-WindowsFeature –Name Failover-Clustering,File-Services –IncludeManagementTools 
 New-Cluster -Name CloudSQLServer1337 -Node CloudSQL1337N1,CloudSQL1337N2 -StaticAddress -NoStorage
 #Enable Storage Space Direct on the cluster
 Enable-ClusterS2D
At this stage, we have all the features enabled, lets add some SSD drives to the servers, note here, that the size of the disc will influence the performance of the disk itself. A disk <128 GB of Size will translate to a maximum throughput of 100 MB/s while a disk > 512 GB will translate to a maximum throughput of 250 MB/s , more details about this P10/20/30 in https://docs.microsoft.com/en-us/azure/storage/storage-premium-storage pstorage After having a couple of disks created, it is time to start creating the VirtualDisks to be used by SQL Server 5 6 We can use this storage to install our SQL Server 2016 Failover Cluster Instance, independent of it being on premises or cloud! Information about disk distribution by enclosure: 7   Doing a test with Diskspd, representative of what could be an OLTP Workload (75% reads + 25% writes) could easily reach a 250 MB/s throughput with a single file; with more parallel executions and different configurations, this numbers could sky-rocket but I would leave that for another time/article. 8 While this was running, I could see the network traffic between the nodes going on :) inline with the WriteActivity (500 Mbps~62MB/s): 9 And out of curiosity, in one of the workloads, I took one of the nodes offline, just to see the network activity to put the volume in 100% healthy state, and it peaked with 2GBit/s , basically 250MB/s that translate the maximum troughput of a P30 disk. 10 This post is just to basically create awareness about this new Windows Server feature and how we could leverage it to benefit SQL Server implementations, so naturally, there are details that are not covered in this post, such as workloads running that  leverage NVMe+RDMA+S2D for high throughput. All the best! Paulo Condeça]]> Troubleshooting in SQL Server Transactional Replication on *very* busy Subscribers + WindDbg + WPR & WPA https://docs.microsoft.com/archive/blogs/paulocondeca/troubleshooting-in-sql-server-transactional-replication-on-very-busy-subscribers-winddbg-wpr-wpa Wed, 23 Nov 2016 13:37:55 GMT https://blogs.msdn.microsoft.com/paulocondeca/?p=165
  • Publisher to Distributor | Log Reader Delivery Latency = Ok
  • Distributer to Subscriber | Distributor Delivery Latency = Nok
  • MSdistribution_history was throwing state 2 messages (distrib agent reader thread waits on writer thread )
  • REPL Stored Procedures *sp_MSins/upd/del_*** where executing fast (at the microsecond scale)
    • At this point I had all the evidences of the issue being indeed writing to the subscriber but the replication stored procedures were executing fast at the subscriber and the volume of data being replicated hadn't exponentially changed! So... I was intrigued about this... Basically replication was telling me that sql server engine was too slow but sql server was telling me it was executing its stuff fast enough. Well... Lets take full memory dump of DISTRIB.exe and check see if there is anything interesting to find, just took 2 dumps using procdump with 5 minutes delay between them: 1 Looking at the debugger, analyzing the dump, an exercise has to be be done to guess the functionality of the functions based on its name (ok... I have to admit, I miss this part of having source code access and private symbols back when I was at Microsoft to better troubleshoot, but... It can still be done using the public symbols to a certain degree of correctness) One of the first things I do while analyzing dumps is to use "!runaway", "!runaway 2" and "!runaway 4", between the 2 dumps. The main idea is to analyze the delta of user time and kernel time for each thread and see if anything looks wrong. Nothing looked wrong in this 5 minutes delta, i can see that thread 0, 5 and 6 had work to do, with more weight for thread 0 and 6, well, this a sign they are working. 2 Lets check what is this thread 0 and 6, since this are the ones that had higher User Mode Time and Kernel Mode Time. Thread 0, is likely related with the reader thread - sp_MSget_repl_commands that pulls data from the distribution database and stores them in an internal queue. The call stack looks good. 3 Going for Thread 6, this is likely related with the writer thread, basically running the sp_MSIns/Upd/Del in the subscriber database (writing queued commands). The call stack looks good. 6 So at this stage, I can't see any issue with the Distrib.exe (calls rdistcom.dll) process and can't also see any issue with the duration of the Replication stored procedures but while I was analyzing dm_exec_procedure_stats I noticed that the cached time vs last execution time was almost the same! This triggered the investigation to classic "plan cache internals" mode and it was quickly found memory pressure. In this case custom replication stored procedures are being used and although the logic is reduced to a minimum,it takes too much time to compile (sys.dm_exec_cached_plans is your friend here, use it and look for the compile time. Compile time > 1.000.000 microseconds 7 Execution time < 100 microseconds 8 Well,  there are already many blogs and official Microsoft documentation about plan cache internals and memory pressure and also replication troubleshooting. Driven my curiosity, ran a WPR @ TEST ENV, just to see how would a writer thread latency look like under WPA, here it is bellow (basically modified repl procedures by adding a waitfor delay to simulate a slow procedure execution): 9 The resolution for this situation was around changing memory parameters to cover a very specific scenario. Hope you find this reading interesting. All the best! Paulo Condeça.          ]]>         Failed to bring your Mission Critical SQL Server Always On Failover Cluster Online after Patching!? https://docs.microsoft.com/archive/blogs/paulocondeca/failed-to-bring-your-mission-critical-sql-server-always-on-failover-cluster-online-after-patching Mon, 25 Apr 2016 13:37:21 GMT https://blogs.msdn.microsoft.com/paulocondeca/?p=67 https://connect.microsoft.com/SQLServer/feedback/details/2633114 . This type of validations can be easily put into the product and increase the reliability of the rolling-upgrade. Back to root cause analysis, relying on one of my favorite tools to see what registry calls (among other things...) are being made in the OS, the Process Monitor (Sysinternals tools) aka ProcMon. Started to analyze events from the RHS.exe from the time the resource couldn't come online, the tricky part here is that to find the root cause of the issue, doesn't have to necessarily be different from a "SUCCESS state" of a call, meaning that we will need to invest 5 minutes of our time and use our hawk eyes to find the issue around the time the resource failed to come online! Bellow, is the error output from Cluster.Log, looking at the failure string, it is very suggestive of being related with replication(REPL) - DoREPLSharedDataUpgrade. 1 Next action: Start ProcMon, reproduce the Issue (Bring the Cluster Resource Online), stop ProcMon, copy the file (*.PML) to another box and start analyzing the file by filtering process to RHS.EXE to reduce the noise of all activity of the server. Humm... The repldata folder is point to a path that no longer exists...! Fix this to the proper directoy, and bring the cluster resource back online, hopefully there are no further issues and instance is successfully patched - online, Up & Running! Regards, Paulo Condeça.]]> Ninja Troubleshooting- User Interface freezing during SQL Server database restore. https://docs.microsoft.com/archive/blogs/paulocondeca/ninja-troubleshooting-user-interface-freezing-during-sql-server-database-restore Fri, 31 Jul 2015 09:57:00 GMT https://blogs.msdn.microsoft.com/paulocondeca/2015/07/31/ninja-troubleshooting-user-interface-freezing-during-sql-server-database-restore/ Started with gathering performance counters to jump into the analysis and even though there wasn’t clear CPU pressure (all green above :) ), found spikes that matched the UI Freeze on a specific CPU (11) as observed bellow: Basically the Processor % Privileged Time indicates the percentage of time a thread runs in privileged mode also known as kernel mode. When your application calls operating system functions (for example to perform file or network I/O or to allocate memory), these operating system functions are executed in privileged mode. I recommend using the Performance Analysis of Logs (PAL) Tool to assist on performance counter analysis. The PFE (Premier Field Engineering) has the culture to do Root Cause Analysis…So... Let’s dig a little deeper :) , using WPR (Windows Performance Recorder) that is included on Windows Assessment and Deployment Kit and do a trace while the issue is being reproduced… Analyze the trace with Windows Performance Analyzer: First trace, without any change on the Operating System:
    Since we want to focus on % Privilege Time, we want to analyze the DPC Duration by Module and Processor:   Hummm… Interesting… The NDIS.SYS is taking most of compute time, more precisely the function ndisInterruptDpc , that leaded to RSS - Receive Side Scaling (improves the system performance related to handling of network data on multiprocessor systems) Since this is a virtualized multiprocessor system, it was to time check if RSS was enabled on the NIC used for Backup (https://technet.microsoft.com/en-us/library/dn383582.aspx), it wasn’t enabled. At this point, it was clear that this Vcore (CPU 11) wasn’t able to cope with all the Network Related Workload.  Next step, enable RSS on NIC used for backup, reproduce the workload, trace it and analyze again in WPA, the result can be clearly observed in the data bellow:   Conclusion, when RSS is enabled on the Backup NIC, the Network Stack workload is scaled out by 4 CPUs thus reducing the “spike workload” from +/-5 seconds to +/- 1 second spread across 4 CPUS and no longer having UI Freezes. Interesting that VMware also has an article related with this situation:
    https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2008925 “Receive Side Scaling (RSS) RSS is a mechanism which allows the network driver to spread incoming TCP traffic across multiple CPUs, resulting in increased multi-core efficiency and processor cache utilization. If the driver or the operating system is not capable of using RSS, or if RSS is disabled, all incoming network traffic is handled by only one CPU. In this situation, a single CPU can be the bottleneck for the network while other CPUs might remain idle.“ Tools used:
    Performance Counters
    Performance Analysis of Logs
    Windows Performance Recorder
    Windows Performance Analyzer

    Regards,
    Paulo Condeça

    ]]>         Building elastic and fault tolerant Data Platform solutions with Azure, SQL Server and HDInsight https://docs.microsoft.com/archive/blogs/paulocondeca/building-elastic-and-fault-tolerant-data-platform-solutions-with-azure-sql-server-and-hdinsight Wed, 24 Jun 2015 07:19:00 GMT https://blogs.msdn.microsoft.com/paulocondeca/2015/06/24/building-elastic-and-fault-tolerant-data-platform-solutions-with-azure-sql-server-and-hdinsight/ I suggest you to invest 2 minutes of your time to take a look of this session! [embed]https://www.slideshare.net/PauloCondea/building-elastic-and-fault-tolerant-data-platform-solutions-with-azure-sql-server-and-hdinsight-76132032[/embed] -Paulo Condeça]]>