SQL Server in Enterprise

Check the Available Memory of Default Pool

Shiyang Qiu — Thu, 30 Mar 2017 18:03:24 GMT

One capacity planning best practice of using SQL In-Memory OLTP engine is to bind the database with a resource governance pool. The calculation of memory resource of pools is from here, and I translate it into below TSQL code to check both configured value and available value in run time. [sql] declare @MinMemoryPercent decimal(4,3) declare @MaxAvailableMemoryKB bigint declare @InusedMemoryKB bigint declare @ResPoolMinMemoryCapKB bigint declare @AvailableSharedMemoryKB bigint select @MinMemoryPercent = sum(min_memory_percent)/100, @InusedMemoryKB = sum(used_memory_kb) from sys.dm_resource_governor_resource_pools select @MaxAvailableMemoryKB = available_physical_memory_kb from sys.dm_os_sys_memory set @ResPoolMinMemoryCapKB = @MinMemoryPercent * @MaxAvailableMemoryKB if(@InusedMemoryKB>@ResPoolMinMemoryCapKB) set @AvailableSharedMemoryKB = @MaxAvailableMemoryKB - @InusedMemoryKB else set @AvailableSharedMemoryKB = @MaxAvailableMemoryKB - @ResPoolMinMemoryCapKB select @MaxAvailableMemoryKB/1024.0 as [Max_Available_Memory_MB] ,@InusedMemoryKB/1024.0 as [Inused_Memory_MB] ,@ResPoolMinMemoryCapKB/1024.0 as [Resource_Pool_Min_Memory_Cap_MB] ,@AvailableSharedMemoryKB /1024.0 as [Available_Shared_Memory_MB] [/sql] ------------------------------------------- Posted by Shiyang Qiu, 2017 Mar 30 Thanks to Frankie Lai for peer review.

SQL16: Database in Delete-Only Mode

Shiyang Qiu — Thu, 23 Mar 2017 16:19:28 GMT

What will happen when the disk of In-memory OLTP checkpoint file pair is full in SQL Server 2016? Here is the learning from a recent test: #1 Change in SQL 2016 This is a change in SQL Server 2016. We will mark the database in delete-only mode (DOM) for memory-optimized tables when we are out of storage space. In SQL Server 2014, the user database will be suspended, and now it could still be online. SQL Error 41822 will be raised when new insert is triggered. Msg 41822, Level 17, State 1, line 5 There is insufficient disk space to generate checkpoint files and as a result the database is in delete-only mode. In this mode, only deletes and drops are allowed. #2 How to tell database is in a delete-only mode In current version of SQL Server, you can check by sending a non delete statement to user database. #3 How to resume the insert After you deleting records or freeing more disk space, you can run a manual CHECKPOINT command to bring database back to normal from Delete-Only Mode. Or it will be back to normal by next auto CHECKPOINT. Posted by Shiyang Qiu, Mar 23, 2017

DBCC CLONEDATABASE can not Copy Encrypted Objects

Shiyang Qiu — Fri, 18 Nov 2016 14:31:28 GMT

There is a new command DBCC CLONEDATABASE from latest SQL Server 2014 SP2, SQL Server 2016 SP1 release. It's designed to collect the database objects definition, index statistic and query store info without table data. It's useful when support team need to investigate the performance issue like suboptimal execution plan remotely. Using DBCC CLONEDATABASE could keep all index stats and query store beside the real production data which may be sensitive or with huge size. It may be used to keep a version of database without index stats or query store info. For more detail you could check KB 3177838. Beside what has been documented in KB 3177838, there is one known non supported object, the encrypted objects. There will be no warning when you run the command, the encrypted objects will be copied but there will be error when calling the encrypted objects in cloned database. Make sure you provide the decrypted plain text if that object what you need to investigate or keep the source in other way. Posted by Shiyang Qiu, Nov 18, 2016 Special thanks to Parikshit Savjani for KB 3177838 revision

SQL16 Full Backup Size Difference on Primary Replica and Secondary Replica

Shiyang Qiu — Sun, 16 Oct 2016 15:56:45 GMT

Database full backup could be taken from Always On Availability Groups Active Secondary with COPY_ONLY option. In a recent SQL2016 project, we observe that the copy only full backup size are different when taking backup from Primary Replica and Secondary Replica.

Symptom

We configure 3-nodes AlwaysOn AG with SQL Server 2016 CU1 64bit. Synced mode is used for every replica. The status of each replica is synchronized. Before full backup task, there are many data modification. There is no other workload during the whole test. The backup file size and backup duration are different. When we analyze the backup statement output, we find out the number of processed page for log on Primary is much larger than on Secondary. Copy_only Full Backup Statement Output on Primary Replica Processed 6216 pages for database 'testing_db', file 'testing_db_data' on file 1. Processed 3979919 pages for database 'testing_db', file 'testing_db_dir' on file 1. Processed 16333281 pages for database 'testing_db', file 'testing_db_log' on file 1. BACKUP DATABASE successfully processed 20319416 pages in 170.540 seconds (930.839 MB/sec). Copy_only Full Backup Statement Output on Secondary Replica Processed 6216 pages for database 'testing_db', file 'testing_db_data' on file 1. Processed 3973037 pages for database 'testing_db', file 'testing_db_dir' on file 1. Processed 1334436 pages for database 'testing_db', file 'testing_db_log' on file 1. BACKUP DATABASE successfully processed 5313689 pages in 40.881 seconds (1015.464 MB/sec).

Why the Processed Pages of Log are Different

Data pages and log pages are copied into the backup set when full backup are being executed. The main reason of copying log is to make sure we could restore a backup file to get a transactionally consistent database. And a full database backup contains more than the transaction log from the start of the backup to the end of the backup. After reviewing the code of current SQL Server, here is how SQL Server 2016 decides which range of log a full backup will contain:

Full Backup Type:	The start LSN of the log included in a database backup is the minimum of:
Normal Full Backup on Primary Replica:	m_ckptLSNm_OldestPageLSN m_oldActXact m_replLSN m_dbMirroringLSN m_HostLogLSN m_HkRecoveryLSN m_HADRonLSN
COPY_ONLY Full Backup on Primary Replica:	m_ckptLSNm_OldestPageLSN m_oldActXact m_replLSN m_dbMirroringLSN m_HostLogLSN m_HkRecoveryLSN m_HADRonLSN
COPY_ONLY Full Backup on Secondary Replica:	m_ckptLSNm_OldestPageLSN m_oldActXact m_replLSN m_dbMirroringLSN m_HostLogLSN m_HkRecoveryLSN

SQL Server will calculate the start LSN of the log included in a database backup by above table. Normal full backup and copy_only full backup count same set of internal LSN. Comparing to copy_only full backup on Secondary, m_HADRonLSN is included on Primary. This is the reason why backup size on Primary is larger than Secondary. m_HADRonLSN is mainly to record the min LSN that has not been shipped to replicas. m_HADRonLSN is the min of all the secondary truncation points which include all the truncation points. From SQL 2016, m_HADRonLSN is also used in the Secondary fast seeding. Taking log backup will also move m_HADRonLSN forward when all the secondary replicas are in synced status.

Suggestion

Basing on the finding, if there is no differential backup in your backup plan, you can take COPY_ONLY full backup on Secondary to get better backup size. If the full backup is scheduled on Primary (when you need differential backup or have no license for readable secondary feature), you could to also schedule the log backup first to move m_HADRonLSN forward. Posted by Shiyang Qiu, Oct 16, 2016 Special thanks to Simon Su for the SQL source code review.

Reporting Service Execution Log Retention

Shiyang Qiu — Mon, 12 Sep 2016 16:25:54 GMT

The default logging retention value is 60 days in SSRS. If you want to modify the retention period, you could use SQL Server Management Studio to connect to SSRS instance, check the Advanced page of Server Properties, and change the value of ExecutionLogDaysKept. From the GUI it reads “Valid values for this property include -1 and 0 through 2,147,483,647. If the value is 0 entries are not deleted from the Execution Log Table.” SSRS Product Group confirm there is a known issue that only -1 will keep the log indefinitely. Value 0 means keep 0 day of log; the log will be deleted when the retention job is trigger. Every day at 2AM local time, the retention job will be triggered from each SSRS server to archive the log. Stored Proc ExpireExecutionLogEntries in SSRS catalog database is called. If you change the SSRS web/app server time zone setting, you should restart SSRS web/app server to make the retention job running in the right time. Let’s check the coding of this stored proc: [sql] if exists (select * from ConfigurationInfo where [Name] = 'ExecutionLogDaysKept' and CAST(CAST(Value as nvarchar) as integer) = -1) begin return end delete from ExecutionLogStorage where DateDiff(day, TimeStart, getdate()) >= (select CAST(CAST(Value as nvarchar) as integer) from ConfigurationInfo where [Name] = 'ExecutionLogDaysKept') [/sql] You will see only -1 could keep the log indefinitely. For a heavy production environment, we don’t recommend you keeping the log forever. Because it will increase the size of SSRS’s catalog database. In my testing environment, size of log table with 1,572,866 log entries is 900MB. When the retention job is running to delete huge number of log, it takes time and there will be exclusive table lock on table ExecutionLogStorage. No report generation will be done before the lock is released. In my testing environment (Azure Standard D2 v2 VM), it takes 112 seconds to delete 1,572,866 log entries. And it causes report execution error. So you should avoid schedule the regular report subscription at 2AM. Posted by Shiyang Qiu, Sep 12, 2016 Special thanks to Chris Finlan

SQL16 Reporting Service Execution Log for Mobile Report

Shiyang Qiu — Wed, 31 Aug 2016 17:45:47 GMT

Update: SQL 2016 SP1 makes the change! One area of feedback we rec’d was around the inability to see any logs in the report catalog related to specific mobile reports. With this release, you now can see basic information in the execution logs for mobile reports, including the folder path, report name, user who ran the report and when they ran it. https://blogs.msdn.microsoft.com/sqlrsteamblog/2016/11/16/whats-new-in-sql-server-2016-sp1-for-reporting-services/ I feel proud that my feedback makes SQL better. Shiyang Nov 17, 2016 ******************************************************************************************** SQL Server Reporting Services provide a built-in log function to store rich information of execution history of the report. That information could be used to create performance baseline; trouble shoot performance etc. You could check more information from: Report Server ExecutionLog and the ExecutionLog3 View.

What's new for SQL 2016 Mobile Report Execution Logging

From SQL 2016, Microsoft introduce a new type of mobile report which bases on Datazen. In SQL 2016 RTM, the execution logging granularity of mobile report is embedded server side dataset. It's different from paginated report execution logging, which logging granularity is report. Every time we generate a mobile report with data refresh, for every dataset inside the mobile report, SSRS will process the dataset from data source, then render it as JSON to client app. There will be 2 log records, the 1st record has no value of ItemPath, which Format is SHAREDDATASETJSON. The 2nd record has no value of Format, which ItemPath is the path of dataset, TimeRendering is always 0. They share the same ExecutionId.

Current recommendation

My recommendation is to apply the naming rule of dataset: put mobile report name as prefix. Then you could find out the execution basing on the name of dataset and ExecutionId. Below is one example: [sql] use ReportServer go select [InstanceName] ,max([ItemPath]) as [ItemPath] ,[UserName] ,[ExecutionId] ,min([TimeStart]) as [FirstTimeStart] ,max([TimeEnd]) as [LastTimeEnd] ,max([TimeDataRetrieval]) as [TotalTimeDataRetrieval] ,sum([TimeProcessing]) as [TotalTimeProcessing] ,sum([TimeRendering]) as [TotalTimeRendering] ,sum([ByteCount]) as [TotalByteCount] ,max([RowCount]) from dbo.ExecutionLog3 group by [InstanceName],[ExecutionId],[UserName] having max([Format]) = 'SHAREDDATASETJSON' order by [FirstTimeStart] asc go [/sql] Posted by Shiyang Qiu, Aug 31, 2016 Special thanks to Chris Finlan

Data Warehouse, Clustered Columnstore Indexes and SAS

Shiyang Qiu — Fri, 12 Aug 2016 14:50:50 GMT

Clustered Columnstore Indexes (CCI) is a game changer for data warehouse workload, which is available from SQL Server 2014 Enterprise Edition. I worked for a PoC project last year to adopt Clustered Columnstore Indexes for data warehouse. The PoC method is simple but the improvement is very significant. Below is some detailed.

The Pain and the Result

Customer run a 353GB data warehouse on SQL 2008 R2, max memory is 20GB. The ELT tool is SAS, SSAS cube is built on top of data warehouse. The SSAS model is changed frequently and full process is ran every day due to their dynamical business nature. As the physical memory size is small, there is heavy IO loading to storage during data process and customer plan to increase physical memory to 200GB. By changing the top 30 tables to CCI tables, the database size reduces from 353GB to 75GB, the largest table size reduces from 46GB to 4.5GB! The duration of daily ELT reduces from 35 seconds to 121 seconds when the SQL max memory is 20GB, reduces from 292 seconds to 49 seconds when the SQL max memory is 200GB. The duration of cube process reduces from 35 min to 15 min when the SQL max memory is 20GB, reduces from 21 min to 14 min when the SQL max memory is 200GB. So we are happy to see the huge improvement after adopting CCI without new hardware expense.

Migration Method

Step 1: Find out top tables

In this PoC there is no unsupported data type, no unique key, primary key or foreign key in every fact table as it's a well design data warehouse, I use below SQL find out top 30 tables in terms of size: [sql] use [db_name] go select top 30 i.object_id, t.name as table_name, s.name as [schema_name], p.rows as rowc_ounts, sum(a.total_pages) / 128.0 as total_space_mb, sum(a.used_pages) / 128.0 as used_space_mb, (sum(a.total_pages) - sum(a.used_pages)) / 128.0 as unused_space_mb into #t_top_tables from sys.tables t inner join sys.indexes i on t.object_id = i.object_id inner join sys.partitions p on i.object_id = p.object_id and i.index_id = p.index_id inner join sys.allocation_units a on p.partition_id = a.container_id left outer join sys.schemas s on t.schema_id = s.schema_id where t.name not like 'dt%' and t.is_ms_shipped = 0 and i.object_id > 255 group by i.object_id,t.name, s.name, p.rows order by total_space_mb desc select * from #t_top_tables [/sql]

Step 2: Generate T-SQL to drop all existing indexes and create Clustered Columnstore Indexes(CCI)

Below T-SQL is ran in same session in generate drop all existing indexes and create CCI statement: [sql] declare @schema sysname, @table sysname, @index sysname, @sql nvarchar(4000) declare cur cursor for select o.schema_name, o.table_name, i.name as index_name from sys.indexes i inner join #t_top_tables o on i.object_id=o.object_id where (i.index_id>0 and i.index_id<255) and i.name not like '_WA_Sys_%' open cur fetch next from cur into @schema,@table,@index while @@fetch_status=0 begin set @sql='drop index ['+@schema+'].['+@table+'].['+@index +']' select @sql fetch next from cur into @schema,@table,@index end close cur deallocate cur select 'CREATE CLUSTERED COLUMNSTORE INDEX cl_'+table_name+' ON '+schema_name+'.'+table_name from #t_top_tables drop table #t_top_tables [/sql]

Step 3: Run the generated script

Before you run the generated script in production, I recommend you to test in testing environment to measure the duration. Another suggestion is to turn database recovery model to simple to avoid growth of t-log.

SAS ODBC Error

Customer hit below error when they testing the SAS code: ERROR: CLI describe error: [Microsoft][ODBC SQL Server Driver][SQL Server]Cursors are not supported on a table which has a clustered columnstore index. : [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. The statement is a simple select, but SAS translates it to a cursor. With SAS tech support, we finally find out this is related to a SAS optimization configuration: Block read buffer size. Below is how to set the READBUFF=0 in SAS Management Console: 1. Right click the Library in SMC -> Properties -> Options -> Advanced Options – > Optimization -> Block read buffer size 2. set this property to 0. https://support.sas.com/documentation/cdl/en/bidsag/61236/HTML/default/viewer.htm#a002668484.htm Posted by Shiyang, Aug 12 2016

%ErrorLevel% and $LASTEXITCODE

Shiyang Qiu — Wed, 10 Aug 2016 13:40:35 GMT

This is the follow-up post of the previous PowerShell and Control-M. There is one scenario I haven't discussed: how to get the cmd.exe batch result from PowerShell. This is not a new question but there is no clear answer you could find in Internet. Here is a post from PowerShell team: ErrorLevel equivalent, you could read that $LASTEXITCODE is designed as equivalent to %ErrorLevel% In Cmd.exe, %ErrorLevel% is a builtin variable which indicates the success or failure of the last executable run. $LASTEXITCODE, Contains the exit code of the last Win32 executable execution.

Problem

With the cmd.exe batch, the observed result of %ErrorLevel% and $LASTEXITCODE are different in some cases. %ErrorLevel% would not be changed for the "internal" batch command like "GOTO", but $LASTEXITCODE result will be changed. Below is one example, which is a command batch template with error handling in the old days. [vb] @echo off ping %1 if %errorlevel% NEQ 0 GOTO :Error echo %1 is reachable. GOTO :End :Error echo Error happens, %1 is not reachable. :End [/vb] If the batch is ran in cmd.exe, 127.0.0.1 and 999.999.999.999 are passed as parameter, expected %ErrorLevel% result 0 and 1 are returned. But the $LASTEXITCODE is always 0 as the ping command is not the last command in the batch file.

How to address

We can add one exit code as the last commend in the batch file to align the $LASTEXITCODE result same as %ErrorLevel%. Below is the updated version of the sample: [vb] REM Sample 2, with exit line @echo off ping %1 if %errorlevel% NEQ 0 GOTO :Error echo %1 is reachable. GOTO :End :Error echo Error happens, %1 is not reachable. :End exit /b %errorlevel% [/vb] Now $LASTEXITCODE result is always same as %ErrorLevel%. So to the PowerShell + Control-M combination, we will need to add exit /b %errorlevel% to the cmd wrapper template for case like PowerShell script recursion. Posted by Shiyang Qiu, Aug 10, 2016

Extract all the connection strings from SSIS packages

Shiyang Qiu — Wed, 03 Aug 2016 19:42:24 GMT

One important task in SSIS upgrade/migration project is the assessment of data provider types, source and destination connection strings in the all packages. The output is used to understand which data source and destination are involved, which data providers are in used. You could use SSDT to check SSIS package but you need a quicker way if the number of package is huge. In my recent project, the number of SSIS package is larger than 600, and below is PowerShell method I use to retrieve the information in 5 min.

Step 1: Export SSIS packages from msdb

The first step is to store all SSIS packages in one main folder. I use the PowerShell from Jamie Thomson to export SSIS packages from msdb. Here is the link: Export all SSIS packages from msdb using Powershell

Step 2: Retrieve the data providers and strings

Once the packages are stored in the folder, said "C:\ssis_data_provider_check", you could use below PowerShell to dig out information of data providers and connection strings. [ps] ############################################################ ## Retrieve the data providers and strings ## ## Version 0.1 ## Created Date 3 Aug, 2016 ## Created By Shiyang Qiu (shiyang.qiu@microsoft.com) ############################################################ cls # $path is the folder of SSIS pacakage $path = "C:\ssis_data_provider_check" Push-Location $path #$all_type_file is the txt file to store the type of data provider $all_type_file = "$path\all_provider.txt" #If the file exists, remove it if(Test-Path $all_type_file) {Remove-Item $all_type_file -force} #Findout the all provider types, each data type will be one line in the $all_type_file $Lines = dir -I *.dtsx -R | select-string -Pattern "Provider=" foreach ($L in $Lines) { #RegExp is used to extract the name of data provider $L.Line -match '(Provider=)\w+' | Out-Null; $Matches[0] >> $all_type_file } #Find out all the data connection strings and store into text file for each data provider. $all_type_array = get-content .\all_provider.txt | sort| Get-Unique foreach($type in $all_type_array) { dir -I *.dtsx -R | select-string -Pattern $type | Format-Table -auto -Property path, lineNumber, Line | out-string -Width 4096 | out-file $path\$type.txt } [/ps] Here is the result sample: -- Posted by Shiyang, Aug 3, 2016

PowerShell and Control-M

Shiyang Qiu — Tue, 02 Aug 2016 16:10:42 GMT

PowerShell is the best scripting tool for various tasks of Microsoft products, Control-M is one popular automation tool in enterprise. But the current Control-M could not call PowerShell script directly. I work in a customer project recently to migrate their legacy database maintenance jobs from dos batch to PowerShell, which scheduled by Control-M. Here are my findings and sample code to address the challenges. PowerShell version I use is PowerShell 5.0. You could use below PowerShell command to check the version in your environment. [powershell] $PSVersionTable.PSVersion [/powershell]

Findings and Solutions

Finding	How to address
Control-M could not call PowerShell directly.	Cmd batch "wrapper" will be used to call PowerShell script.
Absolute path is needed in the cmd batch wrapper to call the PowerShell script.	Put the cmd batch wrapper and PowerShell script in same folder, and use %~dp0 to set the working folder.
The PowerShell Try-Catch block could not capture all the errors in default setting. There are 2 types of errors in PowerShell, non-terminating errors and terminating errors. Only terminating errors will stop the execution and raise exception. And the Try-Catch block will then catch exception. Ref: https://technet.microsoft.com/en-us/magazine/2009.01.windowspowershell.aspx	Set $ErrorActionPreference to change all non-terminating errors to terminating errors. Use Try-Catch block for the central error handling.
The way to pass back execution result to caller. For PowerShell, there are "return" and "exit" option. The "return" won't stop the execution of PowerShell, the "exit" will. The challenge is how PowerShell script call other PowerShell script or itself as recursion.	"return" is used for function call, "exit" is used for wrapper call. Inside PowerShell script, the call to other PowerShell script is also through the wrapper.
How to combine with other enterprise monitoring tool like SCOM.	Define your own error codes and write them to event log.
How to control the debug information output.	Add a switch parameter, but it has to be int data type as switch type value($true or$false) is not supported by from cmd.

Sample Code

Batch Wrapper

[vb] @echo off setlocal REM ********************************************************************************** REM * Name : control_m_demo.bat REM * For : Wrapper of control_m_demo.ps1 REM * By : Shiyang Qiu (shiyang.qiu@microsoft.com) REM ********************************************************************************** REM * There are 2 parameters, if any parameter is missing go to help if "%1" == "" goto :HELP if "%2" == "" goto :HELP REM * Set the absolute path SET SET_FOLDER=%~dp0 pushd "%SET_FOLDER=E%" REM * control_m_demo.ps1 and control_m_demo.bat are in same folder REM * %3 parameter is a hidden switch to control the output of debuggig info powershell -file control_m_demo.ps1 %1 %2 %3 goto :END :HELP echo. echo Command usage: use control_m_demo [para1] [para2] to check if para1/para2 < 1 echo. echo [para1] = parameter 1 echo [para2] = parameter 2 echo. :END popd endlocal exit /b %errorlevel% [/vb]

PowerShell Script

[ps] ############################################################ ## File Name: control_m_demo.ps1 ## ## Version 0.1 ## Created Date 2 Aug, 2016 ## Created By Shiyang Qiu (shiyang.qiu@microsoft.com) ############################################################ param ( [Parameter(Mandatory=$true, Position=1)] [int]$Para1, [Parameter(Mandatory=$true, Position=2)] [int]$Para2, #Modify the switch to 0 when deploying the script. 1 is for dev and t-shooting [Parameter(Mandatory=$false, Position=3)] [ValidateSet(0,1)] [int]$_Test = 1 ) ## Change the default error action to "Stop" $ErrorActionPreference = "stop" ## Change the working folder Push-Location $PSScriptRoot ### Function PrintTestInfo is used to print debug info to screen function PrintTestInfo($S) { write-host "$(get-date -Format yyyy-MM-dd_HH:mm:ss) Debug output: $S" -ForegroundColor Yellow } ### Function IfDivResultLessThan1 is to calc para1/para2 function IfDivResultLessThan1 { ## when para2 is 0, it will trigger error $Result = $Para1/$Para2 ## return 1 when result < 1 ## return 0 when result >=1 ## error will be captured by outsdie try-catch block if ($Result -lt 1) {return 1} else {return 0} } ########## Begin of Main Logic ###################### Try { if($_Test) {PrintTestInfo("begin the demo")} if(IfDivResultLessThan1 -eq 1){PrintTestInfo("$para1/$para2 < 1")} else {PrintTestInfo("$para1/$para2 >= 1")} ## Exit 0 to wrapper when there is no error if($_Test) {PrintTestInfo("without error")} exit 0 } catch { if($_Test) { $ErrorMsg = $Error[0].ToString() $ErrorInvocationInfo = $Error[0].InvocationInfo.line PrintTestInfo("Error info: $ErrorMsg.") PrintTestInfo("Command: $ErrorInvocationInfo.") } ## Exit 1 to wrapper when there is error exit 1 } ########## End of Main Logic ###################### [/ps] The result HTH, Shiyang Special thanks to David Ho for brain storming this solution. --------- Aug 10 Update: https://blogs.msdn.microsoft.com/shiyangqiu/2016/08/10/errorlevel-and-lastexitcode/