Synchronous Commit Emphasizes Minimizing Data Loss Over Performance

Synchronous commit availability mode is desirable when minimizing data loss is paramount for your high availability solution and prioritizes Recovery Point Objective over performance.

In the following link, it says this about synchronous commit.

Availability Modes (Always On Availability Groups)

Synchronous-commit mode emphasizes high availability over performance, at the cost of increased transaction latency. Under synchronous-commit mode, transactions wait to send the transaction confirmation to the client until the secondary replica has hardened the log to disk.

This blog will demonstrate how to measure the performance impact on a high transaction rate workload that is dependent on auto-commit behavior in SQL Server and how explicitly defining transactions in that same workload can mitigate the impact and provide better performance.

The Transaction Must be Written at the Primary and The Synchronous Secondary To Commit

To illustrate the impact of synchronous commit, take a simple INSERT command initiated on a database at the primary replica. That insert must be hardened on the transaction log file of the primary replica and of the transaction log file on the secondary replica and then acknowledgement must arrive from both replicas before SQL Server can commit the transaction. When performing hundreds of these commands a second in a workload, the impact performance can be significant.

For more internal details on how synchronous commit works, see the following blog:

AlwaysON – HADRON Learning Series:  How does AlwaysON Process a Synchronous Commit Request

Demo – Measure Workload Performance in Synchronous Commit

Let’s demonstrate the impact of auto-commit mode when executing a workload against an availability group database configured for synchronous commit, and how to measure the throughput.

Enable Performance Monitor Counters

Enable the following counters in performance monitor on the primary replica:

For availability group database agdb in our example:
    SQL Server:Databases:Log Bytes Flushed/sec
    SQL Server:Databases:Log Flushes/sec
    SQL Server:Databases:Write Transactions/sec
    SQL Server:Database Replica:Transaction Delay

Run the workload, observe execution time, estimate transaction performance

The following workload inserts 10,000 rows 10 times, and uses auto-commit behavior, which will result in 100,000 transactions. It will report the average time it took to insert 10,000 rows in seconds and the average time it took to insert a single row in milliseconds.

use agdb
go

declare @y int=0
declare @x int=0
declare @starttime datetime
declare @endtime datetime
create table test (col int)
create table results (duration float)

while @x<10
        begin
        Set @starttime =getdate()
                     while @y<10000

                    begin
                            insert test (col)
                            select @y
                            set @y=@y+1
                      end                                      
        set @endtime =getdate()
        insert results (duration)
        select datediff(millisecond,@starttime, @endtime)/1000.0
        set @x=@x+1
        set @y=0
        end

select * from results
select avg(duration) as 'avg duration in sec for 10,000 INSERTs', avg(duration)/10 as 'avg duration in ms for each INSERT' from results

drop table results
drop table test

After we create our availability group and database, and configure it for synchronous commit with a single secondary replica, we run our workload. The report is that our average INSERT took 4.4 ms to complete the round trip, getting hardened on the log file at the primary and secondary replicas. The batch took 7 minutes 50 seconds to complete.

HADR_SYNC_COMMIT waittype reports synchronous commit latency

HADR_SYNC_COMMIT wait type measures the latency when performing transactions in an availability group database when configured for synchronous commit. To reset the instance’s waittype counter values (they accumulate over time), execute the following to clear the data:

DBCC SQLPERF('sys.dm_os_wait_stats' , CLEAR)

Test to demonstrate the severity of this waittype on the instance. There is a large number of these waits reported.

Review the performance monitor counters

Let’s review the performance monitor counters collected during the batch. Log Bytes Flushed/sec for the database is pretty low, less than 1 mb/sec.

Compare Log Flushes/sec to Write Transactions/sec For every write transaction performed, there is a flush to disk. This is what we would expect in auto-commit mode. Here we see 211 transactions are completing every second and for each transaction there is a corresponding flush to disk, on the database log file at the primary replica and on the database log file at the secondary replica.

Confirm estimated transaction time using performance monitor

The workload reported ‘avg duration for each transaction in MS’ as 4.4 ms.  We can use performance monitor to make a second confirmation calculation that this is accurate: divide one second by an average of 211 Write Transactions performed in a second, which is 1/211=.0047 seconds or 4.7 ms. This number is very similar to the number that our workload results reported. This is the time it took to capture the log change, deliver it to the secondary, write it to the log file on the primary and secondary and report back to the client that it is complete.

Do not Depend on Auto-Commit - Define Explicit Transactions

The batch has been modified with an explicit transaction in the workload in such a way that 10,000 INSERT operations will be committed together. This is much more efficient considering the requirements of synchronous commit, that each transaction requires the hardening of the operation to the log files of each replica and the network round trip needed to accomplish that work. Note the BEGIN TRAN and COMMIT TRAN commands added to the batch and commented.

Run the workload using explicit transactions, observe execution time, estimate transaction performance

use agdb
go

declare @y int=0
declare @x int=0
declare @starttime datetime
declare @endtime datetime
create table test (col int)
create table results (duration float)

while @x<10
        begin
        Set @starttime =getdate()
         begin tran                            --BEGIN 10000 row transaction
                     while @y<10000

                    begin
                            insert test (col)
                            select @y
                            set @y=@y+1
                      end                                      
         commit tran                            --COMMIT 10000 row transaction
        set @endtime =getdate()
        insert results (duration)
        select datediff(millisecond,@starttime, @endtime)/1000.0
        set @x=@x+1
        set @y=0
        end

select * from results
select avg(duration) as 'avg duration in sec for 10,000 INSERTs', avg(duration)/10 as 'avg duration in ms for each INSERT' from results

drop table results
drop table test

Our results are almost instantaneous. The results completed in a couple seconds!

HADR_SYNC_COMMIT waittype reports synchronous commit latency

Having reset the SQL Server wait counters using DBCC SQLPERF (described earlier) we see HADR_SYNC_COMMIT waits are significantly lower.

Review the performance monitor counters

Reviewing performance monitor, Log Bytes Flushed/sec spiked to 10.4 mb/sec versus our earlier test in which it averaged less than 1 mb/sec. This shows how much more throughput SQL Server is able to accomplish when not committing a few bytes for individually small transactions.

When comparing Log Flushes/sec to Write Transactions/sec the correlation is not as clear. During batch execution, Log Flushes/sec spike to 181, and Write Transactions are much lower, since the workload itself performed only 10 explicit transactions instead of 100,000.

Conclusion

Synchronous commit can introduce latency to your high transaction workloads, this can be especially impactful in environments that must complete large ETL batches on the weekends or overnight in order to prepare the data for the next business day. Using explicit transactions can optimize workload performance and still take advantage of the strengths synchronous commit offers (eliminate data loss).

If secondary replicas are hosted greater distances from the primary replica, network latency can begin to impact availability group synchronization’s ability to keep the secondary up to date with the changes made at the primary replica.

Symptom - Sustained or Growing Log Send Queue

When network latency becomes an issue the most common symptom you will observe is sustained or growing log send queue. Simply defined, log send queue is the changes stored in the transaction log of your availability group database on the primary replica that have not yet arrived and hardened to the transaction log of the availability group on the secondary replica.

Why should you care? The log send queue represents all the data that can be lost from a catastrophic failure. If your primary replica was lost in a sudden disaster, and you were forced to fail over to the secondary replica where these changes have not yet arrived, your business would incur data loss.

Log send queue is a per-database measurement. Here are ways you can actively monitor your availability group database log send queue.

Add the Log Send Queue size (KB) column in AlwaysOn Dashboard In SQL Server Management Studio Object Explorer, right click on your availability group and choose Show Dashboard. The dashboard will appear in the right pane. Right-click the column header above the list of databases in your availability group and choose to add Log Send Queue Size (KB).

The column is added and reports the current queue size in KB.

Add the SQLServer:Database Replica:Log Send Queue Counter On the secondary replica, add the SQLServer:Database Replica:Log Send Queue counter for the database instance whose log send queue you are monitoring.

Measure Network Latency Impact Using Performance Monitor

In the following demo we measure the impact of network latency on log send queue growth. We can use SQL Server performance monitor counters on the primary and secondary replicas during synchronization of a load which results in log send queue growth. Compression is disabled during this load test in order to put more load on the network.

On the secondary replica, launch Performance Monitor and add the following counters:

SQLServer:Database Replica:Log Bytes Received/sec for appropriate database instance

SQLServer:Database Replica:Recovery Queue for appropriate database instance

On the primary replica, launch Performance Monitor and add the following counters:

SQLServer:Databases:Log Bytes Flushed/sec for appropriate database instance

Network Interface:Sent Bytes/sec for appropriate adapter instance

Once you have Performance Monitor running, initiate the load that leads to log send queuing, and monitor the counters.

Review the performance monitor counters on the secondary, we can see Log Bytes Received/sec for our availability group database is averaging 2.1 mb/sec. Note that the Log Send Queue for the database is continually growing meaning that there are a growing number of changes that have occurred at the primary replica that are stored in the transaction log there that have not yet arrived and hardened on the transaction log of the local database on the secondary replica.

Now, on the primary replica, check the availability group database’s Log Bytes Flushed/sec. We see that our load is generating and hardening on average 7.1 mb/sec to the local transaction log.

Clearly, if we are generating over 7 mb/sec of logged changes at the primary but only receiving 2 mb/sec of changes at the secondary, we can expect log send queue growth. These observations can lead us to begin to consider network latency as a possible cause for our growing log send queue.

On the primary, check the Network Interface:Bytes Sent/sec for the correct network adapter instance. On average, the primary is sending 2.2 mb/sec, and we can expect that nearly all of that is to the secondary replica where we observed comparable Log Bytes Received/sec.

Measure your network throughput independently of SQL Server

In order to better understand how fast an application can push changes to the remote server, use a third party network bandwidth performance tool, like iPerf or NTttcp. The following demo is using iPerf on the same operating environment as the prior demo.

These tools use a default port to communicate with each other, so it may be necessary to open that port, in the case of iPerf, it uses 5201 by default.

IMPORTANT It is important to configure the tool to use a single connection to send data, since SQL Server uses a single connection to send synchronization changes from the primary to the secondary replica.

Test network throughput with iPerf

On the remote secondary replica, launch iPerf with the –s parameter and it will begin to listen for a connection over port 5201.

iperf3 –s

On the SQL Server hosting the primary replica, launch iPerf using the following parameters, and it will begin a transmission with the remote SQL Server.

Here are the parameters.

-c ip address of destination server
–P 1 Use a single connection in the test
–t 10 Perform 10 one second transfers

Here is the command line;

iperf –c <<secondaryipaddress>> –t 10 –P 1

iPerf is reporting it can send between 2.12 and 2.50 mb/sec to the remote secondary server. This rate is very comparable to the rates we observed above in our load test using SQL Server.

Testing a server that is closer to your primary replica can provide a useful foil when doing this test. Below, I run the same test from the primary this time to a destination server in the same data center and subnet as the primary server. Here are the test results, noticeably higher throughput when the destination server is in closer proximity and same subnet.

Test network throughput with NTttcp

Below is a demonstration of using NTttcp.exe in the same environment as above.

On the remote secondary replica (the receiver), launch NTttcp as follows, here –m 1 is for single threaded, and the IP address of the local receiver server.

NTttcp.exe –r –m 1,0,10.1.0.4 –rb 2M –a 16

On the primary replica (the sender), initiate the test by executing the following, again supplying the IP address of the receiver. This test is for 10 seconds (-t 10):

NTttcp.exe –s –m 1,0,10.1.0.4 –l 128k –a 16 –t 10

The results report about 2.7 mb/sec which is similar to the testing results obtained from iPerf and from our performance monitor log.

In Conclusion

This testing can help you to determine if network latency is impacting log send queue for remote asynchronous replicas in your availability group. Log send queuing can impact your recovery point objective (RPO) and as described earlier in this blog translate to potential data loss.

For more information on recovery point objective and the potential for data loss see

Monitor Performance for AlwaysOn Availability Groups

If you find that network latency is having an impact, consider using netsh or another network trace tool to check for retransmits, dropped packets, etc.

Symptoms of Availability Group Database on Primary Replica

Primary Replica Database Reported Not Synchronizing / Recovery Pending

Primary Replica - Attempts to Access Database or Perform DML Fail

Cause SQL Server unable to access database log file

2017-07-13 06:42:44.74 spid1s      Error: 17053, Severity: 16, State: 1. 2017-07-13 06:42:44.74 spid1s      SQLServerLogMgr::LogWriter: Operating system error 21(The device is not ready.) encountered. 2017-07-13 06:42:44.74 spid1s      Write error during log flush. 2017-07-13 06:42:44.75 spid51      Error: 9001, Severity: 21, State: 4. 2017-07-13 06:42:44.75 spid51      The log for database 'testdb' is not available. Check the event log for related error messages. Resolve any errors and restart the database. 2017-07-13 06:42:44.75 spid51      State information for database 'testdb' - Hardened Lsn: '(34:840:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM' 2017-07-13 06:42:44.75 spid23s     Always On Availability Groups connection with secondary database terminated for primary database 'testdb' on the availability replica 'SN2' with Replica ID: {10b9a12b-afed-48f3-b79c-9ecc5cada754}. This is an informational message only. No user action is required. 2017-07-13 06:42:44.75 spid51      Database testdb was shutdown due to error 9001 in routine 'XdesRMFull::CommitInternal'. Restart for non-snapshot databases will be attempted after all connections to the database are aborted. 2017-07-13 06:42:44.77 spid18s     State information for database 'testdb' - Hardened Lsn: '(34:840:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM' 2017-07-13 06:42:44.86 spid18s     Error: 17053, Severity: 16, State: 1. 2017-07-13 06:42:44.86 spid18s     fcb::close-flush: Operating system error (null) encountered.

Resolve Primary Replica Database Not Synchronizing / Recovery Pending

use master go alter database testdb set online go

2017-07-13 06:57:41.97 spid58      Setting database option ONLINE to ON for database 'testdb'. 2017-07-13 06:57:41.97 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM 2017-07-13 06:57:41.97 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM' 2017-07-13 06:57:42.00 spid58      State information for database 'testdb' - Hardened Lsn: '(34:856:1)'    Commit LSN: '(34:824:2)'    Commit Time: 'Jul 12 2017  1:19PM' 2017-07-13 06:57:42.00 spid58      Starting up database 'testdb'. 2017-07-13 06:57:42.17 spid58      2 transactions rolled forward in database 'testdb' (9:0). This is an informational message only. No user action is required. 2017-07-13 06:57:42.27 spid30s     0 transactions rolled back in database 'testdb' (9:0). This is an informational message only. No user action is required. 2017-07-13 06:57:46.98 spid36s     Always On Availability Groups connection with secondary database established for primary database 'testdb' on the availability replica 'SN2' with Replica ID: {10b9a12b-afed-48f3-b79c-9ecc5cada754}. This is an informational message only. No user action is required.

Symptoms of Availability Group Database on Secondary Replica

Secondary Replica Database Reported Not Synchronizing

Secondary Replica – Attempts to Suspend and Resume synchronization Fail

Cause SQL Server unable to access database log file

2017-07-13 07:07:20.01 spid1s      Error: 17053, Severity: 16, State: 1. 2017-07-13 07:07:20.01 spid1s      SQLServerLogMgr::LogWriter: Operating system error 21(The device is not ready.) encountered. 2017-07-13 07:07:20.01 spid1s      Write error during log flush. 2017-07-13 07:07:20.01 spid25s     Error: 9001, Severity: 21, State: 5. 2017-07-13 07:07:20.01 spid25s     The log for database 'testdb' is not available. Check the event log for related error messages. Resolve any errors and restart the database. 2017-07-13 07:07:20.19 spid25s     Error: 3313, Severity: 21, State: 2. 2017-07-13 07:07:20.19 spid25s     During redoing of a logged operation in database 'testdb', an error occurred at log record ID (34:896:1). Typically, the specific failure is previously logged as an error in the Windows Event Log service. Restore the database from a full backup, or repair the database. 2017-07-13 07:07:20.38 spid25s     Always On Availability Groups data movement for database 'testdb' has been suspended for the following reason: "system" (Source ID 2; Source string: 'SUSPEND_FROM_REDO'). To resume data movement on the database, you will need to resume the database manually. For information about how to resume an availability database, see SQL Server Books Online.

Resolve Secondary Replica Database Not Synchronizing

• Do not use a file share that is part of a Distributed File System (DFS) Namespace.

Avoid Data Loss In Your Availability Group Databases

Creating an availability group in order to make your application highly available when running on Azure virtual machines (IaaS) is very popular. The availability group listener requires special configuration steps in Azure as opposed to running on premise. 

For more information on step by step configuration of Azure availability group listener, see the following article for Azure resource managed availability group.

Configure an internal load balancer for an Always On availability group in Azure

The following article will assist in troubleshooting the failure to connect to the availability group listener.

NOTE This troubleshooter focuses on an availability group listener configured with resource managed internal load balancer, which is the most popular type of listener currently deployed.

Testing the Azure availability group listener

To test an Azure availability group listener, do not test the listener connectivity on the primary replica. Instead, test listener connectivity to the primary replica using the listener, from the secondary replica. We know the secondary replica already has certain connectivity with the primary and that there are SQL Server client tools installed on it, that can be used for testing connectivity to the listener. Use a tool like sqlcmd for connectivity testing.

Troubleshooting Failed Azure listener connectivity

Familiarizing yourself with the article that steps through the Azure listener creation and configuration will help you understand the major pieces of the Azure listener configuration that need to be investigated to determine root cause. The majority of all listener connection issues are a result of incorrect configuration of one of the following:

• Firewall or network security group configuration

• IP Clustered resource creation and configuration (See section of article 'Configure the cluster to use the load balancer IP address')

• Internal load balancer creation and configuration (See section of article 'Create and configure the load balancer in the Azure portal)'

Turn off Windows Firewall on the server hosting the primary replica (Firewall and NSG configuration)

The simplest way to ensure Windows Firewall is not blocking connectivity to the listener is to turn off Windows Firewall on the Windows server hosting the primary replica of the availability group, that which you are trying to connect to with your listener connectivity test.

If you find that connectivity succeeds, turn Firewall back on, and be sure to allow inbound connection through ports 1433 (port SQL Server is listening on) and 59999 (probe port).

Investigate Network Security Group (NSG) for rules that block probe port or SQL Server port connectivity (Firewall and NSG configuration)

It is possible that a network security group rule is preventing connectivity to the virtual machine. For example, the following network security group has an inbound rule allowing SQL connectivity but it is overruled by a rule whose priority value is lower.

For more information review the following section 'Configure a Network Security Group inbound rule for the VM' in this article:

Connect to a SQL Server Virtual Machine on Azure (Resource Manager)

Verify RHS.EXE is listening on probe port (59999) (IP Clustered Resource creation and configuration)

On the server hosting the primary replica, verify that RHS.EXE process is listening on the probe port you configured the internal load balancer with.

View the output with notepad and search for a process listening on port 59999 (probe port):

Use Task Manager to verify that the process listening on 59999 is RHS.EXE. If some other process is already listening on port 59999, then it will be necessary to reconfigure both the IP resource's probe port and the internal load balancer probe port to a different value so there is not a conflict.

Review the internal load balancer configuration (Load Balancer creation and configuration)

Using the article on creating the Azure availability group listener, step through the configuration steps 1-4 listed below and use the Azure portal to visually verify the configuration of each step below is correct:

1. Create the load balancer and configure the IP address
2. Configure the backend pool
3. Create a probe
4. Set the load balancing rules

If a visual of the internal load balancer checks out, you can use network tracing to verify that the load balancer is pinging the probe port on the server hosting the primary replica. On the server hosting the primary replica, run netsh from the command line for 30 seconds:

netsh trace start capture=yes tracefile=d:\nettrace.etl maxsize=200 filemode=circular overwrite=yes report=no

To stop the trace run

netsh trace stop

Open the .ETL file using a tool like Network Monitor and filter for probe port activity. No traffic on the probe port of the virtual machine hosting the primary replica indicates one of two possibilities:

1. The internal load balancer is not configured correctly.
2. There is a network security group, firewall or corporate polcy that is interfering with the probe port traffic.

Here we can see that when we filter on port 59999 we see activity which indicates that the load balancer appears to be performing correctly.

Problem Large Send Queue

You discover that the log send queue for a given availability database has grown very large, which threatens your Recovery Point Objective (RPO) and the transaction log has grown very large on the primary replica, possibly threatening to fill the drive. The cause may have been for various reasons: you discover that the availability database synchronization was inadvertently suspended by an administrator, or even by the system and the problem was not identified until now.

Scenario 1 The database is not large

If it is reasonable to re-initialize the secondary replica copy of the database using a backup and restore of the database whose log file has grown very large, then you can allow synchronization to proceed to drain the log send queue or if the size of the transaction log is much larger than the database itself, remove the database from the availability group, shrink the log file and add it back into the availability group, re-seeding your secondary replicas with the newly added database.

1. In SQL Server Management Studio (SSMS) remove the database from the availability group on the primary replica.
2. Put the database in simple recovery mode on the primary replica. This will allow you to shrink the log file.
3. Shrink the log file down to a reasonable size. To shrink the log file, in SSMS’ Object Explorer right-click the database, choose Tasks and then Shrink and then Files. For File type choose Log and click OK.
4. Once log file has ben shrunk, configure the database for full recovery mode and then right-click the availability group, choose Add Database and use the wizard to add the database back and re-initialize the secondary replica with it.

Scenario 2 The database is very large, making re-initialization prohibitive to the secondary replicas

If the data portion of the database is very large combined with the remoteness of one or more secondary's, it may not be reasonable to use Scenario 1. Another option is to apply the log backups onto the database at the secondary, to 'catch up' the database at the secondary. Following are the steps.

   1. Determine the transaction logs that must be applied. Query the last_hardened_lsn for the database on the secondary. Connect to the secondary and run the following supplying the database name as the only predicate:

select distinct dcs.database_name, ds.last_hardened_lsn from 

sys.dm_hadr_database_replica_states ds join sys.dm_hadr_database_replica_cluster_states dcs

on ds.group_database_id=dcs.group_database_id

where dcs.database_name='AutoHa-sample'

You will get results like this.

   2. Query the log backups in MSDB to find what log backup the secondary LSN falls in the range of.

select name, backup_set_id, backup_start_date, backup_finish_date, first_lsn, last_lsn from msdb..backupset

where first_lsn<'93000012832800001' and last_lsn>'93000012832800001'

We see one row returned, we need to apply all logs starting with this log on the secondary replica:

Alternative to find the first transaction log backup Query sys.dm_hadr_database_replica_states.last_hardened_time for the database (same query as above). Use File Explorer to view the transaction log backup files and order by modified date. Compare the last_hardened_time with the backup modified dates to determine what transaction log backup you should start with, selecting the transaction log backup whose modified date is just beyond the last_hardened_time. Query that transaction log to verify the last_hardened_time falls in that transaction log backup:

restore headeronly from disk= N'f:\backups\AutoHa-sample_backup_2017_01_20_194401_3869314.bak’

   3. Take database out of the availability group on the secondary. Connect to the secondary replica and execute the following to take the database out of the secondary replica.

alter database [AutoHa-sample] set hadr off

On the secondary the database will be in Restoring and ready to apply logs to.

   4. Apply transaction logs to database on the secondary replica Apply the transaction log backups. Here is a view of the log backup files.  We identified the one called 'AutoHa-sample_backup_2017_01_20_194401_3869314' as containing our secondary database last_hardened_lsn.

Begin restoring with that log being sure to restore with 'no recovery' all the transaction log backups.

restore log [AutoHa-sample] from disk='\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_194401_3869314.trn' with norecovery
go
restore log [AutoHa-sample] from disk='\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_194500_8646900.trn' with norecovery
go
restore log [AutoHa-sample] from disk='\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_194600_9809513.trn' with norecovery
go
restore log [AutoHa-sample] from disk='\\sqlserver-0\Backups\AutoHa-sample_backup_2017_01_20_200000_6780254.trn' with norecovery
go

   5. Add the database back into the availability group on the secondary and resume synchronization. Connect to the secondary replica and execute the following to add the database back into the availability group and resume sync.

alter database [AutoHa-sample] set hadr availability group = [contoso-ag]
go
alter database [AutoHa-sample] set hadr resume
go

You can view the availability group database on the secondary to confirm it is added back and synchronized.

If you have a Service Broker (SSB) application connecting to SQL Server using an AlwaysOn availability group listener, in the event of an unexpected failover, some messages may be lost or stuck in the transmission queue on the old primary (new secondary) after the failover. This could be an automatic failover or manual failover when the SSB application is not stopped.

Incoming (SSB) connections are severed during a availability group failover and those connections use the listener IP address which directs connections to the new primary replica. New incoming SSB connections, established through the listener, will go to the new replica. This is the correct and intended behavior.

The availability group fails over and the new inbound SSB connections reset to the new primary replica:

However, outgoing SSB connections from the old primary replica are left open after a failover, still in session with the initiator, and remain so because the IP addresses are still valid. New outgoing SSB connections are established from the new replica to the initiator, but because the connections from the old primary are still open, incoming SSB messages are still sent to the old primary.

Since Service Broker components are not contained within the availability group, special handling is required to handle the Service Broker connections when the availability group moves to a different replica instance. If you feel the post failover handling for the Service Broker connection is not feasible to be built into your application, please submit a Connect item (https://connect.microsoft.com/sql) so that we are able to understand the community need for this functionality. As an alternative to modifying the code for your application, you could choose to use a SQL Server Failover Cluster Instance which will prevent the issue described above, due to the IP Address remaining the same on failover.

UPDATE: Enhanced Compatibility When Hosting Service Broker in Availability Groups

This behavior has been improved in the following versions of SQL Server:

SQL 2014 SP1/CU12 & SP2/CU5
SQL 2016 RTM/CU6 & SP1/CU3
SQL 2012 SP4

For more information on these improvements, see the following KnowledgeBase article:

FIX: Service Broker endpoint connections aren't closed after an availability group failover in SQL Server

Troubleshooting

If, after the failover, you run the following query on the old primary SSB TARGET and see the value for is_accepted = 0 then you know you have encountered the issue. You will also likely have two or more connections returned here. 

--look for is_accept = 0

select is_accept, * from sys.dm_broker_connections with(nolock)

Another side effect are messages stuck in the sys.transmission_queue on the old primary SSB TARGET.

If Profiler trace is run on the old primary SSB TARGET, you would see the following error:

The message has been dropped because the service broker in the target database is unavailable: ‘The database is in read-only mode.’

SQL Server XEvent ‘broker_message_undeliverable’ can also be collected using SQL Server extended events and will also report that messages cannot be delivered. For more information on using SQL Server extended events, see:

Create an Extended Events Session Using the Wizard (Object Explorer)

Workaround

Host the primary replica on a SQL Failover Cluster Instance (SQL FCI). In the event of a failover, SQL Server restarts and the initiator will re-establish connectivity with the availability group listener and the TARGET will re-establish connections with the initiator. There will be no residual broker connections or in-flight transactions that cannot be resolved.

In this scenario, you still have a secondary replica hosted on a standalone instance of SQL Server which provides another means of high availability.

Maintenance

When failing manually over between availability group replicas for rolling upgrade purposes, stop the Service Broker Application before doing the manual availability group failover to prevent any in-flight transactions from getting stuck.]]> https://docs.microsoft.com/archive/blogs/alwaysonpro/alwayson-readable-secondaries-can-display-misleading-data-log-file-paths Wed, 29 Jun 2016 21:07:05 GMT https://blogs.msdn.microsoft.com/alwaysonpro/?p=3025 Written By: Grant Carter, Senior Premier Field Engineer Reviewed By: Mark Weber – Principal Premier Field Engineer Norm Eberly – Senior Premier Field Engineer Charles Allard – Senior Premier Field Engineer Nick Schueler - Senior Premier Field Engineer Curt Matthews – Senior Escalation Engineer   Problem You may discover on a readable secondary database that is part of an AlwaysOn Availability Group the data and transaction log file locations in sp_helpdb and sys.databases_files may display inaccurate results.   Scenario Results from these DMVs and stored procedures may be unexpected when executing them against the readable secondary if the readable secondary was restored with the move option and placed on a different file path as compared to the file path of the primary replica.   For example, the primary replica may have its files on the E: drive and the secondary replica may have its files on the L: drive. In the example below, you will see that the database is on the E: drive, but it was manually restored using the move command to the L: drive. On the primary replica:

create database [ag1db1]
on
(
       name = 'ag1db1_data'
       , filename = 'E:\SQL\ag1db1_data.mdf'
       , size = 20MB
       , maxsize = 20MB
       , filegrowth = 0
)
log on
(
       name = 'ag1db1_log'
       , filename = 'E:\SQL\ag1db1_log.ldf'
       , size = 20MB
       , maxsize = 20MB
       , filegrowth = 0
)
go

restore database [ag1db1] from disk = '\\sql2014n1\Backup\ag1db1.bak'
with
       move 'ag1db1_data' to 'L:\SQL\ag1db1_data.mdf'
       , move 'ag1db1_log' to 'L:\SQL\ag1db1_log.ldf'
       , norecovery
       , stats=10
go

sp_helpdb ag1db1
go

use ag1db1
go
select name, physical_name
from sys.database_files
go

use master
go
select name, physical_name
from sys.master_files where database_id = db_id('ag1db1')
go

How to enable TDE Encryption on a database in an Availability Group

Always On for SSIS Catalog (SSISDB)

New Availability Group Wizard Adds Support for Encrypted Database

Added Support for Accessing Encrypted Availability Group Database After Failover

EXEC sp_control_dbmasterkey_password @db_name = @dbName, @password = @pwd, @action = N''add''',N'@dbName nvarchar(9),@password nvarchar(20)',@dbName=N'encryptDB',@password=N'1a2b3c4d5e6f7g8h9i0!'

More Information About the Relationship between the SQL Server service master key and the database master key

Database Master Key credentials

For more information see the following resources:

For more information, please see

SQL Server Encryption SQL Server and Database Encryption Keys (Database Engine) CREATE MASTER KEY (Transact-SQL) How to enable TDE Encryption on a database in an Availability Group sp_control_dbmasterkey_password (Transact-SQL)

Benefits

• This increases the chances that high availability can be maintained if one of the automatic failover partners is lost.
• Reduces the need to manually manage failovers or to reconfigure the availability group with another automatic failover partner in the event that one of the automatic failover partners is unavailable.

Configure Multiple Automatic Failover Targets

USE [master] GO ALTER AVAILABILITY GROUP [AG] MODIFY REPLICA ON N'SQLN1' WITH (FAILOVER_MODE = AUTOMATIC) GO ALTER AVAILABILITY GROUP [AG] MODIFY REPLICA ON N'SQLN2' WITH (FAILOVER_MODE = AUTOMATIC) GO ALTER AVAILABILITY GROUP [AG] MODIFY REPLICA ON N'SQLN3' WITH (FAILOVER_MODE = AUTOMATIC) GO

Troubleshooting Automatic Failover Problems

Troubleshooting automatic failover problems in SQL Server 2012 AlwaysOn environments

select replica_server_name, availability_mode_desc,failover_mode_desc, database_id, synchronization_health_desc,synchronization_state_desc,* from sys.availability_replicas as AR inner join sys.dm_hadr_database_replica_states as DRS on DRS.replica_id = AR.replica_id

FAQs

When a health event is detected in SQL16N1 (in this example, I shut down SQL16N1 through SSMS Object Explorer), Cluster checks the next node in the preferred owner list which is SQL16N2. Assuming it is healthy and SYNCHRONIZED, Cluster will successfully failover to SQL16N2.

Now, try adding the availability replicas in the reverse order to the availability group, like this. After adding the replicas, check the properties of the availability group role to confirm the order of the preferred owner list:

When a health issue is detected on SQL16N1 (again shut down SQL Server on SQL16N1) we can see the preferred owner list priority is honored and Cluster attempts to failover to SQL16N3 first and successfully.

For more information see the following link on automatic failover:

Failover and Failover Modes (Always On Availability Groups)