Cause: After investigation we can confirm that the distributor database is with sync with backup option. And that cause the issue

Resolution: At this end to solve the issue, I remove this option that we don't need. I also check this property to all our publised database to be sure on the impact. And get a back in recovery model simple

select @@SERVERNAME,DATABASEPROPERTYEX('DistributionDB','IsSyncWithBackup')

EXECUTE sp_replicationdboption@dbname=  'DistributionDB',@optname=  'sync with backup'  ,  @value=  'false'

select @@SERVERNAME,DATABASEPROPERTYEX('DistributionDB','IsSyncWithBackup')

 
But if you need it, you could also  plan a backup log every hour and every day for your distributation base in recovery model FULL.

Work done : here I'm sharing my troubleshooting step that I did to understand this issue.

Step 1: I run the dbcc loginfo, and I found out that the number of VLF is increasing to 8135. Yesterday it was around 5000. That we can see that the status is 2, so almost all VLF are used and we are not able to truncate them.

Step 2: I had a look to see what the log is waiting to be reuse, we can see that we are pending replication.

select name,log_reuse_wait_descfromsys.databaseswherename='myDBName'

Step 3: I had a look to the Log reader and it was running and replicating the transactions on the distributor database. We saw that inside the verbos log level 2 that we enable.

-Publisher [myPublisherServer] -PublisherDB [myPublisherDB] -Distributor [MyDistributorServer] -DistributorSecurityMode 1  -Continuous -OutputVerboseLevel 2 -Output C:\ReplLog\ReplOutput.txt

Step 4: I enabled the Tracer to see the latency, and we don’t have any latencies on this replication.

Step 5:  “Oldest non-distributed LSN” does not change since a couple of days. I found that after  runing several time a DBCC OPENTRAN :

                        Transaction information for database 'myDBName'.

                       Replicated Transaction Information:
                               Oldest distributed LSN     : (1218321:2940:24)
                               Oldest non-distributed LSN : (1218321:2959:1) --> this part didn't change during serverals days

Step 6: Now the step was to see if the option synchWithBackup is enable the Distributor and the Publisher

On the distributor:  select @@servername, DATABASEPROPERTYEX('DistributionDbName','IsSyncWithBackup')

 This option was at TRUE

On the publisher : select @@SERVERNAME, DATABASEPROPERTYEX('publisherDBName','IsSyncWithBackup')

 This option was at  FALSE

For information this option means : The database is either a published database or a distribution database, and can be restored without disrupting transactional replication.

Step 7: To confirm the issue is due to this option, I put in the same recovery model the distributor database that the use db. Then I run a backup full an log

BACKUP DATABASE[distributionDB]TO  DISK=N'C:\mydump.BAK';

BACKUP LOG[distributionDB]TO  DISK=N'C:\mydump.TRN';

After this backup log all VLF on the published database become to 0 and the transaction log can be truncate now.

Regards,

Michel Degremont | Xbox Live Music - Data Engineer - SQL Server & PDW |

Problem description: on ours SQL Server 2008, when we execute "EXECUTE msdb.[dbo].[sp_cycle_agent_errorlog] ", we got the error message below:

    Msg 22022, Level 16, State 1, Line 0
    SQLServerAgent Error: 32.

Work done:

Step1: We can see that the log is recycled. The file SQLAGENT.OUT contains for each execution the message below:

   [412] Errorlog has been reinitialized.  See previous log for older entries.

Step2: Inside the procmon, we can see a          SHARING VIOLATION

Step3: With process explorer, I didn’t see that another process owns the same file SQLAGENT.OUT

Cause: After investigation, we can see that SQLAgent.exe is locked by himself. If you run Handle you will see that 2 handle are opened. And it should be only one. It look like a handle leak.

Current workaround: Restart SQL Agent service on server.

 NB : currently I am investigating my colleague from Microsoft Support (3rd December 2013)

Regards,

Michel Degremont | Xbox Live Music - Data Engineer - SQL Server & PDW |

Error message:

         --> Inside SCOM Setup installation, you got the error message below

eThe specified SQL Server instance is not valid.  The possible causes of this problem include the following:
* SSRS has not been configured correctly
* Secure Socket Layer (SSL) is configured for the SQL Server Report Server.
* A proxy server is configured for the SQL Server Report Server.e

        --> And if you open your report server http://myserver/reportserver, you got the error message below

"A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)"

Troubleshooting step:

Step 1: Go on the SSRS log file to get more detail. By default is here: C:Program FilesMicrosoft SQL ServerMSRSXX.MSSQLSERVERReporting ServicesLogFiles

Step 2: Open the last ReportServerService...txt, and have a look to the last error at the end of the file.

In my case, I have got this error:

ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.ServerConfigurationErrorException: Unable to load assembly Microsoft.EnterpriseManagement.Reporting.Security, Microsoft.ReportingServices.Diagnostics.Utilities.ServerConfigurationErrorException: Le serveur de rapports a rencontré une erreur de configuration.  ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.EnterpriseManagement.Reporting.Security' or one of its dependencies. The system cannot find the file specified.

Cause: This issue is due to SCOM that it changes the configuration files of the SQL Reporting Service.

Resolution:

Choice 1: Uninstall and reinstall SSRS

Choice 2 : When SCOM 2012 change the SQL Reporting Service 2012 configuration files, it takes the backup of the Reporting Service’s configuration files as: RSReportServer.config.0 and RSReportServer.config.1.

          - So, go to C:Program FilesMicrosoft SQL ServerMSRSXX.MSSQLSERVERReporting ServicesReportServer.

         - Then remove the file RSReportServer.config. And rename RSReportServer.config.0 to  RSReportServer.config

More detail here: How to Install the Operations Manager Reporting Server
http://technet.microsoft.com/en-us/library/hh298611.aspx

Regards,

Michel Degremont | Xbox Live Music - DBA Team - SQL Server & PDW |

Problem description: I got a specific scenario where I have to failover two or more AlwaysOn Availability Groups in the same time.  To explain by another way, I have to create a dependency between 2 Availability Groups. If one of them failover, the second one has to move also on the same node.

In my case, I have to do that because, I’m not allow to put in the same group all databases. And a third application has to perform a query (write/read) between N databases from two different 2 Availability Groups.

Action plan: As it’s not a normal or expecting behavior, I will share with you, how I did to perform this task.

Step 0: Copy past the script PowerShell below on all nodes involved (SQL_Node_1 and SQL_Node_2 and SQL_Node_3) inside C:\HADRON\failoverAutomaticHadronGroup.ps1

import-modulefailoverclusters
$Cluster_Name =  "myWindowsClusterName";

$my_AG1_Name ="AG1";
$my_AG2_Name ="AG2";

# We are looking for the current node of my_AG1_Name
$AG_1  =get-clustergroup  -name$my_AG1_Name-Cluster$ClusterName|select-object  OwnerNode,State

# then We are looking for the current node of my_AG2_Name
$AG_2=get-clustergroup  -name"IngestPub"-Cluster$my_AG2_Name|select-object  OwnerNode,State

if($AG_1.OwnerNode.Name -ne$currentTriggerNode  -and$AG_2.OwnerNode.Name -eq$currentTriggerNode )
{   
          # so we move the group AG_1 to the same node of AG2
          Move-ClusterGroup"$my_AG1_Name"–Node$AG_2.OwnerNode.Name -Cluster$ClusterName ;
}
elseif ($AG_2.OwnerNode.Name -ne$currentTriggerNode  -and$AG_1.OwnerNode.Name -eq$currentTriggerNode )
{
          # so we move the group AG_2 to the same node of IngestPub
          Move-ClusterGroup"$my_AG2_Name"–Node$AG_1.OwnerNode.Name -Cluster$ClusterName
}

Step 1: Open eventvwr.msc

Step 2: Go on the chanel:  Application and Services Logs\Microsoft\Windows\FailoverClustering\Operationnal

Step 3: Select eventID 1201 (The Cluster service successfully brought the clustered service or application 'XXXXXX' online.)

Step 4: Right-click -> Attach Task to this Event

Step5: Check start a program

Step 6: Select the powershell script of the step 0

•  Program/script : powershell.exe
•  Add arguments : -Noninteractive –Noprofile –Command "&{C:\HADRON\failoverAutomaticHadronGroup.ps1}"
  

NB: To modify the task, open Task Scheduler, please go to the  Windows  Scheduled Task Microsoft-Windows-FailoverClustering_Operational_Microsoft-Windows-FailoverClustering_1201.

Regards,

Michel Degremont | Xbox Live Music - DBA Team - SQL Server & PDW |

Problem description: You have got odd number of nodes. To prevent possible ties in the quorum vote you added a witness disk to the cluster and adjusted the quorum mode in the cluster manager.

However, when you perform a failover from SSMS on our SQL Server 2012 Availability group, we got a validation WSFC quorum vote configuration warning with action required

Error message: The current WSFC cluster quorum vote configuration is not recommended for this availability group.

Cause: This issue occurs because of a known issue in Windows Server Failover Clustering.

Resolution: To resolve this issue, you have to apply the hotfix below  on all computers node of your cluster:

A hotfix is available to let you configure a cluster node that does not have quorum votes in Windows Server 2008 and in Windows Server 2008 R2
http://support.microsoft.com/kb/2494036

Current status: After applied the fix, you should see that the warning is vanished.

Regards,

Michel Degremont | Xbox Live Music - DBA Team - SQL Server & PDW |

Hi all,

I would like to share with you an issue that I got recently because  I didn't set  up correctly AlwaysOn to enable readable database on the secondary replica.

Problem description: When I am trying to connect to my application with .Net or PowerShell, I got the following issue only when I add ApplicationIntent=Readonly on my connection string:

Sqlcmd: Error: Microsoft SQL Server Native Client 11.0 : TCP Provider: No connection could be made because the target machine actively refused it..
Sqlcmd: Error: Microsoft SQL Server Native Client 11.0 : Login timeout expired.
Sqlcmd: Error: Microsoft SQL Server Native Client 11.0 : A network-related or instance-specific error has occurred while establishing a connection to SQL Server
. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see
SQL Server Books Online..

Reproduction: I am able to reproduce with SQL CMD: sqlcmd -S myListener -E  -K ReadOnly -Q "SELECT @@SERVERNAME" However, when I remove -K ReadOnly, it works fine.

Other interesting stuff, I can see a login/logout on my primary server inside the profiler trace, but nothing on the secondary

Solution: In my case, the issue was that the read_only_routing port mismatch with the port of my instance. I enclose below what you have to look.

How configure correctly AlwaysOn readable secondary replica:

1.      Add Application Intent in the connection string by adding: ;ApplicationIntent=ReadOnly

2.      ALLOW_CONNECTIONS property of the secondary replica is READ_ONLY.

3.      The READ_ONLY_ROUTING_URL for each replica is set. The TCP port is a port number that is used by the Database Engine of the SQL Server instance

alteravailabilitygroupmyGroupName
modifyreplicaon'myMachine1instanceName1'with
(secondary_role(read_only_routing_url='tcp://myMachine1:1433'))

GO

alteravailabilitygroupmyGroupName
modifyreplicaon'myMachine2instanceName2'with
(secondary_role(read_only_routing_url='TCP://myMachine2:1433'))

4.       The READ_ONLY_ROUTING_LIST option must be set for each replica in the CREATE AVAILABILITY GROUP or ALTER AVAILABILITY GROUP Transact-SQL command, as part of the PRIMARY_ROLE replica options. The READ_ONLY_ROUTING_LIST can contain one or more routing targets. Multiple routing targets can be configured. Routing occurs in the order that targets are specified in routing list.

 alteravailabilitygroupmyGroupName
modifyreplicaon'myMachine1instanceName1'with
(primary_role(read_only_routing_list=('myMachine2instanceName2')))

GO

alteravailabilitygroupmyGroupName
modifyreplicaon'myMachine2instanceName2'with
(primary_role(read_only_routing_list=('myMachine1instanceName1')))

GO

Select Replica_Server_name,Endpoint_url,Secondary_Role_allow_Connections_Desc,Backup_Priority,  Read_only_Routing_URLfrom Sys.AVailability_replicas

selectdns_name,port,is_conformant,ip_configuration_string_from_cluster FROM sys.availability_group_listeners

5.       Then I installed the both KB to fix this issue System.ArgumentException: Keyword not supported: 'applicationintent'.

6.       To enable Kerberos, I configured in Active Directory by a domain administrator for each availability group listener name

Configure SPN in AD for each AG Listener name - A Server Principal Name (SPN) must be configured in Active Directory by a domain administrator for each availability group listener name in order to enable Kerberos for the client connection to the availability group listener. When registering the SPN, you must use the service account of the server instance that hosts the availability replica . For the SPN to work across all replicas, the same service account must be used for all instances in the WSFC cluster that hosts the availability group. (http://msdn.microsoft.com/en-us/library/hh213417.aspx#SPNs)

             setspn -A MSSQLSvc/myListenerName.FQDN:1433  myDomainmyserviceAccount    

7. Configuring the Windows Firewall to Allow SQL Server Access

If you got any connectivity issues, I invite you to read: How to resolve SQL Server connectivity issues

Regards,

Michel Degremont | Xbox Live Music - DBA Team - SQL Server & PDW |

Problem description: When you try to connect to  your SQL Server by using powershell, .Net, sqlcmd... you got this error message:

System.ArgumentException: Keyword not supported: 'applicationintent'.
   at System.Data.SqlClient.SqlConnectionStringBuilder.set_Item(String keyword, Object value)
   at System.Data.Common.DbConnectionStringBuilder.set_ConnectionString(String value)

Cause: The message occurs when you include ApplicationIntent=ReadOnly (AlwaysOn Availability Groups properties) in your connection string. And  the server does not recognize the property.

KB Extract number 2544514: "When the ApplicationIntent property is set to ReadOnly, the client requests a read access operation when it connects to an AlwaysOn-enabled database. The server will enforce the read-only access operation both at connection time, and during a USE database statement, but only to an AlwaysOn-enabled database. When the ApplicationIntent property is set to ReadWrite, the client requests a regular (read-write) access operation when it connects to an AlwaysOn-enabled database. The default setting for the ApplicationIntent property is ReadWrite."

Resolution: You have to install the update that introduces support for the AlwaysOn features from SQL Server 2012

        -  Update 4.0.2 for Microsoft .NET Framework 4 – Runtime Update

       -   An update introduces support for the AlwaysOn features from SQL Server 2012 to the .NET Framework 3.5 SP1

Regards,

Michel Degremont | Xbox Live Music - DBA Team - SQL Server & PDW |

Problem description: We got some mini memory dump. In side our SQL error log, we can see the error message below

          Stack Signature for the dump is 0x00000000950A8E26
          External dump process return code 0x20000001.
          External dump process returned no errors.Error: 8624, Severity: 16, State: 116.
          Internal Query Processor Error: The query processor could not produce a query plan. For more information, contact Customer Support Services.
          ex_raise2: Exception raised, major=86, minor=24, state=116,

          severity=16, attempting to create symptom dump
          ex_raise2: Exception raised, major=86, minor=24, state=116, severity=16

Interesting link:  We found out a similar issue. However, I am in SQL Server 2005 SP4 (9.00.5000.00 (X64)), and the issue below was fixed in SQL 2005 SP2. E.G.    FIX: Error message when you run a query in Microsoft SQL Server 2005: "Msg 8624, Level 16, State 116 Internal Query Processor Error: The query processor could not produce a query plan"

Cause: We got the query from the dump, but you could also get the query with a profiler trace with the events TSQL SQL:BatchCompleted or Stored procedure RPC:Completed with Error and Waring Exception. We the query we are able to reproduce the issue each time. We saw that the dump was generated by statistics stream corrupted (spotted that in the generation script of the objects)

Resolution: We run an update statitics with fullscan to solve the issue on all objects called by the query.We run again the query and the issue didn't occur.

Michel Degremont | Premier Field Engineer - SQL Server Core Engineer |

Work done: To move you database to a new drive, you followed the step below.
  
   Step 1: With Microsoft SQL Server Management Studio, connect to Analysis Server, then right Click on the server and click on Properties.
   Step 2: Edit the path of you data directory. And replace the old path by the new one.
   Step 3: Stop the SQL Server Analysis Services Service.
   Step 4: Move the Data to the new directory location.
   Step 5: Start the SQL Server Analysis Services Service.
   Step 6: verify that you can browser all database.

Cause: Inside the SSAS log, you can see an  issue relating to something not allowing SSAS to update or access this file

     Message: Service started. (Source: ?D:Logmsmdsrv.log, Type: 1, Category: 289, Event ID: 0x41210000)
     Message: The file '\?E:OLAPDatamaster.vmp' could not be deleted.
     Please check the file for permissions. (Source: ?D:Logmsmdsrv.log, Type: 3, Category: 289, Event ID: 0xC121001F)

I run Process Monitor and after SSAS restart I saw an access denied on various file and also on master.vmp.

For insformation:  The master.vmp file is the master version map that contains the GUIDS for all of the objects and the version of each object that currently exists on the server. When the server starts, it looks at the master.vmp file and need to update it ( More information with my colleague Karan)

Resolution: You can solve this issue you can give the right "Modify" to SSAS service account to this new data folder.

Others points:

1. A database can be moved in many ways (SSMS Attach/Detach, AMO, XMLA). You could find some various way in the bol (Move an Analysis Services Database)
2. You can only move the file regarding Data, Backup, Log or Temp. However it's not supported to move manually the BIN and Config directory. You must reinstall your instance.

Michel Degremont | Premier Field Engineer - SQL Server Core Engineer |

Analysis Services is clustered as  a generic service resource on a Windows failover cluster. This applies to 2005, 2008, 2008R2, and 2012.

Since SSAS 2005 version, Analysis Services is fully integrated into clustering Setup (the install of SQL):

        - SSAS 2005 cluster setup is adding the service on all cluster nodes on the same setup
        - SSAS 2008, the setup have to be done node by node Analysis Services is presented to the cluster as generic resource with the type Application Service and doesn’t implement his own explicit IsAlive method. Unlike SQL Server, Analysis Services doesn't have a cluster resource .dll that knows how to do that work like connect to AS and run a simple query to check that it responds.

Possible cause of failure that is detected for an Application Service:

       - The service stopped(ie. crash, or was killed) then the cluster resource would notice
       - A dependency in the group where the generic service is configured failed (a disk fails or goes offline, an IP, virtual name, mount point, SQL server…other dependencies affecting the group)

Note : Application Service will NOT detect a hang situation in SSAS, where all connections are rejected.

Michel Degremont | Premier Field Engineer - SQL Server Core Engineer |