Yet another Kerberos authentication failure troubleshooting scenario. This scenario never stops paying me a visit every now and then. Same type of symptoms.

• SQL Server named instance configured to listen on a static port.
• Logged on user (in a client machine) and the SQL server machine are on two different domains.
• A domain controller has been upgraded to 2012.
• Domain trust has been the same as it was before
• SPNs exists as before, in the working scenario.

Yet, when the user tries to connect to SQL Server instance from SQL Server Management Studio, they get the familiar Kerberos authentication failure error message:

"The target principal name is incorrect.  Cannot generate SSPI context"  

Environment Details

User: DomainA\userA

Server: xxxxx.DomainB.com (this is a standalone SQL Server)

SQL Server Instance: xxxxx\SQLx (version 2014)

SQL Server Domain: DomainB

Current Windows Server for Domain Controllers (Windows Server 2012). This was an upgrade. This is the only change.

Let us collect some data to see what went wrong

As I typically do, after verifying the error message from SSMS, I first wanted to make sure, we have the correct SPNs. So I ran the command (in SQL Server machine, we have the tool so it was ok to run here):

SETSPN -L <SQL Server Instance Service Account>

This shows that we have the required SPN. Since we have the SPNs, I did not go after using Kerberos Configuration Manager, which is another way to verify SPNs and add them if we do not have one. This sure saves a lot of time.

I also use the LDIFDE to make sure we do not have duplicates. So ran, the following command in both user DC and Server DC:

LDIFDE -f check_SPN.txt -t 3268 -d "" -l servicePrincipalName -r "(servicePrincipalName= MSSQLSvc/XXXXXXXX*)" -p subtree

Where XXXXXXXX is our SQL Server Machine when instance SQLx is running.

When we ran it against the SQL Server Machine domain (DomainB), we see that the same SPNs (as from SEPSPN -L output) as we would expect, however, when we ran it against user domain (DomainA), there is no output.

Another most common tool I use is SSPIClient and I leveraged this in this scenario in an effort to get more information. It gave me some additional information, when I ran the tool from the client machine, trying a connection test against the SQL instance xxxxx\SQLx. Two messages are of interest:

SubStatus=0xc000005e -> There are currently no logon servers available to service the logon request

SPN MSSQLSvc/XXXXXXXX .DomainB.com:yyyy not found anywhere in Active Directory

yyyy is the port number where SQL Server is currently listening on, configured as a static port for the instance SQLx

The client was looking for the SPN @DomainA.

Lastly I leveraged the netmon tool as well to double check. It shows also that we are looking for the SPN "MSSQLSvc/XXXXXXXX .DomainB.com:yyyy", @DomainA , which is should not, typically speaking.

I also verified that netlogon server for the machine where the user is logged on is different from the netlogon server for the SQL Server. This was important because  sometimes because they are talking to different netlogon server, which may not be in sync with AD objects.

Initial Assessment

Based on the above findings it made sense to me that we are likely having a domain trust issue, as I thought for a successful Kerberos flow, we do need a two way trust. I also had "two different logon server" clue in mind, which could be an issue. Since I am not expert in Active Directory matters, I sought some help to get to the bottom of this behavior.

First attempt resolving the issue

With the help from our Active Directory expert, we first thought it was a permission issue. The following KB helped in first attempt resolving:

https://support.microsoft.com/en-us/kb/811889/

We provided  "Read all properties" and "Write all properties" to SELF on the SQL Server service account. This was done on the SQL Server machine domain controller (DomainB)

This seemed helped. We are now able to connect to SQL Server from the test client machine with one user. 

Note: I found that with only outbound trust, the authentication flows ok, just need to have proper permissions. May be I need to dig a little more on this part.

The issue came back though

Unfortunately the stubborn Kerberos error continues with other users or it came back after some time.

Renewed effort to finding the root cause and resolve it for all

We continued our digging to find the root cause of the issue.

We first ran the command nltest /verify:xxxxx.DomainB.com to verify trust on the client.

This gave us:

...no such domain exits/no logon servers were available with status code 1355.

This is the similar message as we have seen for our SSPIClient trace test (SubStatus=0xc000005e -> There are currently no logon servers available to service the logon request)

We looked at the local cached Kerberos ticket (ran "klist" from command line in the client), it does not show the Kerberos ticket for our SQL Server instance.

We then ran the following commands from command line and took a netmon trace in the client machine filtered for DNS (to formulate an SPN we do need forward and reverse lookup of the host, so DNS queries need to be tracked), when testing connection to SQL Server from the client machine.

• ipconfig /flushdns
• klist purge

The trace shows DNS query for SQL Server has been triggered, it reached to DC, however it could not resolve the name.

Checking the msdcs for the SQL Server, we found a wrong entry being populated for the SQL Server machine. So we corrected the entry in the logon server DC (for the client) and then replicated the change to all involved DCs. We first had problem in getting the zone information so we re-created the zone and received the updates of the zone. This finally did the trick for us, the issue now got resolved at this point, to the completion. All users are now able to connect over Kerberos.

Recently I had to investigate the following scenario. 

• SQL Server named instance is configured to listen on a dynamic port.
• We change the port to static one, other than the one SQL Server is currently listening on, in SQL Server Configuration Manager.
• After the change, we do not restart the SQL Server Service.
• We expected the existing connection will not fail. In other words we will still be able to continue connecting to the SQL Server the same way as before, until the SQL Server service is restarted.

We find that, the application starts failing to connect. Both .NET application and SQL Server Management Studio, shows the same symptom. 

You may get the following error message:

A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) (Microsoft SQL Server, Error: 10060)

Environment Details

Client: client.fqdn.com

Server: xxxxx.fqdn.com

SQL Server Instance: xxxxx\SQL2012

Listening port before change 61111 (dynamic).

Changed to static port: 51000

It sure prompted when I clicked on apply, that the changes will not apply until we restart the SQL Server Service.

Test Scenario 1: Change the port to static port, from SQL Server Configuration Manager, do not restart SQL Server Service, connect from C# application remotely.

Code snippet I used for testing:

string cnStr = @"server=xxxxx\sql2012;database=TestDB;trusted_connection=sspi;";         
for (; ; ) {

             using (SqlConnection cn = new SqlConnection(cnStr))
             using (SqlCommand cmd = new SqlCommand("SELECT @@SPID", cn))

             {                

 cn.Open();

                   System.Diagnostics.Debug.WriteLine("Connected: SPID: {0}", cmd.ExecuteScalar());

               }

      System.Threading.Thread.Sleep(10000);

}

Error message we get from application:

 {"A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)"}

Took a netmon trace on the server.

 Trace shows the activity on the browser port 1434

udp.Port==1434

 1906        2:12:26 PM
12/16/2014        17.0452530                client.fqdn.com        xx.xx.xx.xx        SSRP        SSRP:Windows stub parser: Requires full Common parsers. See the "How Do I Change Parser Set Options(Version 3.3 or before) or Configure Parser Profile (Version 3.4)" help topic for tips on loading these parser sets. {UDP:624, ESP:308,IPv4:296}

1907        2:12:26 PM
12/16/2014        17.0455763                client.fqdn.com        xx.xx.xx.xx        SSRP        SSRP:Windows stub parser: Requires full Common parsers. See the "How Do I Change Parser Set Options(Version 3.3 or before) or Configure Parser Profile (Version 3.4)" help topic for tips on loading these parser sets. {UDP:625, IPv4:296}

1909        2:12:26 PM
12/16/2014        17.0460581                xx.xx.xx.xx        client.fqdn.com        SSRP        SSRP:Windows stub parser: Requires full Common parsers. See the "How Do I Change Parser Set Options(Version 3.3 or before) or Configure Parser Profile (Version 3.4)" help topic for tips on loading these parser sets.        {UDP:625, IPv4:296}

 So applied filter tcp.Port==51000 and tcp.Flags.Syn==1

1914        2:12:26 PM
12/16/2014        17.1025252                client.fqdn.com        xx.xx.xx.xx        TCP        TCP:Flags=......S., SrcPort=65113, DstPort=51000, PayloadLen=0, Seq=3642803191, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:626, ESP:308, IPv4:296}

1915        2:12:26 PM
12/16/2014        17.1025252                client.fqdn.com        xx.xx.xx.xx        TCP        TCP:Flags=......S., SrcPort=65113, DstPort=51000, PayloadLen=0, Seq=3642803191, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:627, IPv4:296}

1980        2:12:26 PM
12/16/2014        17.6058845                client.fqdn.com        xx.xx.xx.xx        TCP        TCP:Flags=......S., SrcPort=65113, DstPort=51000, PayloadLen=0, Seq=3642803191, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:635, IPv4:296}

2742        2:12:32 PM
12/16/2014        23.6110969                client.fqdn.com        xx.xx.xx.xx        TCP        TCP:[SynReTransmit #1980]Flags=......S., SrcPort=65113, DstPort=51000, PayloadLen=0, Seq=3642803191, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:635, IPv4:296}

 Frame Details for connection failure:

You basically get a SYN re-transmit, after the last connection attempt.

Conclusion

Seems that when the port is changed from dynamic to static, the browser service was aware of the issue, so when client first connected to the browser service, it just handed over the new port information as the browser service connection shows.

Test Scenario 2: Change the port to static port, from SQL Server Configuration Manager, do not restart SQL Server Service, connect from SSMS, running the client machine.

 Observation

1. Connection is successful
2. There is no UDP connection over port 1434
3. SSMS tries a direct connection to old port

Netmon Trace shows Successful handshake

2256        2:09:53 PM 12/16/2014        16.4745724       

sqlservr.exe        xx.xx.xx.xx        xx.xx.xx.xx        TCP        TCP:Flags=......S., SrcPort=64874, DstPort=61111, PayloadLen=0, Seq=2379066210, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:893, IPv4:635}2257        2:09:53 PM
12/16/2014        16.4754496        

sqlservr.exe        xx.xx.xx.xx        xx.xx.xx.xx        TCP        TCP:Flags=...A..S., SrcPort=61111, DstPort=64874, PayloadLen=0, Seq=666085223, Ack=2379066211, Win=8192 ( Negotiated scale factor 0x8 ) = 2097152        {TCP:893, IPv4:635}2258        2:09:53 PM
12/16/2014        16.4760709        

sqlservr.exe        xx.xx.xx.xx        xx.xx.xx.xx        TCP        TCP:Flags=...A...., SrcPort=64874, DstPort=61111, PayloadLen=0, Seq=2379066211, Ack=666085224, Win=517 (scale factor 0x8) = 132352        {TCP:893, IPv4:635}

Frame details for the successful connection.

 And complete conversion is here.

4. Interesting to see how we bypassed the browser service connection, I wonder if the connection information was being cached.

5. So tested restarting SSMS. Now, if I try a connection, it seems I get the following error:

TITLE: Connect to Server

------------------------------

Cannot connect to xxxxxxx\sql2012.

------------------------------

ADDITIONAL INFORMATION:

A network-related or instance-specific error occurred while
establishing a connection to SQL Server. The server was not found or was not
accessible. Verify that the instance name is correct and that SQL Server is
configured to allow remote connections. (provider: TCP Provider, error: 0 - A
connection attempt failed because the connected party did not properly respond
after a period of time, or established connection failed because connected host
has failed to respond.) (Microsoft SQL Server, Error: 10060)

6.  The client side netmon trace for UDP communication shows:

27.2043295        74        3860        8:07:07 AM 12/17/2014        38.3465472                xx.xx.xx.xx        server.fqdn.com        SSRP        SSRP:Windows stub parser. Select the "NetworkMonitor\Windows" profile from the parser profiles drop-down  menu.        {UDP:1392, ESP:1391,IPv4:320}

0.0000570        51        3861        8:07:07 AM 12/17/2014        38.3466042                xx.xx.xx.xx        server.fqdn.com        SSRP        SSRP:Windows stub parser. Select the "NetworkMonitor\Windows" profile from the parser profiles drop-down menu.        {UDP:1393, IPv4:320}

0.0012585        133        3862        8:07:07 AM 12/17/2014        38.3478627                server.fqdn.com        xx.xx.xx.xx        SSRP        SSRP:Windows stub parser. Select the "NetworkMonitor\Windows" profile from the parser profiles drop-down menu.        {UDP:1393, IPv4:320}

So, browser service already know about the change and give the new port information to the client. The information is the same in the server side trace but

 7. Now client tries to connect to port 51000 and gets a reset from the client side, fails subsequently

0.0000000        90        3875        8:07:07 AM 12/17/2014        38.4063314        Ssms.exe        xx.xx.xx.xx        server.fqdn.com        TCP        TCP:Flags=......S., SrcPort=54176, DstPort=51000, PayloadLen=0, Seq=591041439, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:1399, ESP:1391, IPv4:320}

0.0000217        66        3876        8:07:07 AM 12/17/2014        38.4063531        Ssms.exe        xx.xx.xx.xx        server.fqdn.com        TCP        TCP:Flags=......S., SrcPort=54176, DstPort=51000, PayloadLen=0, Seq=591041439, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:1400, IPv4:320}

0.0005405        60        3877        8:07:07 AM 12/17/2014        38.4068936        Ssms.exe        server.fqdn.com        xx.xx.xx.xx        TCP        TCP:Flags=...A.R.., SrcPort=51000, DstPort=54176, PayloadLen=0, Seq=0, Ack=591041440, Win=0        {TCP:1400, IPv4:320}

0.5031681        66        3912        8:07:07 AM 12/17/2014        38.9100617        Ssms.exe        xx.xx.xx.xx        server.fqdn.com        TCP        TCP:Flags=......S., SrcPort=54176, DstPort=51000, PayloadLen=0, Seq=591041439, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:1410, IPv4:320}

6.0006699        62        4686        8:07:13 AM 12/17/2014        44.9107316        Ssms.exe        xx.xx.xx.xx        server.fqdn.com        TCP        TCP:[SynReTransmit #3912]Flags=......S., SrcPort=54176, DstPort=51000, PayloadLen=0, Seq=591041439, Ack=0, Win=8192 ( Negotiating scale factor 0x8 ) = 8192        {TCP:1410, IPv4:320}

8. So behavior is the same in both test scenarios. 

Explanation

Here is the details of what happens behind the scene:

• Once we change the port from dynamic to static, in SQL Server Configuration Manager, the related registry get updated.
• The related registry key is, as an example, for the instance we tested against: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.SQL2012\MSSQLServer\SuperSocketNetLib\Tcp\IPAll
• Browser Service, being a separate service, is notified (becomes aware) about the change.
• When an application or SSMS (essentially client provider/driver SNI layer), tries to connect the next time, it contacts the browser service, which hands over the new port information for the named instance.
• Client (driver/provider SNI layer) for the application, tries to connect to the new port of the SQL Server. However, SQL Server does not listen to the port (there is no end point with the new port number), hence the connection fails.

The behavior is by design, in the SQL Server Releases. SQL Server is not aware of the change, until the Service is restarted, so the warning is appropriate. The browser service, being a separate service is not directly related to SQL Server, however it looks into the registry changes that happen through SQL Server Configuration Manager. 

Comments

To hide the machine information, I either erased or renamed the client/server name or IP addresses involved in my repro scenarios, I reproduced the issue with two of my machines.

My most recent encounter with SSAS DB corruption surfaced slightly differently than normal. Thought I would share to benefit others who may have experienced or is experiencing the issue as I write.

We have a situation, where:

• SSAS Service, hosting numerous multidimensional databases, goes into an unresponsive state
• When we restart SSAS service and try to look into the properties of the instances, connecting to SSAS from SSMS, we get the following error:

Analysis Services issue; Getting an error “failed to retrieve data for this request. (Microsoft.SQLServer.Management.SDK.SFC) Additional information: errors in the metadata manager an error occurred when loading the admin database role from \\?\<data location>\XXXX.0.db\XXXX.0.role.xml (Microsoft.AnalysisServices.xmla)” when he logs in to the instance and trying to pull properties of this instance;

Troubleshooting

Based on the error message, I had the following theories:

1. We have a corruption issue in the DB file, showing up on the error message.
2. SSAS is in a state that I retained the information somewhere, and trying to read the information, upon start up.
3. Master.vmp file is corrupted  

Understanding Master.vmp file

Master.vmp file is located under the Data folder, we configure for our SSAS DB, typically located at C:\Program Files\Microsoft SQL Server\MSAS11.MSSQLSERVER\OLAP\Data (for SSAS 2012), unless you have a custom configuration. This file is the master version map that contains the GUIDS for all of the objects and the version of each object that currently exists on the server. 

When the server starts, it looks at the master.vmp file and attempts to find each of the objects referenced in the file. So we can have following possible corruption scenarios:

1. If there are objects in the data directory that don't have a corresponding GUID in master.vmp, those objects are deleted.
2. If the master.vmp doesn't exist when the server starts, the server creates a new master.vmp file and deletes all of the objects in the data directory.
3. If you have deleted your DB but the master.vmp file keeps a record of the GUID for the DB, this can also result in an inconsistency. 

Check your log files

 Keeping in line with my theory, I first checked the event log.

The event viewer shows this message:
Errors in the metadata manager. An error occurred when loading the Package Detail cube, from the file, '\\?\<data location>\xxxx.0.db\xxxx.cub.xml'.

This message also pointed me to fact that we are likely having a corruption issue, either on DB or on Master.vmp file. 

I haven't checked the msmdsrv.log file, however I am pretty sure I would see the same error. I admit, it's worth looking into this file for more information.

More clues

At this stage, we discussed out another clue.  Our clue to this scenario was the key towards our troubleshooting path: the DB, in the error message, was deleted and is no longer needed.

Connecting the dots

At this stage, I just started hunting for the missing DB.

1. On the Data folder, I find the folder for the DB, which I know is not supposed to be there, because we have deleted it from the SSMS.

2. I also see the <dbname>.x.db.xml file, which is the metadata file for the missing SSAS DB

3. I then opened the <dbname>.x.db.xml file, in notepad, located the object GUID for the DB object, an entry from <dbname>.x.db.xml file. [i.e. Adventure Works SSAS DB file name is Adventure Works DW.1.db.xml]

It would look like this: <ObjectID>D9B263AD-2E34-4C90-BC95-384CD15A48BC</ObjectID> for Adventure Works SSAS DB, in my machine.

4. Then searched master.vmp file for the same GUID

5. Stopped SSAS, removed the entry for this GUID, from master.vmp since we know that the DB is deleted so this entry should not be there any more. [Please take a backup of this file to avoid un-intended circumstances]

The xml node for the entry (which we removed), would look like below:
<VersionMapEntry><ObjectId>D9B263AD-2E34-4C90-BC95-384CD15A48BC</ObjectId><Version>1</Version><ObjectLastUpdated>130509560009251999</ObjectLastUpdated></VersionMapEntry>
<ObjectId>D9B263AD-2E34-4C90-BC95-384CD15A48BC</ObjectId> ---> This refers to the DB object for example

6. Started SSAS DB
Now we are able to see the DBs again, when connected to SSAS from SSMS

Conclusion

Since SSAS works with the file system, if the files under Data folder, are in a locked state, for some reason, when we try to write on those file, we may have inconsistent situation, which may lead to DB corruptions. If you suspect any such situation, please check the master.vmp file or the xxxx.db.xml file for the DB (or object) in question for clues. Application event log or msmdsrv.log file also may give you more information about the error condition. Understand what role master.vmp plays will help avoid or resolve some of the tricky DB corruption issues. 

References

Tampering master.vmp file may result in losing all Analysis Services Databases

How to restore SSAS Database, tabular mode or multidimensional mode.

The scenario is:

• We have SharePoint 2013 configured in one Server with SSRS 2012 integrated mode
• This is one node SharePoint Server, with Central Admin and Front End on the same machine, from the architecture standpoint.
• Excel Services pointing to the PowerPivot instance installed on a separate machine though
• We create BISM Data Source, pointing to a tabular mode SSAS Database deployed on the same machine, where PowerPivot instance is running, no error during the Data Source creation.

In other words:

Machine A: SharePoint Central Admin + WFE + SSRS 2012 Integrated Mode + Excel Services

Machine B: PowerPivot Instance for the Excel Services + Tabular Mode SSAS instance.

We have SP1 on the SQL Server 2012.

Interesting is, after some environmental change, when we try to create PowerView report, using the BISM Data Source, we get the following error message:

"An error occurred while loading the model for the item or data source 'EntityDataSource'. Verify that the connection information is correct and that you have permissions to access the data source."

This is the same error that is discussed on the connect bug, and gives some hints on how to get around the error:

https://connect.microsoft.com/SQLServer/feedback/details/715205/you-cannot-modify-the-data-source-of-a-power-view-report

From Verbose log, we verified the error message, found that C2WTS was not started. Once we started this service, we found the error message now changed to:

Unexpected       Throwing Microsoft.ReportingServices.Diagnostics.Utilities.CannotRetrieveModelException: , Microsoft.ReportingServices.Diagnostics.Utilities.CannotRetrieveModelException: An error occurred while loading the model for the item or data source 'http://<URLToGallary>/bismfilename.bism'. Verify that the connection information is correct and that you have permissions to access the data source. ---> Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'TemporaryDataSource'. ---> Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: A connection cannot be made to redirector. Ensure that 'SQL Browser' service is running. ---> System.Net.Sockets.SocketException: No such host is known     at System.Net.Sockets....

Subsequently, we took the following steps

• Made sure the browser service is running, ports (2382 and others) are opened on the firewall for the SharePoint and the SSAS Tabular mode servers.
• Netmon shows we did not try to connect to browser service or tabular instance port, but I do see the PowerPivot activity on PowerPivot instance port. Also, there was no attempt to connect to the tabular mode instance.
• Profiler trace does not show activity obviously to tabular mode instance.
• IP:port or SSASServer:port combination also does not show connection working
• However, the UDL connection (or VS tabular mode project connection) from SharePoint to the SSAS Server were both successful

All indications were pointing to the fact that something happened when the file was loaded, however, I could not make sense where the error was coming from.

We then found a temporary workaround by using an RSDS file to connect to the BISM file and run a PowerView report using this method. In the process we found that, the RSDS would only work, if we used stored credentials, which led to further investigation in the Kerberos troubleshooting path. In this scenario, the C2WTS service account changed at some point which perhaps broke the original C2WTS Kerberos settings that had been done, although I was informed that, prior to having the issue, we did not have to setup Kerberos. So we started troubleshooting, in the Kerberos path, thinking that constraint delegation is failing for the C2WTS service account.

In this direction, we took the following steps:

• Looked at the "Delegation" tab for C2WTS Service Account in Active Directory, it is currently set to "Do not trust this user for delegation"
• We changed the settings to "Trust this user for delegation to specified services only", "Use any authentication protocol", we made sure, the SPN for the Tabular mode SSAS instance is included in the list. This is constraint delegation to SSAS Tabular mode instance.
• We added the SPNs for the tabular instance, for the service account it is listening on, prior to setting the constraint delegation.

setspn -S MSOLAPSvc.3/<SSASServer:TabularInstanceName> <tabularmode service Account>
setspn -S MSOLAPSvc.3/<SSASServer.FQDN:TabularInstanceName> <tabularmode service Account>

After these changes, the broken PowerView report with BISM data source started working again.

I am puzzled as to how it took the troubleshooting path we had to take to resolve the issue, the error message was sure misleading to point to other potential issues.

Special thanks to my colleague, Josh Gerszewski, who helped me investigate this issue.

Happy PowerViewing..... 

SSAS instance (tabular or multidimensional) is not starting up or when we try to start this up, this goes to hang situation right away so we cannot connect to the instance.

For tabular mode SSAS instance, you may see the error:

 Error(s):  an error occurred when loading the model 

Troubleshooting

One of the reasons it may have the issue starting up is because we may have a model database file corrupted some how. To encounter the situation, we typically try the following steps, knowing that the instance is not in a start mode.

1. Rename the existing Data folder to say Data_old (for a typical installation, the path to this folder is: C:\Program Files\Microsoft SQL Server\<MSAS11.Instance>\OLAP\Data)
2. Create new "Data" folder on the same location, under C:\Program Files\Microsoft SQL Server\<MSAS11.Instance>\OLAP (for example, your path may be different)
3. Restart the SSAS instance

At this point the instance typically will start up, if the issue is really related to DB file corruption.

So now we can do one of the following to get our databases back, if we have any user SSAS Databases:

1. Copy the DB folder from the old Data folder
2. Restore the SSAS DB, if we have the backup or
3. Redeploy SSAS DB Project, from the SQL Server Data Tools or BIDS

I recently used step 1 for few of my similar situations, thought I would share, if this helps, with one form of restoring your SSAS DB. This can be used for a non start-up scenario as well, may be when you simply want to take the SSAS DB from one instance to another instance, hence I am sharing.

Steps to copy the SSADB folder

We can use the following steps to get our SSAS database back from the old "Data" folder to new "Data" folder

1. Stop the tabular mode SSAS DB
2. Copy the DB folder, the corresponding .xml file and the master.vmp file from old Data folder to new Data folder

For example, for my SSAS instance, I have the following database, that I may want to restore/copy from old to new Data folder.
AdventureWorksDW2012Multidimensional

(the steps are same for tabular and multidimensional, so I am picking multidimensional DB that I have available)

Inside the installation location, you file fine the following items for this DB, under Data folder (for me the location is: C:\Program Files\Microsoft SQL Server\MSAS11.MSSQLSERVER\OLAP\Data):
a) Folder for the database: AdventureWorksDW2012Multidimensional.0.db
b) XML File defining the database: AdventureWorksDW2012Multidimensional.7.db.xml

The XML file basically has all the information the SSAS DB, so it is kind of like metadata for the DB

Another most import file for your DB is the master.vmp file, which keeps track of the GUID of the DBs that you created (and other objects), the same GUID should show up in both the .db.xml file and the master .vmp file
In .db.xml file for my DB is:
<ObjectID>0D5F1C9B-EE4E-4F00-B98E-D680E0AF75DA</ObjectID>
If you look for this object ID, in master.vmp file, it has to be found, we have to have the match and I found it.
<ObjectId>0D5F1C9B-EE4E-4F00-B98E-D680E0AF75DA</ObjectId>

The idea is that, if want to restore the DB, by just copying the Folder, then we have to take the .xml file and maste.vmp file as well.

Please note that once we start the SSAS instance, the master.vmp file will be updated with the database files you only copy, so in case your master.vmp fle has other DB information, this will be lost. This means that, if you have another database to copy, with the steps above, you will have to copy the master.vmp file again, from the old location, since this file will have the GUID for the other Database (s) you are copying.

This is one way of getting your DB back to your SSAS instance.

If you now start the SSAS instance, you should see the DB after refresh.

Scenario

We would like to parameterize SSIS 2012/2014 Package connection manager with the environment variables. We would like to make sure the connection parameter picks up the value from the Environment variables during runtime.

Step 1: Create an SSIS Package

We have a very simple SSIS package with on Data Flow Task with one OLEDB Source and one Flat File Destination connection manager.

  
Figure 1: Simple Data Flow Task with one OLEDB Source and one Flat File Destination

Step 2: Parameterize connection manager of interest.

Right click on the Connection
Manager, choose “Parameterize”

 Figure 2: "Parameterize" option on the connection manager.

Step 3: Choose ConnectionString from the Parameterize dialog box:

Figure 3: Pick up "ConnecString" from the drop down for Parameterize Property

Notice the default value that is picked is the same you created the connection string for your source. Click ok. The parameter will show up under the “Parameter” tab under SQL Server Data Tools for 2012

Figure 4: Confirm the connection information under "Parameters" tab in SQL Server Data Tools, for your package

Step 4: Execute the Data Flow Task or the package, in SSDT, make sure this
runs to the completion

Step 5: Deploy Project to SQL Server 2012, under SSISDB (at this point, you are under SSMS, connect to SQL Server, hosting the Integration Services Catalog)

Figure 5: Deploy your project to SSISDB

Step 6: Right click on Environments and choose “Create Environment….” (you are not under SSMS, connect to SQL Server, hosting the Integration Services Catalog)

Figure 6: Create Environment Variable for the parameter

Step 7: Right click on the Project (you are not under SSMS, connect to SQL Server, hosting the Integration Services Catalog)

Figure 7: Configure your SSIS Project/Package to associate parameter with the environment variable

Step 8: Configure Parameter to refer to the environment variable

Under “Parameters” tab, observe the parameter value:

Figure 8a: "Parameters" under "Configure..." dialog box

Observe the “Connection Managers” information

Figure 8b: "Connection Manager" under "Configure..." dialog box

Reference the Environment Variable under the “Configur….” Dialog

Figure 8c: Reference Environment variable for the Parameter (s)

Step 9: Set the value for the Prod variable, for your runtime value you want to point to

Figure 9: Set the value for the Environment Variable you created

Step 10: Change the parameter binding under package configuration (Right click on project or package level under SSIS DB, in SQL Server, choose "Configure")

Figure 10: Bind the parameter manually, with the environment variable.

Please note that this is the manual step that we need to perform, when we have different environment variables bound to the same parameter (i.e. one for dev, one for prod)

Step 11: Execute package by picking the Environment variable bound to the parameter

Figure 11: Execute package, dialog box 1, value is not set automatically.

Please note the red mark, this goes away, once you select the environment variable (please note that this binding has to be performed manually, as in step 10, with configuration)

Figure 12: Select "checkbox" for the environment variable to associate the parameter with the environment variable bound in step 10.

Troubleshooting:

a)      Note that the file will be created upon successful completion of the project

b)      You can use the profiler trace against the source SQL Server to make sure it is picking up the environment variable value

c)       You can look at the package execution report in 2012 to make sure it picked up the right value.

d)      For safety, I erased the security related portion but here is the report looks like:

Figure 13: SSIS Runtime report for confirmation that it picked up new value and ran to success

Please note, run time value is different than the design time value for the Connection Manager

Reference:

http://msdn.microsoft.com/en-us/library/hh213230.aspx

http://blogs.msdn.com/b/mattm/archive/2011/07/22/parameterized-connection-managers.aspx

http://blogs.msdn.com/b/mattm/archive/2011/07/25/all-about-server-environments.aspx

• You try to open the properties dialog box from SSMS, for the connected SSAS instance (right click on the SSAS instance, choose properties)
• You get the following error message:

File system error: The following error occurred while opening the

file \\?\<mount point on a network share>\MDTempStore_***_***_***.tmp: Access is denied,.

Error in the metadata manager. An error occurred when loading the Tem System cube, from the

file  \"\?\<mount point on a network share>\Tfs_Analysis.0.db\Team System.*.cub.xml",

(Microsoft.AnalysisServices)

To resolve the issue, we took the following steps:

• First we created the data folder structure (along with Temp and Log folder structure) on the shared drive (on the mount point drive) i.e. T:\OLAP\data, T:\OLAP\Log, T;|OLAP\Temp etc. We then updated the msmdsrv.ini file with the Data, Log, Temp folder information
• We recreated the Tfs_Analysis SSAS database, issuing the creation step from TFS Manager. This is to clean up.
• We then added the user (SSAS Service Account) to the SSAS instance level security, as an Admin to SSAS. This actually made the difference, along with the clean data folder structure
• Now, when we issue the processing command, we do not get "Access Denied" error message any more

SSAS does need sufficient read and write access to any location, it writes/reads the data along with the temporary files it needs to create, during DB creation, processing etc. We can likely take process monitor log to look at the path it tries to go to and also the file it tries to create. SSAS Service account is the one that tries to get access to on the Data/Log/Temp locations, as part of its activities. If it does not have proper permission, we may see problem.

Happy troubleshooting....

• You connect to SSAS from SSMS, as a local admin (administrator account)
• The service is running under "Network Service" Account.
• You try to add an user, under "security" for the instance (Right click on the instance->properties)
• When you click ok, you get the following error message:

 TITLE: Microsoft SQL Server Management Studio

------------------------------ 

File system error: The following error occurred while opening the file '~OLAP\Config\msmdsrv.tmp': Access is denied. .

.......

.......

This happened because the service account does not have enough privilege, to create the file under the location. The solution would be to change the service account to "Local System" or higher privilege domain account

Tips:

If you try to look under the folder, ~OLAP\Config, you won't see the file has been created or exists. SSAS tries to create this temporary file for its own purposes but since the service account does not have right, it cannot the necessary file (s), thus gives the error message. You can try taking a process monitor to see it gets "access denied" when trying to access the file, however, the file actually is not there.

1. .NET application uses Microsoft.AnalysisServices.Server component to create the Server object to connect SQL Server Analysis Services.

2. The code snippet used, to load Microsoft.AnalysisServices.Server component into the application domain, is

objAS = CreateObject("Microsoft.AnalysisServices.Server")

The code above works fine in x86 environment, however the code fails to execute, giving the following error message, under 64 bit environment:

can't load Microsoft.AnalysisServer.Server in 64bit environemnt - "Cannot create Activex Component"

After investigation, I found that, the issue can happen under the following when, for Analysis Management Object (AMO) component for SSAS 2008, we do not install the components properly so that the 32 bit version exe can load 32 bit version of AMO and 64 bit version exes can load 64 bit AMO components

There are two possible we can get around to loading issue of the assembly. We can either load the corresponding assembly as:

oAssembly = Assembly.Load("Microsoft.AnalysisServices, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91, processorArchitecture=MSIL")

or we can also load the corresponding assembly as:

oAssembly = Assembly.LoadFile("C:\Program Files\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.AnalysisServices.DLL")

Now if we create the Server object, as below, we should have a problem.

oAS = oAssembly.CreateInstance("Microsoft.AnalysisServices.Server")

If the above workarounds does not work for some scenarios, then you can use the following technique:

Step 1:

Download and Install AMO (Analysis Management Object) component (s) from SQL Server 2008 Feature packs:
http://www.microsoft.com/en-us/download/details.aspx?id=16978

Microsoft® SQL Server® 2008 R2 Analysis Management Objects  Analysis Management Objects (AMO) is a .NET Framework object model that enables software developers to create client-side applications to manage and administer Analysis Services objects.

Audience(s): Customer, Partner, Developer

X86 Package(SQLSERVER2008_ASAMO10.msi)
X64 Package (SQLSERVER2008_ASAMO10.msi)
IA64 Package(SQLSERVER2008_ASAMO10.msi)

for 32 bit x86 option and for 64 bit x64 bit option

Step 2:

Register the DLL

Register the .NET Assembly manually into the registry this way:
C:\Program Files\Microsoft SQL Server\100\SDK\Assemblies>C:\windows\microsoft.net\Framework64\v2.0.50727\regasm.exe Microsoft.AnalysisServices.DLL (for 64 bit)
C:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies>C:\windows\microsoft.net\Framework\v2.0.50727\regasm.exe Microsoft.AnalysisServices.DLL (for 32 bit)

Reference
http://www.advancedinstaller.com/forums/viewtopic.php?f=2&t=7837

For .NET Framework 2.0 the 32-bit version of "Regasm.exe" is installed in this folder:
Code: Select all C:\windows\microsoft.net\Framework\v2.0.50727\regasm.exe
and the 64-bit version in this folder:
Code: Select allC:\windows\microsoft.net\Framework64\v2.0.50727\regasm.exe
So you have to use the respective regasm exe and the respective DLL version with the appropriate path for 32 bit and 64 bit version respective

Step 3:

Verify the component in the registry for COM interoperability

After this, the component should show you under register as:
Computer\HKEY_CLASSES_ROOT\Microsoft.AnalysisServices.Server

Step 4:

Write code to create COM Component in .NET code.

Now that you have the DLL, you can use the code:
oAS = CreateObject("Microsoft.AnalysisServices.Server")

After this step, you should have see the error message any more.

Hope this helps in future AMO programming scenario, where you need to troubleshoot AMO Assembly ioading issues.

Tips:

Use Fusion logging technique, to understand, why might be having the assembly loading issues.

Happy Analysis Management Object programming....

For simplicity, in my observation, the total report processing time is divided into the following:

1. Step 1) The time between when you click on a report to render and when report with parameters page and "View Report" button shows up and
2. Step 2) The time between when you choose parameters and click on the "View Report" button and the report is rendered based on your selection. This is typically the report execution time, for which the ExeuctionLog2 view would show all the      information.

If you have the default values for all parameters, then your report will be directly rendered first time, so, the time for 1) and 2) will be combined to give you impression that report is running for a long time.

If you need to choose the parameter values, then, it is easy to see the 2 separate times slices for 1) and 2) above, you can see the Report Viewer Control first, to choose the parameter and click "View Report", this is strictly the time related to step 1. In some cases, the time slice in this step can be significant, which may not be well understood, if we do not pay attention, during your report design. This is what we have discovered in one case and related the same for other, as the behavior was the same.

In our scenario, we had report with multiple parameters and they were bound to separate datasets (embedded) which would populate drop down lists with multi-select options. There was a known issue prior to SSRS 2008 R2 SP2, related to number of items in a dropdown parameter list, which was addressed already so we know that was not the problem in our scenario.

Without giving the details about the datasets, I just want to point out the Technet article, Report Embedded Datasets and Shared Datasets (Report Builder and SSRS), which discusses the datasets and parameters in details, the same design (embedded datasets for parameters) was used in our scenario. The concentration of this blog is how I found out where the time is spent in scenario step 1 (time slice 1) above.

Basically, I used the following tools to track down where the time is spent:

Fiddler Trace

1. SSRS HTTP Trace
2. SSRS Verbose Logging.
3. ExecutionLog2 View
4. Profiler Trace against the data source, especially observe the time before the actual report execution takes place.

ExecutionLog2 View would only explain the time in step 2 above.

So I used the Fiddler Trace, which showed that, for scenario, which includes, time for 1) and 2) above (no default parameter values set), it was significant time on step 1) and time in step 2 would be expected. It would seem, from the Fildder trace, that the GET request is taking all the time.

With further research, I then had SSRS HTTP Tracing and SSRS Verbose logging turned on and finally I was able to track down the time.

HTTP Trace would nicely show the GET request for step 1, when the request started and when it finished, which would be equal to the GET request in the Fiddler trace.

Researching on the SSRS Verbose Log, I would see that we would have request for each parameter list population, as a form of establishing a connection and then populating the dataset associated the parameter. If you have multiple parameters, you would see one connection and dataset pair for each one, if the parameter is bound to the embedded dataset.

Profiler trace helped to understand the time for step 2 first (matches the report execution time, especially TimeDataRetrieval matching the time with ExecutionLog2 View) but then with the information available from SSRS Verbose Log, this time profiler would clearly show all the connections, for each parameter population.

Please note, again, this is the time that was spent, before you can click on "View Report" choosing the parameters. If you have default values set, for parameters, you would have to carefully look at each possible connections to the data source (s), to put these two (step 1 and 2 above) together.

It was easy to see then why it was taking time before I could see the report page with parameters and "View Reports" and select the parameter and click the button to view the report.

In the scenario, where the default parameter values are set, the report will render combining the time in step 1) and 2) above, so you would have to definitely look for the first step carefully and should not only rely on the ExecutionLog2 View.

I hope, if you follow these steps, for your next report performance issue, where, it matches the above scenario, you will find the missing time and take appropriate steps (i.e. query tuning, report redesign etc.) to address the performance issue. In this regard, the following blog post will be helpful further performance tuning:

SSRS and SSAS-Sourced Parameter Dataset Performance Issues

Happy SSRS Performance Troubleshooting…..