ADRMS and SafeNet Luna HSM - Error when Protecting Content
I recently had a customer run into an interesting error when integrating a SafeNet Luna SA5 Network-based HSM with their ADRMS infrastructure.
Problem
When the customer attempted to protect content, he received an error "A problem occurred when contacting the restricted permission service".
We verified that the customer could reach the Certification and Licensing sites without an issue.
To further isolate the problem, we followed the steps to set up Server and Client-side tracing. On the server, we saw the error below in DebugView when we attempted to rights-protect content.
[DirectResult] Error (Id=0) System.Runtime.InteropServices.COMException (0x800706D9): There are no more endpoints available from the endpoint mapper. (Exception from HRESULT: 0x800706D9)
At the same time, the IIS logs showed HTTP 500 errors when attempting to license content.
The Application event log on the ADRMS server had the following events:
*Log Name: Application
Source: Active Directory Rights Management Services
Date: 7/3/2012 10:51:10 AM
Event ID: 205
Task Category: Logging
Level: Warning
Keywords: Classic
User: N/A
Computer: server.domain.org
Description:
The Active Directory Rights Management Services (AD RMS) logging service could not authenticate the data integrity or sender identity of a message. This message will be added to the bad queue table in the AD RMS logging database.
Parameter Reference
Context: Logging.OnMessageQueueMessage
ErrorCode: MissingSignature
*
*Log Name: Application
Source: Active Directory Rights Management Services
Date: 7/3/2012 10:51:10 AM
Event ID: 72
Task Category: Logging
Level: Warning
Keywords: Classic
User: N/A
Computer: server.domain.org
Description:
The AD RMS message could not be digitally signed.
Parameter Reference
Context: ComponentBase.LogResults
RequestId: {3ebf05e8-0b1e-45e8-b590-6ddab54e1faa}.4:1
System.Security.Cryptography.CryptographicException
Message: Provider DLL failed to initialize correctly.
Context: ComponentBase.LogResults
*
To determine if this was specific to the HSM or some larger issue, we re-installed ADRMS with the exact same setup without using the HSM. The user was then able to protect and consume content without a problem. After re-installing with the HSM in place, the error returned. Also, when attempting to rights protect content with the HSM, the user was prompted to provide either Windows Live ID credentials or use domain credentials. When the user chose domain credentials it threw the error. Without the HSM in place, the user was not prompted for credentials.
Resolution
When working with the customer's on-site HSM technician, he noted that they were using strongprotect on the communication channel between the Luna HSM client on the ADRMS server and the Luna HSM. He noted that this had caused problems in the past with a Microsoft Certificate Authority. We recreated the channel by following the steps below:
Open an Administrative Command Prompt
cd "C:\Program Files\LunaSA\CSP"
Register /partition /highavail
IISReset
After performing these steps, the user was able to rights-protect and consume content from Microsoft Word. After we completed this, we ran Register /strongprotect and we were still able to protect content.
