Encryption in SMB 3.0: A protocol perspective

Encryption is one of the new SMB 3.0 security enhancements in Windows Server 2012 RTM. It can be enabled on a per-share basis, or enforced for all shares on the server. SMB 3.0 uses AES-CCM [RFC5084] as encryption algorithm, and this also provides data integrity (signing).

This blog takes a protocol walk on the topic through [MS-SMB2], and provides a sample test vector. [MS-SMB2] describes SMB 2.x and SMB 3.0 dialects. Throughout this blog, I use the terms SMB2 and SMB3 interchangeably because both share the core SMB2 Packet format. Details on encryption configuration aspects can be found in the blogs listed in references.

 

Capability activation

• The client and server advertise encryption support via the SMB2_GLOBAL_CAP_ENCRYPTION capability flag during SMB 3.0 dialect negotiation. Then at least one of the following two options must be happen:
• For enabling encryption for a whole session, SessionFlags of SessionSetup response includes SMB2_SESSION_FLAG_ENCRYPT_DATA. This is the case where encryption is enforced on the whole server.
• For a per-share enabled encryption, ShareFlags of TreeConnect response includes SMB2_SHAREFLAG_ENCRYPT_DATA.

 

When encryption is active for a given exchange, it is applied before submission to the transport. An exchange could consist of a single request or response, or a series of compounded chain operations. The sender performs encryption as an outer layer after the SMB2 messages in a given submission have been assembled.

 

Cryptographic keys

 

Ko = SMB3KDF (Ki, Label, Context)

SMB3KDF() is defined as the KDF algorithm in Counter Mode, as specified in [SP800-108] section 5.1, with 'r' value of 32 and 'L' value of 128, and HMAC-SHA256 as the PRF.

 

Ki – Key derivation key, used as an input to the KDF. For SMB 3.0, Ki is the SessionKey.

Label – the purpose of this derived key, encoded as string and length for SMB 3.0.

Context – the context information of this derived key, encoded as string and length for SMB 3.0.

L – An integer that specifies the length of the derived keying material Ko, L is 128 bits for SMB 3.0 cryptographic keys. Note that L is a constant since all SMB 3.0 keys are 16 bytes in length (SigningKey, EncryptionKey, DecryptionKey, and ApplicationKey).

Ko – Keying material output from the KDF, a binary string of length L, where Ko is the leftmost L bits of KDF result.

 

• ServerInKey (ClientToServer) = SMB3KDF (SessionKey, "SMB2AESCCM\0", "ServerIn \0")

EncryptionKey (Client) = DecryptionKey (Server) = ServerInKey (ClientToServer)

• ServerOutKey (ServerToClient) = SMB3KDF ( SessionKey, "SMB2AESCCM\0", "ServerOut\0")

EncryptionKey (Server) = DecryptionKey (Client) = ServerOutKey (ServerToClient)

 

The server and client generate the encryption keys upon session establishment of the primary channel. If the server is configured for encryption (i.e. SmbServerConfiguration.EncryptData) and the Connection.ClientCapabilities includes the SMB2_GLOBAL_CAP_ENCRYPTION, the server generates the EncryptionKey and DecryptionKey and sets SMB2_SESSION_FLAG_ENCRYPT_DATA flag in the SessionFlags field of the SessionSetup response; the client must also generate its encryption and decryption keys.

 

Upon successful TreeConnect, if Session.EncryptData is FALSE and Share.EncryptData (e.g. the share is configured to require encryption), the server generates its EncryptionKey and DecryptionKey and sets the SMB2_SHAREFLAG_ENCRYPT_DATA bit in the ShareFlags field of the TreeConnect response; the client must also generate its encryption and decryption keys.

 

Note: These cryptographic keys are all derived from the SessionKey. As a result, SMB 3.0 signing and encryption is as secure as the session key. Not only must this key be unique and very random, but also it needs be kept secret.

 

Transformed message

 

A transformed message consists of a transform_header followed by its encrypted SMB2 message.

The transform_header has the following fields:

ProtocolId (4 bytes): 0xFD, 'S', 'M', and 'B' (in network order).

Signature (16 bytes): Signature of the encrypted message.

Nonce (16 bytes): An implementation-specific value that is unique for every encrypted message within a session.

OriginalMessageSize (4 bytes): The size in bytes of the SMB2 message.

Reserved (2 bytes): Set to zeros and ignored.

EncryptionAlgorithm (2 bytes): the value 0x0001 is for SMB2_ENCRYPTION_AES128_CCM.

SessionId (8 bytes): Uniquely identifies the established session for the command.

 

With the SMB2_ENCRYPTION_AES128_CCM algorithm, SMB 3.0 encryption implementation uses the leftmost 11 bytes of the Nonce field in the transform_header, let’s call this Aes128Nonce. Per [RFC3610], the nonce passed to AES-CCM can be between 7 and 13 bytes. Windows SMB 3.0 made the trade-off choice of 11 bytes.

 

Note that it would almost be impossible that all possible nonce values can be consumed on a channel, because it would take over several centuries before a sender could recycle a nonce on a given channel. 

 

Encrypting the message

 

The sender builds the transformed message with these specifics. AES-CCM [RFC5084] is called with the following inputs:

-        AES key: Session.EncryptionKey.

-        Aes-nonce: Aes128Nonce must be used.

-        Plaintext: The SMB2 message including the header and the payload.

-        The optional authenticated data (AAD): The SMB2 transform_header excluding the ProtocolId and Signature fields; these are the 32 bytes starting from the Nonce field.

 

The AES-CCM outputs are:

-        Ciphertext: the encrypted SMB2 message

-        Message authentication code: the Signature field of the transform_header.

 

The sender appends the encrypted SMB2 message to the transform_header and sends it to the receiver.

 

Decrypting the message

 

The message is decrypted using:

-        The EncryptionAlgorithm in the transform_header.

-        The Session.DecryptionKey of the Session that corresponds to the SessionId in the transform_header.

-        The AAD passed to AES-CCM is the transform_header excluding the ProtocolId and Signature fields.

-        The nonce passed to AES-CCM is the Aes128Nonce; that is the first 11 bytes of the Nonce field.

 

The signature returned by the decryption algorithm is then verified against the Signature in the transform_header.

 

Encryption clauses

 

The sender encrypts the message if any of the following conditions is satisfied:

• If the sender is sending a response to an encrypted request.
• If Session.EncryptData is TRUE and the request or response being sent is not NEGOTIATE.
• If Session.EncryptData is FALSE, the request or response being sent is not NEGOTIATE or SESSION_SETUP or TREE_CONNECT, and <TreeConnect|Share>.EncryptData is TRUE.

 

Note: TreeConnect.EncryptData is on the client side. Share.EncryptData is on the server side.

 

Review of encryption clauses

 

• All clauses exclude any operation which does use a SessionId. A SessionId is needed to find the Session object and derive the encryption and decryption keys from its session key. For example, if the client sends a non-encrypted ECHO, Windows 8 server will respond with a non-encrypted response.
• Clause “response to an encrypted request”: if the sender encrypts the request, the receiver will respond in the same way by encrypting the response. There is however a prerequisite that encryption is active, i.e. encryption keys have been generated. For example, if the client encrypts an ECHO, Windows 8 server responds in-kind by encrypting the response. 
• Clause "Session.EncryptData is TRUE": SMB 3 session setup encryption goes as follows:

-- Initial session setup messages are un-encrypted as there is no session object (key etc).
-- Session binding requests must be signed, but not encrypted. Note: Windows-based server does not encrypt session setup responses for session binding, regardless whether the client encrypts the session binding request. On the other side, Windows client does not encrypt session binding requests when Session.EncryptData is TRUE.
-- Spontaneous re-authentication of a valid session must be encrypted, otherwise the server returns STATUS_ACCESS_DENIED.
-- Re-authentication of an expired session is encrypted as well.
-- For re-authentication at reconnection after a broken connection, if the client lost all of its connections to the server, then the client would reconnect and create a new session (initial session setup). If the client lost some of its connections but not all, then the client would reconnect and bind to the existing session (session setup binding).

• Clause “Session.EncryptData is FALSE and <TreeConnect|Share>.EncryptData is TRUE”:

If the client performs TREE_DISCONNECT before a LOGOFF, the logoff will not be encrypted.

 

Compounded messages

 

It is expected that all operations in an encrypted message belong to the SessionId in the transform_header. The encrypted message is sent as a single submission to the underlying transport, there is no provision for a next transformed message in the transform_header.

 

Related compounded messages

 

In case of related compounded requests, all the requests/responses belong to the same SessionId. The sender indicates to the receiver that it is using the SessionId, TreeId, and FileId of the previous operation or resulted from the receiver processing that previous operation.

 

Unrelated compounded messages

 

From a broader scope, the expected protocol behaviors for sending an encrypted message can be summarized as follows, be it for compounded requests/responses or not.

The sender should not send an encrypted request separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption. In other words, the sender should not use a SessionId to convoy the encrypted traffic of other authenticated sessions.

The receiver should not send an encrypted response separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption. Windows-based servers will respond in-kind to what the client sends. If the client violates the above statement regarding the SessionId, a Windows 8-based server will currently send the response as such.

 

Oplock and Lease Breaks

 

Oplock break notifications/acknowledgments/responses must be encrypted when encryption is active. For an Oplock, the FileID is used to derive the SessionId which is set in the notification/acknowledgement/response. See more details in MS-SMB2.

Lease break notifications - sent by the server - do not have a SessionId, and as a result, are neither signed nor encrypted. Lease keys are not tied to a particular session from the client.

However, Lease break acknowledgements sent by the client - and their responses sent by the server - must be encrypted when encryption is active. The client is responsible for selecting a session associated with one of the existing opens associated with that Lease Key. The response is sent on the session that receives the acknowledgment.

 

Test vector

 

This sample data should be considered “as-is”. It should also be noted that examples do not replace normative protocol specifications. The reference must be [MS-SMB2].

 

The test program running on a Windows 8 client negotiates SMB3 and communicates with a Windows 2012 server. It opens a file and WRITEs the following content. It then READs back the file.

This is the content written and read:

Smb3 encryption testing

Hex value:

536D623320656E6372797074696F6E2074657374696E67

 

These outputs show the encryption and decryption of the WRITE and READ commands.

The decrypted content is verified to be same at the end of the SMB2 READ response.

 

SessionId 0x8e40014000011
SessionKey 0xB4546771B515F766A86735532DD6C4F0
SigningKey 0xF773CD23C18FD1E08EE510CADA7CF852
EncryptionKey (Client) 0x261B72350558F2E9DCF613070383EDBF
DecryptionKey (Client) 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
ApplicationKey 0x77432F808CE99156B5BC6A3676D730D1

 

Header.Command 0x0009 WRITE
Encryption of the request ---
Key 0x261B72350558F2E9DCF613070383EDBF
AES-128-CCM nonce 0x66E69A111892584FB5ED52
AAD
0x
66E69A111892584FB5ED524A744DA3EE87000000000001001100001400E40800
SMB2 packet
0x
FE534D4240000100000000000900400008000000000000000400000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
3100700017000000000000000000000015010000390000020100000039020000
00000000000000007000000000000000536D623320656E6372797074696F6E20
74657374696E67
transform_header.Signature 0x81A286535415445DAE393921E44FA42E
transform_header.Nonce 0x66E69A111892584FB5ED524A744DA3EE
transform_header.OriginalMessageSize 0x87
transform_header.SessionId 0x8e40014000011
Encrypted message
0x
25C8FEE16605A437832D1CD52DA9F4645333482A175FE5384563F45FCDAFAEF3
8BC62BA4D5C62897996625A44C29BE5658DE2E6117585779E7B59FFD971278D0
8580D7FA899E410E910EABF5AA1DB43050B33B49182637759AC15D84BFCDF5B6
B238993C0F4CF4D6012023F6C627297075D84B7803912D0A9639634453595EF3
E33FFE4E7AC2AB
Transformed message
0x
FD534D4281A286535415445DAE393921E44FA42E66E69A111892584FB5ED524A
744DA3EE87000000000001001100001400E4080025C8FEE16605A437832D1CD5
2DA9F4645333482A175FE5384563F45FCDAFAEF38BC62BA4D5C62897996625A4
4C29BE5658DE2E6117585779E7B59FFD971278D08580D7FA899E410E910EABF5
AA1DB43050B33B49182637759AC15D84BFCDF5B6B238993C0F4CF4D6012023F6
C627297075D84B7803912D0A9639634453595EF3E33FFE4E7AC2AB
Decryption of the response ---
Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
Transformed message
0x
FD534D42A6015530A18F6D9AFFE22AFAE8E66484860000000000000011000014
00E4080050000000000001001100001400E40800DBF46435C5F14169293CE079
E344479BF670227E49873F458672C3098DAC467DD5809F369D67409166515787
1483E01F7BECD02064EAC3E235F913668BBC2F097980D4B378F1993EFF6E60D1
77309E5B
transform_header.Signature 0xA6015530A18F6D9AFFE22AFAE8E66484
transform_header.Nonce 0x86000000000000001100001400E40800
transform_header.OriginalMessageSize 0x50
transform_header.SessionId 0x8e40014000011
AES-128-CCM nonce 0x8600000000000000110000
AAD
0x
86000000000000001100001400E4080050000000000001001100001400E40800
Decrypted SMB2 packet
0x
FE534D4240000100000000000900210009000000000000000400000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
11000000170000000000000000000000

 

Header.Command 0x0008 READ
Encryption of the request ---
Key 0x261B72350558F2E9DCF613070383EDBF
AES-128-CCM nonce 0xA5123A25F983E245983F41
AAD
0x
A5123A25F983E245983F413B8B429AF271000000000001001100001400E40800
SMB2 packet
0x
FE534D4240000100000000000800400008000000000000000500000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
3100000017000000000000000000000015010000390000020100000039020000
0000000000000000000000000000000000
transform_header.Signature 0xE93601498B76D6F7A72D5EF9B6C79FAF
transform_header.Nonce 0xA5123A25F983E245983F413B8B429AF2
transform_header.OriginalMessageSize 0x71
transform_header.SessionId 0x8e40014000011
Encrypted message
0x
9A464F709AA663F8C2FC3907D63CBF6F98B1E3DD649ED366009FD0B40A365224
718E5440E053F6E01AE462FDB721BF91C3A6E52E14F9EFF005F445761289FF12
72908B52754C8FCB949F228AC104A66204289A205BCBC47509D04AF9A907002B
96863358B3B7CBA5E377930074FCDF3550
Transformed message
0x
FD534D42E93601498B76D6F7A72D5EF9B6C79FAFA5123A25F983E245983F413B
8B429AF271000000000001001100001400E408009A464F709AA663F8C2FC3907
D63CBF6F98B1E3DD649ED366009FD0B40A365224718E5440E053F6E01AE462FD
B721BF91C3A6E52E14F9EFF005F445761289FF1272908B52754C8FCB949F228A
C104A66204289A205BCBC47509D04AF9A907002B96863358B3B7CBA5E3779300
74FCDF3550
Decryption of the response ---
Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
Transformed message
0x
FD534D42ABD518B68C2F04D7879F482B689EB83F870000000000000011000014
00E4080067000000000001001100001400E40800493D6FE2BDBEB435CF5F5469
70C7BB57BF20E713C75A3D045507E0D68E5C0346659D6FFB8AC1504A786CA2BB
89C9E7FE4F313E910A04180D2D0EA7DF636329E5A3285984500EF86FE9D55DA4
FAB9531CFDD4C551D47F3C73124BB4590A45052B694048B991CCF5
transform_header.Signature 0xABD518B68C2F04D7879F482B689EB83F
transform_header.Nonce 0x87000000000000001100001400E40800
transform_header.OriginalMessageSize 0x67
transform_header.SessionId 0x8e40014000011
AES-128-CCM nonce  0x8700000000000000110000
AAD
0x
87000000000000001100001400E4080067000000000001001100001400E40800
Decrypted SMB2 packet
0x
FE534D4240000100000000000800210009000000000000000500000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
11005000170000000000000000000000536D623320656E6372797074696F6E20
74657374696E67

 

References:

 

[MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3 Specification

[SP800-108] National Institute of Standards and Technology. "Special Publication 800-108, Recommendation for Key Derivation Using Pseudorandom Functions", October 2009, https://csrc.nist.gov/publications/nistpubs/800-108/sp800-108.pdf

[RFC5084] Housley, R., "Using AES-CCM and AES-GCM Authenticated Encryption in the Cryptographic Message Syntax (CMS)", RFC 5084, November 2007, https://www.ietf.org/rfc/rfc5084.txt

SMB 3 Security Enhancements in Windows Server 2012

https://blogs.technet.com/b/filecab/archive/2012/05/03/smb-3-security-enhancements-in-windows-server-2012.aspx

Encryption in SMB3

https://blogs.msdn.com/b/openspecification/archive/2012/06/08/encryption-in-smb3.aspx