BCryptEncrypt is failing when AES-128-GCM algorithm called with 16 byte IV

Question

BCryptEncrypt is failing when AES-128-GCM algorithm called with 16 byte IV

Subbiah Senguttuvan (XC-AN/EPA1) 0

I am trying to use Microsoft Bcrypt libraries to encrypt using AES 128 bit GCM. The BCryptEncrypt function works well as long as IV size is <= 12 bytes. When I use 16 byte IV then BCryptEncrypt fails with INVALID PARAMETER(0xC000000D) Error. Are there any other ways to reconfigure Algorithm selection and encrypt data with 16 byte IV

Subbiah Senguttuvan (XC-AN/EPA1) 0

Here is my code snippet.

int appAES128b_GCMEncryption(byte AES_PlainText[], byte AES_CipherText[], byte AES_Key[], byte Nonce[], byte AuthData[], byte Mac[], int len_input, int len_Nonce, int len_AuthData, int len_Mac)
{
	int returnValue;
	const int AES_Key_Length = 16;
	BCRYPT_ALG_HANDLE       hAlg = NULL;
	BCRYPT_KEY_HANDLE       hKey = NULL;
	NTSTATUS                status = STATUS_UNSUCCESSFUL;
	DWORD                   cbCipherText = 0,
		cbPlainText = 0,
		cbData = 0,
		cbKeyObject = 0,
		cbBlockLen = 0;

	PBYTE                   pbCipherText = NULL,
		pbPlainText = NULL,
		pbKeyObject = NULL;

	unsigned int ulen_Mac = len_Mac;

	//Additional information
	CipherMode_additional_info.dwInfoVersion = 1;
	CipherMode_additional_info.cbSize = 0x40;
	CipherMode_additional_info.pbNonce = NULL;
	CipherMode_additional_info.cbNonce = 0;
	CipherMode_additional_info.pbAuthData = NULL;
	CipherMode_additional_info.cbAuthData = 0;
	CipherMode_additional_info.pbTag = NULL; //Receives Authenticated tag from BCryptEncrypt
	CipherMode_additional_info.cbTag = 0; //use BCryptGetProperty to query the BCRYPT_AUTH_TAG_LENGTH property
	CipherMode_additional_info.pbMacContext = NULL;
	CipherMode_additional_info.cbMacContext = 0; //If BCryptEncrypt and BCryptDecrypt calls are not being chained, this member must be set to NULL. use BCryptGetProperty to query the BCRYPT_AUTH_TAG_LENGTH property
	CipherMode_additional_info.cbAAD = NULL; //This member is used only when the BCRYPT_AUTH_MODE_IN_PROGRESS_FLAG flag in the dwFlags member is set. On the first call to BCryptEncrypt or BCryptDecrypt you must set this field to zero.
	CipherMode_additional_info.cbData = 0; //This member is used only when the BCRYPT_AUTH_MODE_IN_PROGRESS_FLAG flag in the dwFlags member is set. On the first call to BCryptEncrypt or BCryptDecrypt you must set this field to zero.
	CipherMode_additional_info.dwFlags = 0; // BCRYPT_AUTH_MODE_CHAIN_CALLS_FLAG; //This flag is used when chaining BCryptEncrypt or BCryptDecrypt function calls.If calls are not being chained, this member must be set to zero.



	returnValue = -1;
	ulen_Mac = len_Mac;
	// Open an algorithm handle.
	if (!NT_SUCCESS(status = BCryptOpenAlgorithmProvider(
		&hAlg,
		BCRYPT_AES_ALGORITHM,
		MS_PRIMITIVE_PROVIDER,
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptOpenAlgorithmProvider\n", status);
		returnValue = -2;
		goto Cleanup;
	}

	//Configure as GCM
	if (!NT_SUCCESS(status = BCryptSetProperty(
		hAlg,
		BCRYPT_CHAINING_MODE,
		(PBYTE)BCRYPT_CHAIN_MODE_GCM,
		sizeof(BCRYPT_CHAIN_MODE_GCM),
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptSetProperty\n", status);
		returnValue = -2;
		goto Cleanup;
	}

	// Calculate the size of the buffer to hold the KeyObject.
	if (!NT_SUCCESS(status = BCryptGetProperty(
		hAlg,
		BCRYPT_OBJECT_LENGTH,
		(PBYTE)&cbKeyObject,
		sizeof(DWORD),
		&cbData,
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptGetProperty\n", status);
		returnValue = -2;
		goto Cleanup;
	}

	// Allocate the key object on the heap.
	pbKeyObject = (PBYTE)HeapAlloc(GetProcessHeap(), 0, cbKeyObject);
	if (NULL == pbKeyObject)
	{
		//wprintf(L"**** memory allocation failed\n");
		returnValue = -2;
		goto Cleanup;
	}

	// Calculate the block length.
	if (!NT_SUCCESS(status = BCryptGetProperty(
		hAlg,
		BCRYPT_BLOCK_LENGTH,
		(PBYTE)&cbBlockLen,
		sizeof(DWORD),
		&cbData,
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptGetProperty\n", status);
		returnValue = -2;
		goto Cleanup;
	}


	// Calculate the size of the Tag 
	if (!NT_SUCCESS(status = BCryptGetProperty(
		hAlg,
		BCRYPT_AUTH_TAG_LENGTH,
		(PUCHAR)&authTagLength,
		sizeof(authTagLength),
		&cbData,
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptGetProperty\n", status);
		returnValue = -2;
		goto Cleanup;
	}

	if ((ulen_Mac >= authTagLength.dwMinLength) & (ulen_Mac <= authTagLength.dwMaxLength))
	{
		if ((ulen_Mac - authTagLength.dwMinLength) % authTagLength.dwIncrement != 0)
		{
			//printf("Mac Tag length error");
			returnValue = -2;
			goto Cleanup;
		}
		CipherMode_additional_info.cbTag = ulen_Mac;
		CipherMode_additional_info.pbTag = (PBYTE)HeapAlloc(GetProcessHeap(), 0, CipherMode_additional_info.cbTag);
		if (NULL == CipherMode_additional_info.pbTag)
		{
			//wprintf(L"**** memory allocation failed\n");
			returnValue = -2;
			goto Cleanup;
		}
	}
	else
	{
		//printf("Mac Tag length error");
		returnValue = -2;
		goto Cleanup;
	}

	// Allocate a buffer. The buffer is consumed during the  encrypt/decrypt process.
	if (len_Nonce > 0)
	{
		CipherMode_additional_info.pbNonce = (PBYTE)HeapAlloc(GetProcessHeap(), 0, len_Nonce);
		if (NULL == CipherMode_additional_info.pbNonce)
		{
			//wprintf(L"**** memory allocation failed\n");
			returnValue = -2;
			goto Cleanup;
		}
		CipherMode_additional_info.cbNonce = len_Nonce;
		memcpy(CipherMode_additional_info.pbNonce, Nonce, CipherMode_additional_info.cbNonce);
	}

	if (len_AuthData > 0)
	{
		CipherMode_additional_info.pbAuthData = (PBYTE)HeapAlloc(GetProcessHeap(), 0, len_AuthData);
		if (NULL == CipherMode_additional_info.pbAuthData)
		{
			//wprintf(L"**** memory allocation failed\n");
			returnValue = -2;
			goto Cleanup;
		}
		CipherMode_additional_info.cbAuthData = len_AuthData;
		memcpy(CipherMode_additional_info.pbAuthData, AuthData, CipherMode_additional_info.cbAuthData);
	}
	else if (0 == len_AuthData)
	{
		CipherMode_additional_info.pbAuthData = NULL;
	}

	// Generate the key from supplied input key bytes.
	if (!NT_SUCCESS(status = BCryptGenerateSymmetricKey(
		hAlg,
		&hKey,
		pbKeyObject,
		cbKeyObject,
		(PBYTE)AES_Key,
		AES_Key_Length,
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptGenerateSymmetricKey\n", status);
		returnValue = -2;
		goto Cleanup;
	}

	cbPlainText = len_input;
	pbPlainText = (PBYTE)HeapAlloc(GetProcessHeap(), 0, cbPlainText);
	if (NULL == pbPlainText)
	{
		//wprintf(L"**** memory allocation failed\n");
		returnValue = -2;
		goto Cleanup;
	}

	memcpy(pbPlainText, AES_PlainText, len_input);


	//Calculate output buffer size
	if (!NT_SUCCESS(status = BCryptEncrypt(
		hKey,
		pbPlainText,
		cbPlainText,
		(PBYTE)&CipherMode_additional_info,
		NULL,
		NULL,
		NULL,
		NULL,
		&cbData,
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptEncrypt\n", status);
		returnValue = status;
		goto Cleanup;
	}
	cbCipherText = cbData;
	pbCipherText = (PBYTE)HeapAlloc(GetProcessHeap(), 0, cbCipherText);
	if (NULL == pbCipherText)
	{
		//wprintf(L"**** memory allocation failed\n");
		returnValue = -2;
		goto Cleanup;
	}


	// Use the key to encrypt the plaintext buffer.
	// For block sized messages, block padding will add an extra block.
	if (!NT_SUCCESS(status = BCryptEncrypt(
		hKey,
		pbPlainText,
		cbPlainText,
		(PBYTE)&CipherMode_additional_info,
		NULL,
		NULL,
		pbCipherText,
		cbCipherText,
		&cbData,
		0)))
	{
		//wprintf(L"**** Error 0x%x returned by BCryptEncrypt\n", status);

		returnValue = status;
		goto Cleanup;
	}
	else
	{
		//wprintf(L"Successful encryption by BCryptEncrypt\n", status);
		returnValue = cbCipherText;

		memcpy(AES_CipherText, pbCipherText, cbCipherText);
		memcpy(Mac, CipherMode_additional_info.pbTag, CipherMode_additional_info.cbTag);
	}


Cleanup:

	if (hAlg)
	{
		BCryptCloseAlgorithmProvider(hAlg, 0);
	}
	if (hKey)
	{
		BCryptDestroyKey(hKey);
	}
	if (pbCipherText)
	{
		HeapFree(GetProcessHeap(), 0, pbCipherText);
	}
	if (pbPlainText)
	{
		HeapFree(GetProcessHeap(), 0, pbPlainText);
	}
	if (pbKeyObject)
	{
		HeapFree(GetProcessHeap(), 0, pbKeyObject);
	}
	if (CipherMode_additional_info.pbTag)
	{
		HeapFree(GetProcessHeap(), 0, CipherMode_additional_info.pbTag);
	}
	if (CipherMode_additional_info.pbNonce)
	{
		HeapFree(GetProcessHeap(), 0, CipherMode_additional_info.pbNonce);
	}
	if (CipherMode_additional_info.pbAuthData)
	{
		HeapFree(GetProcessHeap(), 0, CipherMode_additional_info.pbAuthData);
	}
	return returnValue;
}

Tong Xu - MSFT 2,546 Reputation points Microsoft External Staff

2023-08-31T06:12:50.1166667+00:00

Hi, @Subbiah Senguttuvan (XC-AN/EPA1)
Any ideas on my answer? Have your problems solved?
Subbiah Senguttuvan (XC-AN/EPA1) 0 Reputation points

2023-08-31T09:20:15.8233333+00:00

Hi @Tong Xu,

Thank you for the response. I still trying to figure out how CBC is relevant for GCM.

My Problem is not solved yet. I referred Encrypting Data with CNG and customized for AES GCM, thats where I observe 12 byte size IV limitation. Are there any possibility to extend CNG to handle 16 byte IV AES GCM encryption?
Tong Xu - MSFT 2,546 Reputation points Microsoft External Staff

2023-09-01T03:30:39.1566667+00:00

Hi, @Subbiah Senguttuvan (XC-AN/EPA1)

About extend CNG to handle 16-byte IV AES GCM encryption.
The answer is negative.

NIST SP 800-38D Page 8 says: For IVs, it is recommended that implementations restrict support to the length of 96 bits, to promote interoperability, efficiency, and simplicity of design.
Windows Cryptography API is designed basing on it. The length of the IV depends on the protocol and mode.

There are more information and samples in wiki.
Subbiah Senguttuvan (XC-AN/EPA1) 0 Reputation points

2023-09-04T13:14:08.63+00:00

Hi @Tong Xu,

Thank you for the confirmation. It helps. I will find another solution.
Tong Xu - MSFT 2,546 Reputation points Microsoft External Staff

2023-09-18T06:04:50.9433333+00:00

Hi, @Subbiah Senguttuvan (XC-AN/EPA1)
Do you have any update on other solution?
Subbiah Senguttuvan (XC-AN/EPA1) 0 Reputation points

2023-09-21T09:46:24.46+00:00

Hi @Tong Xu,

Not yet. I am still researching suitable implementation that can fit into my environment.

1 answer

Your answer

Tong Xu - MSFT 2,546 Reputation points Microsoft External Staff

2023-08-31T06:12:50.1166667+00:00

Hi, @Subbiah Senguttuvan (XC-AN/EPA1)
Any ideas on my answer? Have your problems solved?
Subbiah Senguttuvan (XC-AN/EPA1) 0 Reputation points

2023-08-31T09:20:15.8233333+00:00

Hi @Tong Xu,

Thank you for the response. I still trying to figure out how CBC is relevant for GCM.

My Problem is not solved yet. I referred Encrypting Data with CNG and customized for AES GCM, thats where I observe 12 byte size IV limitation. Are there any possibility to extend CNG to handle 16 byte IV AES GCM encryption?
Tong Xu - MSFT 2,546 Reputation points Microsoft External Staff

2023-09-01T03:30:39.1566667+00:00

Hi, @Subbiah Senguttuvan (XC-AN/EPA1)

About extend CNG to handle 16-byte IV AES GCM encryption.
The answer is negative.

NIST SP 800-38D Page 8 says: For IVs, it is recommended that implementations restrict support to the length of 96 bits, to promote interoperability, efficiency, and simplicity of design.
Windows Cryptography API is designed basing on it. The length of the IV depends on the protocol and mode.

There are more information and samples in wiki.
Subbiah Senguttuvan (XC-AN/EPA1) 0 Reputation points

2023-09-04T13:14:08.63+00:00

Hi @Tong Xu,

Thank you for the confirmation. It helps. I will find another solution.
Tong Xu - MSFT 2,546 Reputation points Microsoft External Staff

2023-09-18T06:04:50.9433333+00:00

Hi, @Subbiah Senguttuvan (XC-AN/EPA1)
Do you have any update on other solution?
Subbiah Senguttuvan (XC-AN/EPA1) 0 Reputation points

2023-09-21T09:46:24.46+00:00

Hi @Tong Xu,

Not yet. I am still researching suitable implementation that can fit into my environment.

Answer 1

Hi, @Subbiah Senguttuvan (XC-AN/EPA1)
Welcome to Microsoft Q&A!

AES-GCM iv size is 12. Other IV lengths will require additional calculations.
https://crypto.stackexchange.com/questions/41601/aes-gcm-recommended-iv-size-why-12-bytes.
And see the discussion: https://github.com/v2ray/v2ray-core/issues/2130.
If you want 16b IV, it's recommended to use CBC mode. Please refer to Encrypting Data with CNG.

Thank you.

If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Share via

BCryptEncrypt is failing when AES-128-GCM algorithm called with 16 byte IV

1 answer

Your answer