How to handle SEC_I_MESSAGE_FRAGMENT when performing a DTLS handshake via the SChannel SSPI?

Question

How to handle SEC_I_MESSAGE_FRAGMENT when performing a DTLS handshake via the SChannel SSPI?

Haddon CD 21

When performing a DTLS handshake using the SChannel SSPI in Windows 10 - for which there is no documentation - how should the application handle a SEC_I_MESSAGE_FRAGMENT result from AcceptSecurityContext (ASC) or InitializeSecurityContext (ISC)?

I understand that I am meant to send the recieved fragment to the other party and call ASC/ISC again to obtain the next fragment, but when I call ASC again, this time with an empty input SECBUFFER_TOKEN, I receive nothing in the output token buffer, and it returns SEC_I_MESSAGE_FRAGMENT - suggesting it is expecting input data.

Presumably I am not successfully indicating to ASC that I want it to give me the next fragment, so how do I do this?

I have created a standalone example that reproduces my issue in this GitHub gist:
https://gist.github.com/haddoncd/381c5e9542e977ca238ff16229bd9a0e/c881132ced94995402b617f38a5c8b6f8669b637

I have also included in the gist example output from the program which shows in detail the inputs that lead to the issue:
https://gist.github.com/haddoncd/381c5e9542e977ca238ff16229bd9a0e/c881132ced94995402b617f38a5c8b6f8669b637#file-example_output-txt

Wilhelm Noeker 1 Reputation point

2021-05-11T10:05:21.583+00:00
I tinkered with your code example for a while but didn't come up with any improvement. (The SECBUFFER_EXTRA on input struck me as odd, but apparently it just belongs there as a peculiarity of the DTLS world.)

What I did find are two pieces of code on GitHub that look very promising:

Microsoft's WindowsProtocolTestSuites, written in C#. RDP over UDP seems to use DTLS, and the ProtoSDK/Sspi/DTLS and ProtoSDK/MS-RDPEMT/SecureChannel folders contain related code.

A combined TLS/DTLS library by Christopher Dickson. Client only, but plain C, and looks very well written.

HTH
Haddon CD 21 Reputation points

2021-05-11T16:37:09.957+00:00

@Wilhelm Noeker , thanks.

I had actually already seen the WindowsProtocolTestSuites code, which AFAICT is doing the same as I am, but I stopped short of compiling and stepping through with a debugger to verify that.

I hadn't seen libmatoya before, so I'll look into that to see if I can reverse engineer from there.

The SECBUFFER_EXTRA on input was something I learned from https://stackoverflow.com/a/65853027, which appears to be semi-confirmed by the WindowsProtocolTestSuites, which IIRC passes an empty buffer?
The extra buffer is presumably used by the server to generate a client-address-specific cookie for the HelloVerifyRequest mechanism (which is not in TLS - it was added in DTLS to prevent DoS amplification).

Accepted answer

0 additional answers

Your answer

Wilhelm Noeker 1 Reputation point

2021-05-11T10:05:21.583+00:00

I tinkered with your code example for a while but didn't come up with any improvement. (The SECBUFFER_EXTRA on input struck me as odd, but apparently it just belongs there as a peculiarity of the DTLS world.)

What I did find are two pieces of code on GitHub that look very promising:

Microsoft's WindowsProtocolTestSuites, written in C#. RDP over UDP seems to use DTLS, and the ProtoSDK/Sspi/DTLS and ProtoSDK/MS-RDPEMT/SecureChannel folders contain related code.

A combined TLS/DTLS library by Christopher Dickson. Client only, but plain C, and looks very well written.

HTH
Haddon CD 21 Reputation points

2021-05-11T16:37:09.957+00:00

@Wilhelm Noeker , thanks.

I had actually already seen the WindowsProtocolTestSuites code, which AFAICT is doing the same as I am, but I stopped short of compiling and stepping through with a debugger to verify that.

I hadn't seen libmatoya before, so I'll look into that to see if I can reverse engineer from there.

The SECBUFFER_EXTRA on input was something I learned from https://stackoverflow.com/a/65853027, which appears to be semi-confirmed by the WindowsProtocolTestSuites, which IIRC passes an empty buffer?
The extra buffer is presumably used by the server to generate a client-address-specific cookie for the HelloVerifyRequest mechanism (which is not in TLS - it was added in DTLS to prevent DoS amplification).

Answer 1

Pu Xu 81 Microsoft Employee

According to my test, the only WRONG thing you did is forget to RESET context.handle once initialized during handshaking.

c
isc_status = InitializeSecurityContextW(
    &creds,
    context.initialized ? &context.handle : nullptr,
    nullptr,
    context_reqs,
    0,
    SECURITY_NATIVE_DREP,
    isc_input_buffers,
    0,
    &context.handle, // HERE
    &out_buffer_desc,
    &context.attrs,
    &context.expiry
);

asc_status = AcceptSecurityContext(
    &creds,
    context.initialized ? &context.handle : nullptr,
    &in_buffer_desc,
    context_reqs,
    SECURITY_NATIVE_DREP,
    &context.handle, // HERE
    &out_buffer_desc,
    &context.attrs,
    &context.expiry
);

BTW, you don't need to call DeleteSecurityContext on the old CtxtHandle because it's been invalid after the call.

Haddon CD 21 Reputation points

2021-09-11T10:35:00.933+00:00

Hi Pu Xu, thanks for your reply, looks like that was exactly it.

It seems I was mislead by this line from the docs for InitializeSecurityContextW:

When using the Schannel SSP, on calls after the first call, pass the handle returned here as the phContext parameter and specify NULL for phNewContext.

I've updated my gist with working code for for the benefit of anyone who finds this question.

Share via

How to handle SEC_I_MESSAGE_FRAGMENT when performing a DTLS handshake via the SChannel SSPI?

0 additional answers

Your answer