Answer (thanks to Gary Nebbett)
detailsupdate
Many thanks to Gary Nebbett for taking pity on me and lending his considerable experience tracking this down.
Also many thanks to Metin Savignano for the weeks he also spent helping me on this problem and trying different configurations.
Without them, I would have continued to bang my head on the wall for another few years with no progress.
I hope Microsoft fixes this Outlook bug.
Outlook EMABLT32.DLL bug
Bug in Outlook (Office15, Office16, Microsoft365) DLL EMABLT32.DLL incorrectly calls for "userSMIMECertificate" instead of "userSMIMECertificate;binary". 2-byte patch developed by Gary Nebbett changes string pointer to correct "binary" version. Here are the patched EMABLT32.DLL. Compare ("cmp -lb") with your original to make sure it is only the two-byte change before replacing yours. BEWARE:Windows seems to put the old one back sometimes.
Microsoft 365 (Office16) (129416 bytes sha256..77bb81a3e C:\Program Files (x86)\Microsoft Office\root\Office16\EMABLT32.DLL )
Patched sha256..2b62c7c751011
$ cmp -lb original patched
12419 70 8 14 ^L
12420 150 h 154 l
Office16 (165272 bytes sha256..47561d10dcbce C:\Program Files\Microsoft Office\root\Office16\EMABLT32.DLL )
Patched sha256..322292f932
$ cmp -lb original patched
16586 173 { 313 M-K
16587 174 | 200 M-^@
Office15 (125728 bytes sha256..f2b158f8f4db C:\Program Files\Microsoft Office 15\root\office15\EMABLT32.DLL )
Patched sha256..f310013071
$ cmp -lb original patched
15857 200 M-^@ 130 X
15858 26 ^V 32 ^Z
Now I finally have an answer to the question:
"What exactly is the contents of the LDAP userSMIMECertificate attribute" (see stackexchange)
openssl smime -sign -in empty.txt -text -outform der -out userSMIMECertificate.der -nodetach -signer mycert.pem -inkey mykey.pem -certfile intermediatecerts.pem
The contents of userSMIMECertificate.der is what goes on the wire (in the OCTET). Sample pcap dump is here
Alternative Fix
An alternative solution that un-does Outlook disabling AIA chaining like what browsers do anyway. This is also thanks to Gary's keen sense and eye. Here is the link. Please read caveats on that link re security.
Making it all work.
If you are reading this, I assume you have a working s/mime setup and certificate already.
FYI: Actalis.it still offers free s/mime certificates. Google them.
- Either 1) apply the 2-byte patch to your EMABLT32.DLL to see userSMIMECertificate;binary; or 2) allow Outlook to chain (follow AIA path) from the cert in userCertificate;binary by making the Registry fix.
- Start Outlook
- Add your ldap server to the Outlook Address Books. You can test with either ldap.ends2ends.com(1) or ldapx.ends2ends.com(2) (no SSL and default Base).
- Restart Outlook
- If you installed the patch correctly or alternate fix, you should be able to send an encrypted email to checkme@ends2ends.com. You should also be able to send encrypted emails to anyone using (1) or (2) or whatever ldap/AD server you have set.
Notes:
-(1) ldap.ends2ends.com This is a ldap-2-dns gateway with dnssec validation and hence securely scales globally. Send a signed email (no body) to checkme@ends2ends.com which should respond with the dns record you need to paste into your dns zonefile. Your dns must be secured with dnssec.
-(2) ldapx.ends2ends.com This is an example of a ldap "key-server" similar to what pgp does. Send a signed email to checkme@ends2ends.com. In approx 5 min your public key data will be publicly available to anyone that includes this address book and they will be able to send you encrypted email w/o first exchanging certificates. This one is just for test so it does not do a full certificate validation before accepting new certs/signatures. A production version would validate certs chains to CAs before publishing.
-(3) userSMIMECertificate as presented here and in (1)+(2) works out of the box for Outlook for Mac, 9Folders, .. many more w/o bug fix. Works with MS Outlook w/ bug fix.