Answered my own question.
Basically followed the very good video at https://www.youtube.com/watch?v=59YwW8FrLm8
and also made sure my OIDC scope was set, without it you will get an unreadable auth token.
i'm using apache httpd mod_auth_openidc, my config looks like:
OIDCProviderMetadataURL https://login.microsoftonline.com/d989a3a0-9761-4e01-844b-fef2b7c1396c/.well-known/openid-configuration
OIDCRedirectURI http://localhost/foo
OIDCClientID 59d7f244-6541-464b-9ea4-6cdba60509b1
OIDCClientSecret .............
OIDCCryptoPassphrase .................
OIDCScope api://9e59d98b-d747-4a9a-96fd-178f04d08b02
OIDCRemoteUserClaim upn
OIDCPassClaimsAs headers
ProxyPass /foo http://localhost:1234
ProxyPassReverse /rest http://localhost:1234
<Location /foo>
AuthType openid-connect
Require valid-user
</Location>
this proxies to my restapi with http request header containing:
OIDC_access_token: eyJ0eXAiOiJKV1QiLCJhbGci....