@Alfredo Revilla (MSFT) hi sorry it took me so long to get the information together.
Had to send this as an answer as it is the only way i could get the forum to allow me to post the reply.
We are able to obtain the bearer token:
curl --location --request POST 'https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Cookie: fpc=xxxxxxxxxxxxxxxxxxxx; stsservicecookie=estsfd; x-ms-gateway-slice=estsfd' \
--data-urlencode 'client_secret=xxxxxxxxxxxxxxxxxxxxxxxxxx' \
--data-urlencode 'scope=api://xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/.default' \
--data-urlencode 'client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
Then when attempting to access the site using our bearer token we get the 401.71 2147500037 error code.
The application log shows:
HTTP Error 401.71 - Unauthorized
You do not have permission to view this directory or page.
Most likely causes:
The authenticated user does not have access to a resource needed to process the request.
Things you can try:
Create a tracing rule to track failed requests for this HTTP status code. For more information about creating a tracing rule for failed requests, click here.
Detailed Error Information:
Error Code 0x80004005
Requested URL http://xxxxxxxxxx:80/
Physical Path D:\home\site\wwwroot
Logon Method Not yet determined
Logon User Not yet determined
This is the generic Access Denied error returned by IIS. Typically, there is a substatus code associated with this error that describes why the server denied the request. Check the IIS Log file to determine whether a substatus code is associated with this failure.
View more information »
Then the IIS log shows this:
2022-09-21 16:21:19 XXXXXXXXXXXXXXXXX GET / X-ARR-LOG-ID=2017f9cd-64d3-4305-924d-029d37c53390 80 - ::1 AlwaysOn ARRAffinity=270fc76c7a748acb7bb3a328ed3b3e85783de79ee41831feff7c3c2118b4802a - XXXXXXXXXXXXXXXXX.azurewebsites.net 401 71 2147500037 705 693 13
So it looks like the bearer token is letting me in. But then I'm getting some sort of failure due to lack of permissions.
When I set this same thing up on my test site, it works and I can access the page.
The Azure web application folder permissions are probably the culprit here but I don't really know what to look at in terms of how to grant this access.
So my enterprise azure web app team needs to update something but we don't know what.
Should we open a support ticket to get assistance with that?
What are you using to crawl? One option is to use Azure Cognitive Search which allows key-based and role-based authentication.