The answer is no, you cannot use custom policy authentication on IIS side. It must be done inside particular application only. This is due to how client certificate authentication works in Windows: certificate is mapped to a security principal using implicit or explicit mapping.
In implicit mapping, client certificate is mapped to security principal with UPN contained in client certificate's SAN extension. Of course, issuing CA must be trusted for client authentication (added to NTAuthCertificate DS store). This is a default authentication method on IIS and Windows in general (smart cards use implicit mapping using UPNs only).
In explicit mapping, a certificate is compared either with one configured in 1:1 mapping or *:1 (many to one). In 1:1 mapping, reference certificate is added to IIS configuration and mapped to security principal. Exact certificate match is required. The downside of this method is that you will have to use this procedure for every certificate and principal and repeat every time client certificate is renewed.
In many to one mapping, you can define either, subject or issuer, or a combination of both to map to single security principal. this allows different certificates to match criteria, but criterias aren't flexible and use only Issuer/Subject field combination. No support for certificate extensions.
So, again, the answer is no, if you have access, then implement extension authorization policy inside web application.
I want to trust certificates issued by CA containing specific policy identifier number in certificate policies field in certificate. For example, one set of certificates contain "policy identifier = 1.2.3.4" and other set of certificates contain "policy identifier = 5.6.7.8" in certificate policies field. And I want to trust only certificates which ones are containing 1.2.3.4 in certificate policies field on server side. And reject certificates containing 5.6.7.8.
Thanks