HDInsight on AKS deployment fails when using Azure Policy Add-On

Question

We are in the process of running a PoC to deploy Spark on Azure. For this we are trying out HDInsight on AKS (Spark).

When creating the HDInsight on AKS cluster node pool, we are encountering an issue when Azure Policy is enabled on the underlying AKS Cluster (enforced by CCoE team through a DeployIfNotExists policy).

Deployment: deployment.apps/upgrade-operator

Pod: pod/upgrade-operator-XXX

Container: upgrade-operator-webhook

Error:

Caused by: io.kubernetes.client.openapi.ApiException: class V1Status {
    apiVersion: v1
    code: 400
    details: null
    kind: Status
    message: the API version in the data (networking.k8s.io/v1) does not match the expected API version (crd.projectcalico.org/v1)
    metadata: class V1ListMeta {
        _continue: null
        remainingItemCount: null
        resourceVersion: null
        selfLink: null
    }
    reason: BadRequest
    status: Failure
}

We suspect there is a conflict between the calico CRD's deployed when enabling Azure Policy Add-On for AKS and the resources deployed by the HDInsights cluster node pool.