I'm assuming that the Private Endpoint you're referring to is https://learn.microsoft.com/en-us/azure/app-service/networking/private-endpoint. Whether your build agent is hosted on-prem with configured VPN or hosted in an Azure VM that's part of your configured vnet or peered, make sure your firewall is configured to allow traffic from document endpoints:
*.visualstudio.com:
- https://login.microsoftonline.com
- https://app.vssps.visualstudio.com
- https://{organization_name}.visualstudio.com
- https://{organization_name}.vsrm.visualstudio.com
- https://{organization_name}.vstmr.visualstudio.com
- https://{organization_name}.pkgs.visualstudio.com
- https://{organization_name}.vssps.visualstudio.com
dev.azure.com:
- https://dev.azure.com
- https://*.dev.azure.com
- https://login.microsoftonline.com
- https://management.core.windows.net
- https://vstsagentpackage.azureedge.net
EDIT: Furthermore, you can check https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-rm-web-app-deployment?view=azure-devops#web-app-deployment-on-app-service-environment-ase-is-not-working for additional troubleshooting steps that may be preventing deployment to your Private Endpoint. Even though this doc is referring to ASE, it also applies to Private Endpoint.