I understand that you are encountering a "404 The specified blob does not exist" error intermittently while retrieving blobs via the Apache proxy.
This issue may be due to Authentication or incorrect URL problems.
Here are the few things you need to check:
- In the case of an Azure Data Lake storage account configured with an Azure network private endpoint, I suggest configuring the private endpoint for both blob and DFS endpoints.
- https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints#creating-a-private-endpoint
- Ensure that Azure storage blob data contributor permissions are granted for the authentication account or service principal.
- https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access?tabs=portal#assign-an-azure-role
- If the issue persists, I recommend enabling Azure Storage diagnostics logs. Reproduce the problem, and carefully analyze the logs for additional insights.
If you need further assistance or clarification on any of the outlined steps, please don't hesitate to reach out and we would be happy to assist you.