Hello Uros Stojiljkovic,
Greetings! Welcome to Microsoft Q&A Platform.
I understand that you’re encountering a 403 Forbidden error while trying to write a Parquet file using PySpark. This error typically indicates that your request lacks the necessary permissions to perform the operation.
Please consider checking the below to troubleshoot the issue further,
Ensure that the account or service principal you’re using has the necessary permissions to write to the specified location. This might involve checking your cloud storage permissions (e.g. Azure Blob Storage).
Verify that your authentication credentials (e.g., access keys, tokens) are correctly configured and have not expired.
Double-check the path you’re writing to and ensure it is correct. Sometimes, a typo or incorrect path can lead to access issues.
Ensure that your Spark environment is correctly configured with the necessary access credentials. This might involve setting environment variables or configuring Spark properties.
If you’re working within a corporate network, there might be network policies or firewalls that restrict access to certain resources and if you are using a virtual network (vNet) with service endpoints for the storage account, ensure that the vNet and the APIs are properly configured to allow traffic between them. Check if any network security groups or route tables are blocking the communication.
Review the logs and diagnostics of both the APIs and the storage account to gather more information about the error and identify any potential issues.
Please check the permissions required when using Account Access Key with RBAC- https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access?tabs=portal, https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory,
Hope the above information helps! please let us know if you have any further queries. I’m happy to assist you further.
Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.