AI Studio Integration with Azure Search: Permission and Authentication Issues

Question

I am trying to securely configure Azure AI Studio, and therefore, I have the AI Studio Hub, Azure Search, and OpenAI securely set up with private endpoints, all associated with VNETs with public access disabled.

My Search instance already has several preloaded indexes. When I try to configure the search in AI Studio, I encounter the following errors:

ERROR NUMBER 1: When attempting to link an existing index within AI Studio (Create an index)

Failed to create vector index.

"The file(s) couldn't be uploaded because the storage is behind a virtual network, the supplied credentials might not have access, or the storage account permissions might not be set up correctly. If access was granted recently, it may take several minutes to update in the system - you can simply try again later."

Error NUMBER 2: When trying to create a new index

During the entire process of creating an Experiment/Job in Azure ML Studio, in the LLM - Crack, Chunk, and Embed Data stage, I get the following error:

Error Code: ScriptExecution.WriteStreams.Authentication  

Native Error: "Permission denied while trying to write: Some(This request is not authorized to perform this operation using this permission.)."  

PermissionDenied(Some(This request is not authorized to perform this operation using this permission.))  

Error Message: "Authentication failed when trying to access destination. Make sure you have correct permissions set up. Ok(This request is not authorized to perform this operation using this permission.)" | session_id=8f620d18-c110-4e5a-be13-eb4b46356a60

• I am not sure exactly where additional permissions might be required.
• We have already assigned all the necessary IAM roles across all components involved in the process.
• What additional checks could be performed to ensure we aren't missing any permissions?

Answer 1

The issues you're encountering seem to revolve around restricted access due to virtual network settings and storage permissions, which often happen when resources like storage accounts or other Azure services are locked behind private endpoints. Here are a few steps to troubleshoot both errors:

Error 1: Failed to Create Vector Index

1. Storage Access via VNET:
   • Ensure that the storage account hosting your index files is accessible within the same virtual network as your AI Studio.
   • Check that Private Endpoints are properly configured for the storage account and AI Studio.
   • Confirm that the Network Rules on the storage account allow traffic from your AI Studio's subnet.
2. Permissions:
   • Make sure the AI Studio's managed identity or service principal has the appropriate roles on the storage account (e.g., Storage Blob Data Contributor or Storage Blob Data Owner).
   • Double-check that these roles apply at the correct scope (account, container, or specific blob level).
3. Firewall Settings:
   • Verify that the storage account firewall settings are not inadvertently blocking access. Ensure the Trusted Microsoft Services toggle is enabled under the networking settings of the storage account.
4. Timeout Issues:
   • If access was granted recently, the error message hints that there may be a delay in propagating permissions. Try waiting for a few minutes and attempt the operation again.

Error 2: Authentication Failed during Experiment/Job Creation

1. AI Studio and Azure ML Permissions:
   • Ensure that the Azure Machine Learning workspace and the AI Studio Hub have the appropriate access permissions. You may need to assign IAM roles like Contributor or Owner for resources like storage, virtual networks, and AI Studio.
2. Service Principal Permissions:
   • If you’re using a service principal for authentication, verify that it has been granted the necessary permissions, including access to both the storage and the virtual network.
   • Specifically, the role Storage Blob Data Contributor or Storage Blob Data Owner needs to be applied to the service principal on the target storage account.
3. VNET/Subnet Access:
   • Confirm that your Azure Machine Learning compute resources and AI Studio Hub are in the same virtual network or peered networks with correct NSG (Network Security Group) rules in place to allow communication between them.
4. Key Vault Permissions:
   • If you are using a Key Vault to manage credentials, ensure that your AI Studio has the correct Key Vault Reader or Key Vault Contributor permissions to access keys or secrets, especially for storage account keys if you’re not using managed identities.
5. Azure Managed Identity:
   • If you’re using managed identities (which is recommended), ensure that the managed identity of your AI Studio or Azure ML instance is assigned the necessary roles across all involved services (e.g., OpenAI, storage, and search).

Additional Checks:

• Logs & Diagnostic Settings: Enable diagnostic logs on your storage account and AI Studio to capture more detailed errors. This will help you pinpoint where the permission issue lies.
• RBAC Permissions: Review the roles assigned at each scope in the Azure Portal. Sometimes roles are mistakenly applied at the resource group level instead of the specific resource, or vice versa.

If these steps don’t resolve the issue, try running a Network Trace from the VM or service running the experiment to verify network access across services.