How to connect to Azure Blob Storage in Airflow connections

Nandini T 0

Hi Team, I am connecting to Azure Blob storage from Airflow UI connections using Active Directory Auth and sas_token. When I use Wasb_hook inbuilt method for checking file-prefix getting Azure.core.exeptions.ResourceNotFoundError:Not Found error. I think the Azure connection is using Default Azure credentials. Could you please tell how to pass Azure storage account name when using wasb_hook and how to resolve Resource not found error.

Nandini T 0

  File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 181, in execute
     return_value = self.execute_callable()
   File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 198, in execute_callable
     return self.python_callable(*self.op_args, **self.op_kwargs)
   File "/home/airflow/gcs/dags/azure_test.py", line 32, in azure_check_blob_prefix
     if wasb_hook.check_for_prefix(container_name=container,prefix=file_name_prefix):
   File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py", line 240, in check_for_prefix
     blobs = self.get_blobs_list(container_name=container_name, prefix=prefix, **kwargs)
   File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py", line 265, in get_blobs_list
     for blob in blobs:
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/paging.py", line 123, in __next__
     return next(self._page_iterator)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/paging.py", line 75, in __next__
     self._response = self._get_next(self.continuation_token)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_list_blobs_helper.py", line 100, in _get_next_cb
     process_storage_error(error)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_shared/response_handlers.py", line 184, in process_storage_error
     exec("raise error from None")   # pylint: disable=exec-used # nosec
   File "<string>", line 1, in <module>
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_list_blobs_helper.py", line 93, in _get_next_cb
     return self._command(
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
     return func(*args, **kwargs)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_generated/operations/_container_operations.py", line 2718, in list_blob_hierarchy_segment
     map_error(status_code=response.status_code, response=response, error_map=error_map)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/exceptions.py", line 164, in map_error
     raise error
 azure.core.exceptions.ResourceNotFoundError: Not Found

1 answer

KarishmaTiwari-MSFT 20,467 Reputation points Microsoft Employee

2024-03-23T03:48:20.81+00:00

@Nandini T The ResourceNotFoundError often occurs when the access level of your container is set to Private, restricting access.

To resolve this, consider changing the access level of your container:

-In the Azure Portal, navigate to your storage account.

-Click on the Blob service section.

-Select the specific Blob or Container you want to modify.

-Adjust the Access policy to allow either Blob or Container access based on your requirements.

When using the Wasb_hook, you need to provide the storage account name explicitly. The hook expects the following parameters: wasb_conn_id: The connection ID for your Azure Blob storage. container_name: The name of the container. account_name: The Azure storage account name. sas_token: The Shared Access Signature (SAS) token (if applicable).
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to connect to Azure Blob Storage in Airflow connections

1 answer

Your answer