How to connect to Azure Blob Storage in Airflow connections

Nandini T 0

Hi Team, I am connecting to Azure Blob storage from Airflow UI connections using Active Directory Auth and sas_token. When I use Wasb_hook inbuilt method for checking file-prefix getting Azure.core.exeptions.ResourceNotFoundError:Not Found error. I think the Azure connection is using Default Azure credentials. Could you please tell how to pass Azure storage account name when using wasb_hook and how to resolve Resource not found error.

Nandini T 0

  File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 181, in execute
     return_value = self.execute_callable()
   File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/python.py", line 198, in execute_callable
     return self.python_callable(*self.op_args, **self.op_kwargs)
   File "/home/airflow/gcs/dags/azure_test.py", line 32, in azure_check_blob_prefix
     if wasb_hook.check_for_prefix(container_name=container,prefix=file_name_prefix):
   File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py", line 240, in check_for_prefix
     blobs = self.get_blobs_list(container_name=container_name, prefix=prefix, **kwargs)
   File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py", line 265, in get_blobs_list
     for blob in blobs:
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/paging.py", line 123, in __next__
     return next(self._page_iterator)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/paging.py", line 75, in __next__
     self._response = self._get_next(self.continuation_token)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_list_blobs_helper.py", line 100, in _get_next_cb
     process_storage_error(error)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_shared/response_handlers.py", line 184, in process_storage_error
     exec("raise error from None")   # pylint: disable=exec-used # nosec
   File "<string>", line 1, in <module>
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_list_blobs_helper.py", line 93, in _get_next_cb
     return self._command(
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
     return func(*args, **kwargs)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/storage/blob/_generated/operations/_container_operations.py", line 2718, in list_blob_hierarchy_segment
     map_error(status_code=response.status_code, response=response, error_map=error_map)
   File "/opt/python3.8/lib/python3.8/site-packages/azure/core/exceptions.py", line 164, in map_error
     raise error
 azure.core.exceptions.ResourceNotFoundError: Not Found

1 answer

KarishmaTiwari-MSFT 19,032 Reputation points Microsoft Employee

2024-03-23T03:48:20.81+00:00

@Nandini T The ResourceNotFoundError often occurs when the access level of your container is set to Private, restricting access.

To resolve this, consider changing the access level of your container:

-In the Azure Portal, navigate to your storage account.

-Click on the Blob service section.

-Select the specific Blob or Container you want to modify.

-Adjust the Access policy to allow either Blob or Container access based on your requirements.

When using the Wasb_hook, you need to provide the storage account name explicitly. The hook expects the following parameters: wasb_conn_id: The connection ID for your Azure Blob storage. container_name: The name of the container. account_name: The Azure storage account name. sas_token: The Shared Access Signature (SAS) token (if applicable).
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

How to connect to Azure Blob Storage in Airflow connections

1 answer