Hello Team, I am trying to upload the data from snowflake tables (its in AWS) to azure blob storage.I have created the SAS URI Service link .Please refer the screenshot : After running the pipeline i am getting below mentioned error : Please advise . Thanks in advance

Hi @Rohit Kulkarni , According to the screenshot, it looks like you are providing the base URI for the container not the SAS token. If you want to create a SAS token, please follow the instructions from the link below. Thanks! https://learn.microsoft.com/en-us/azure/cognitive-services/translator/document-translation/create-sas-tokens?tabs=Containers https://learn.microsoft.com/en-us/answers/questions/448227/what-are-the-ways-to-revoke-access-to-blob-storage.html https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview --Please 'accept as answer' if the information helps. Thanks--

NoAuthenticationInformation

Accepted answer

Nasreen Akter 10,791 Reputation points

2021-11-23T10:20:51.383+00:00

Hi @Rohit Kulkarni ,

According to the screenshot, it looks like you are providing the base URI for the container not the SAS token. If you want to create a SAS token, please follow the instructions from the link below. Thanks!

https://learn.microsoft.com/en-us/azure/cognitive-services/translator/document-translation/create-sas-tokens?tabs=Containers

https://learn.microsoft.com/en-us/answers/questions/448227/what-are-the-ways-to-revoke-access-to-blob-storage.html

https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview

--Please 'accept as answer' if the information helps. Thanks--
Please sign in to rate this answer.
Rohit Kulkarni 686 Reputation points

2021-11-23T14:56:40.41+00:00

Thanks Nasreen

it is working fine for me now .And one more thing i need to clarify. The files are getting stored in the container. If i run multiple times the pipeline the files are getting created multiple time:

I need only single copy in the container. How to avoid the duplicates. Lease advise

My sink to the blob :

Nasreen Akter 10,791 Reputation points

2021-11-23T16:00:02.5+00:00

@Rohit Kulkarni , would you please provide a screenshot of the DataSet. Thanks!

Rohit Kulkarni 686 Reputation points

2021-11-23T17:23:17.337+00:00

Blob Dataset :

Snowflake Dataset :

Rohit Kulkarni 686 Reputation points

2021-11-23T17:40:22.753+00:00

Blob Dataset :

Snowflake dataset :

Nasreen Akter 10,791 Reputation points

2021-11-23T17:40:44.387+00:00

@Rohit Kulkarni ,

Dataset looks good. From the screenshots above , I see files with different name not with the same name. Would you please clarify a little more, what exactly you are expecting. Thanks!

Rohit Kulkarni 686 Reputation points

2021-11-23T17:44:01.073+00:00

Just consider .The data will be loaded in 1st time the file will be created in containser called :

File 1

and tomorrow again the the pipeline will run and data will be loaded into container with same file name as File1.

It must not create a duplicate of the file. It must overwrite

Regards
RK

Nasreen Akter 10,791 Reputation points

2021-11-23T19:35:28.987+00:00

Hi @Rohit Kulkarni , Everything looks good to me. In this setup, file should be overwritten, should not be duplicate. This is strange that you are getting duplicates files. Would you please give us a screenshot of the duplicate files in the blob storage. Thanks!

Rohit Kulkarni 686 Reputation points

2021-11-24T07:14:12.267+00:00

Today i ran the pipeline and it overwrite the existing file..

And one thing i faced the issue. In one of the Fact table there is 30820286 rows.For this table the file not got created in the container and faced the error.Please refer the screenshot :

Please advise.

Nasreen Akter 10,791 Reputation points

2021-11-24T10:26:50.927+00:00

Hi @Rohit Kulkarni , there is a similar thread with the same data unload issue. would you please check the below link. Thanks.

https://learn.microsoft.com/en-us/answers/questions/355243/data-unload-from-snowflake-to-azure-blob-using-dat.html

Rohit Kulkarni 686 Reputation points

2021-11-24T11:01:24.177+00:00

it is working now

Nasreen Akter 10,791 Reputation points

2021-11-24T16:25:32.217+00:00

@Rohit Kulkarni , great! Would you please mark the answer as 'accepted'. Thanks!
Sign in to comment

Share via

NoAuthenticationInformation

0 additional answers