question

ImranMondal-3977 avatar image
0 Votes"
ImranMondal-3977 asked ImranMondal-3977 commented

copying data from Blob Storage to table storage

Hi Everyone,

We have ETL job running in Databricks and writing the data back to blob storage, Now we have created a table using azure table storage and would like to import the same data (Databricks output) to table storage.

So how to load data from blob to table storage
or
how to write data from databricks to table storage. please help

azure-blob-storageazure-databricksazure-table-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
1 Vote"
KranthiPakala-MSFT answered ImranMondal-3977 commented

Hi @ImranMondal-3977,

Welcome to Microsoft Q&A forum and thanks for your query.

  1. So how to load data from blob to table storage?
    In order to load data from blob storage to table storage, you can use Azure Data factory copy activity to perform the copy job.
    a) Please refer to this to doc to know how to copy data from Azure Blob storage using ADF: Copy and transform data in Azure Blob storage by using Azure Data Factory
    b) Please refer to this to doc to know how to copy data to Azure Table storage using ADF: Copy data to and from Azure Table storage by using Azure Data Factory


  2. How to write data from databricks to table storage
    You can try doing so using the Python SDK as described here: Get started with Azure Table storage and the Azure Cosmos DB Table API using Python (Note: The content in this article applies to both Azure Table storage and the Azure Cosmos DB Table API.)


Hope this info helps.



Thank you
Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you so much for the solution, I will this and will let you know

1 Vote 1 ·

Thank you @KranthiPakala-MSFT for your reply.

As suggested, I used Data Factory to Load CSV data From Blob Storage to Table storage, it worked but I am facing the below issues now.

  • every time the Number of columns is changing in the Source CSV files, which are stored in Blob Storage.

  • Now, I am trying to do incremental load using the last modified option.

  • when I run it next time, the error I am getting is that one particular column that is there in my target column (Table storage) is not available in the source file. this is because every time number of columns is changing in the source file.

My question is, how can I load the column dynamically, If the column is not there it should put as a null.



0 Votes 0 ·