synapse pipeline - blob storage

arkiboys 9,691 Reputation points
2021-07-27T15:04:25.78+00:00

Hello,
The synapse pipelines transfer files into the blob storage as we select the sink as the blob storage folders.
Question:
I know when we created the synapse environment, the data lake storage Gen2 was created by default in the workspace.
Question:
Why is it when files are transferred to the blob storage, then they also can be seen in the storage Gen2 as well?
Is there a relationship between the blob storage and the datalake storage Gen2 ?

Thank you

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,913 questions
0 comments No comments
{count} votes

4 answers

Sort by: Most helpful
  1. Ryan Abbey 1,181 Reputation points
    2021-07-27T20:04:58.577+00:00

    It's one and the same I'd say... just because it's created within the synapse setup doesn't mean it's a separate app to the storage

    0 comments No comments

  2. arkiboys 9,691 Reputation points
    2021-07-28T06:24:05.06+00:00

    So does this mean there is no need to create Azure blob storage and just use the default storage Gen2 ?


  3. MartinJaffer-MSFT 26,086 Reputation points
    2021-08-03T17:44:32.713+00:00

    Hello @arkiboys . If you are not aware, Data Lake Gen 2, is an enhanced version of Azure Storage. Data Lake Gen2 is blob storage with Heirarchical Namespace enabled.

    This said, do not mix the two connectors. Both can be used to read from the same storage account, but only one deals with folders/directories.
    If your storage is a Data Lake Gen 2, only use the Data Lake Gen 2 method for writing.

    Does this make sense?

    0 comments No comments

  4. arkiboys 9,691 Reputation points
    2021-08-04T06:46:33.277+00:00

    Hi,
    I still do not see the difference between the two.
    Our synapse pipelines place data into .parquet files inside the Azure blob storage folders...
    I see that the same folder structures and .parquet files are also added into the storage Gen2 automatically.

    Why is this happening automatically and do I need to use the blob storage?

    Thank you


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.