SaveTable in Datalake Table of Azure Synapse from Synapse Spark pool

Mohammed Sohail 66 Reputation points
2022-11-05T07:20:22.86+00:00

I have create a new lake database called DeltaTables and when I'm trying to write a spark dataframe into that Datalake it's showing me success message but I don't see any table created. where as If I write same dataframe in default lakedatabase which is by default came up in Synapse I can see the Table created. So just need some assistance on how to create a lakedatabase that is similar to default lakedatabase which Synapse has created

The code which I have used is df.write.mode("overwrite).saveAsTable("DeltaTables.Test")

The other difference I see with compared to default lakedatabase is default one allows to create views as well. Attached a snap for referrence
257443-image.png

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,337 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,358 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,915 questions
Azure Data Lake Analytics
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 76,836 Reputation points Microsoft Employee
    2022-11-07T11:10:53.11+00:00

    Hello @Mohammed Sohail ,

    Thanks for the question and using MS Q&A platform.

    As per the repro, we are able to write a spark dataframe into lake database and able to see the table created.

    257844-synapse-savetable.gif

    Even we can query the table which has created.

    257837-image.png

    If you try to create a view on a lake database, you will be getting the below error as this is not supported.

    257845-image.png

    Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators
    0 comments
  2. Mohammed Sohail 66 Reputation points
    2022-11-08T11:31:03.873+00:00

    Thanks @PRADEEPCHEEKATLA-MSFT , Actually the tables are getting created in lake database but I was unable to see, but when I connected to powerBI it's listing me out all the tables. Attached some screen shots for your referrence

    This is from PowerBI
    258252-microsoftteams-image.png

    The below is from Synapse
    258139-image.png

    when you compare those images, there are more Tables showing up in PowerBI which I too don't know that it got created untill I connected to PowerBI. Please let me know If you have any idea regarding the same.

    and Talking about views if you try creating views from Synapse notebook using df.createOrReplaceTempView("<view-name>") it's getting created I think this is creating only in memory but I was just wondering why Synapse has given an option of Views on lakedatabse. If you can clarify on that It would be great

    Thanks much !!