Synapse Studio - Lake Database Designer and Delta

Question

Synapse Studio - Lake Database Designer and Delta

Sascha Müller 1

We have some Issues with the Lake Database Designer.

If we create the Database by Create Table with pySpark then we can neither use the Database Designer nor we can clone that Database.

If we create the Database first with the Designer and then execute our pySpark Create Table Statements then we first do not see any of those created Tables in the Synapse Studio Designer.

If we create them manually, they are created (seems like Drop and Create), BUT they are then in Parquet and not in Delta and we can't choose Delta in the Synapse Studio Designer GUI.

In the Documentation is Delta not mentioned, will Delta also be supported by the Designer?
https://learn.microsoft.com/en-us/azure/synapse-analytics/database-designer/create-empty-lake-database

PRADEEPCHEEKATLA 90,661 Reputation points Moderator

2022-10-17T06:40:02.647+00:00

Hello @Sascha Müller ,

Thanks for the bringing this to our attention.

We are reaching out to internal team to get more information related to your query and will get back to you as soon as we have an update.
Sascha Müller 1 Reputation point

2022-10-28T08:30:23.637+00:00

Hello Guys, thanks for the reply. Any news on this?

Additionally we are wondering why we do not see Views in Synapse Studio (created via SSMS).

Best regards

Sascha
PRADEEPCHEEKATLA 90,661 Reputation points Moderator

2022-11-02T10:35:40.807+00:00

Hello @Sascha Müller ,

We are still awaiting for the response from the internal team and will get back to you as soon as we have an update.
PRADEEPCHEEKATLA 90,661 Reputation points Moderator

2022-11-07T05:59:51.093+00:00

Hello @Sascha Müller ,

We haven't received any response from internal team. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.
Leslie 56 Reputation points

2022-11-29T18:22:20.497+00:00

So there is a difference between creating an object - vs what I call a GIT object. If you create a table via a notebook you won't have the correspondng json artifact in your git repo. This is where the designer - vs the objects vs the GIT repo are failing and causing issues all over the board. I believe they are way behind for getting delta fully supported into the designer and making the designer actually work - I have a brand new bug today that I was just about to post.

Hopefully this helps a bit - get familiar with how the objects appear in json and whether you want to use the designer - right now we are doing our objects via notebook and avoiding the designer. The bad thing about this is we can't easily do CI/CD. Tradeoffs right now but they definitely need to fix some things here.

1 answer

Your answer

PRADEEPCHEEKATLA 90,661 Reputation points Moderator

2022-10-17T06:40:02.647+00:00

Hello @Sascha Müller ,

Thanks for the bringing this to our attention.

We are reaching out to internal team to get more information related to your query and will get back to you as soon as we have an update.
Sascha Müller 1 Reputation point

2022-10-28T08:30:23.637+00:00

Hello Guys, thanks for the reply. Any news on this?

Additionally we are wondering why we do not see Views in Synapse Studio (created via SSMS).

Best regards

Sascha
PRADEEPCHEEKATLA 90,661 Reputation points Moderator

2022-11-02T10:35:40.807+00:00

Hello @Sascha Müller ,

We are still awaiting for the response from the internal team and will get back to you as soon as we have an update.
PRADEEPCHEEKATLA 90,661 Reputation points Moderator

2022-11-07T05:59:51.093+00:00

Hello @Sascha Müller ,

We haven't received any response from internal team. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.
Leslie 56 Reputation points

2022-11-29T18:22:20.497+00:00

So there is a difference between creating an object - vs what I call a GIT object. If you create a table via a notebook you won't have the correspondng json artifact in your git repo. This is where the designer - vs the objects vs the GIT repo are failing and causing issues all over the board. I believe they are way behind for getting delta fully supported into the designer and making the designer actually work - I have a brand new bug today that I was just about to post.

Hopefully this helps a bit - get familiar with how the objects appear in json and whether you want to use the designer - right now we are doing our objects via notebook and avoiding the designer. The bad thing about this is we can't easily do CI/CD. Tradeoffs right now but they definitely need to fix some things here.

Answer 1

Sascha Müller 1

@Leslie Thanks for your comment.

Yes, i would like that all Artefacts (Tables, Views, Stored Proc) would be automatically (e.g. via the Designer) part of the Git.

But currently we need to do this separatly for CD/CI and Backup Purposes.

The Designer has no value to us as long it is how it is.

Share via

Synapse Studio - Lake Database Designer and Delta

1 answer

Your answer