question

SivagnanaSundaramKrithiga-7183 avatar image
0 Votes"
SivagnanaSundaramKrithiga-7183 asked AnnuKumari-MSFT commented

Reading using sqlanalytics connector in spark using Notebooks - Synapse

I am reading a table in the Synapse notebooks using the sqlanalytics connector. When the spark encounters an empty string in a column , it is trying to convert to None/Null.

I am getting error, when the column is NOT NULL .

Column ordinal: 7, Expected data type: NVARCHAR(50) collate SQL_Latin1_General_CP1_CI_AS NOT NULL

I am trying to fill with a default value, so I can read the DataFrame. It doesn't convert it somehow.

Is there any work around to this?



azure-synapse-analytics
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @SivagnanaSundaramKrithiga-7183,

Thanks for the question and using MS Q&A platform.

Could you please share the code which you are trying?

Meanwhile, could you please try the below syntax on the column schema and see if that works.

 CREATE TABLE [dbo].[DimCustomer2] (  
     [CustomerKey] INT NOT NULL,  
     [GeographyKey] INT NULL,  
     [CustomerAlternateKey] nvarchar(15) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL  
 )  
0 Votes 0 ·
SivagnanaSundaramKrithiga-7183 avatar image
0 Votes"
SivagnanaSundaramKrithiga-7183 answered AnnuKumari-MSFT commented

CREATE TABLE [dbo].[DimCustomer2] (
[CustomerKey] INT NOT NULL,
[GeographyKey] INT NULL,
[CustomerAlternateKey] nvarchar(15) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL
)

insert into [dbo].[DimCustomer2] values
(
1,1,'')

189202-image.png



image.png (70.0 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @SivagnanaSundaramKrithiga-7183,

Thanks for the additional details.

We are reaching out to the internal team to get the more details on this issue. I will be update you once I hear back from the team.

0 Votes 0 ·
AnnuKumari-MSFT avatar image AnnuKumari-MSFT PRADEEPCHEEKATLA-MSFT ·

Hi @SivagnanaSundaramKrithiga-7183,

When we create an external table from the existing internal table with "" values where the column is NOT NULL. We will get the same error mentioned by you.

This has been identified as a product bug in Synapse SQL by the product team.

This is the response we got from them: "After investigation, It’s DW issue and we got this exception when we create external table internally. Team is working on the fix.

Meanwhile, workaround solution is to remove NOT NULL constraint from the source table. "

We wil keep you posted as soon as we get the confirmation on the fix.


0 Votes 0 ·

Is there any update on this?

0 Votes 0 ·
AnnuKumari-MSFT avatar image AnnuKumari-MSFT SivagnanaSundaramKrithiga-7183 ·

Hi @SivagnanaSundaramKrithiga-7183 ,
There is an incident created for this issue and product team is looking into the same. I have been following up with the team. They have transferred it to the external table product team. Apologies for the inconvenience. I will keep you posted regarding the updates. Thanks !

0 Votes 0 ·
AnnuKumari-MSFT avatar image
0 Votes"
AnnuKumari-MSFT answered AnnuKumari-MSFT commented

Hi @SivagnanaSundaramKrithiga-7183 ,

We got response from product team on the above issue. Kindly have a look:

" Yes, this is the current behavior of Synapse/Polybase: an empty string gets treated as null on export. The upcoming Gen3 DW (ded SQL Pool) will address this which will preview late 2022. Since the fix is non-trivial, there are no plans to fix for current Gen2.

However, a couple options can be done to get unblocked here:

1. When doing CETAS export, convert the empty strings to a unique placeholder value or even an empty space. The round-trip will work and differentiate null vs. empty fields.

  CREATE EXTERNAL TABLE [dbo].[abc]
     WITH (DATA_SOURCE = [SQLAnalyticsConnectorDataSourceTest1],LOCATION = N'/abc',FILE_FORMAT = [SQLAnalyticsConnectorDefaultFileFormat798e8ca0543342a6b43e0787ab2a7db1])
     AS
     SELECT
     CASE WHEN name = '' THEN ' ' END AS name
     FROM "dbo"."abc1"

2. Use CSV instead of parquet and specify a STRING_DELIMITER. This will natively support null/empty string differentiation as well without any work-arounds. "

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
    Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @SivagnanaSundaramKrithiga-7183 ,
Just following up to see if the answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

0 Votes 0 ·

Hi @SivagnanaSundaramKrithiga-7183 ,
Just following up to see if the answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

0 Votes 0 ·