Azure Synapse pyspark translates STRING datatype into varchar(8000) for external table
Grützmacher, Sven
5
Reputation points
Hi there,
we try to load some external tables in Azure Synpase using a PySpark notebook but the datatypes seem to mismatch. When we initialize the table we execute (stripped down example):
CREATE OR REPLACE TABLE LANDING_DB.Opportunity (
AccountId varchar(255),
Id varchar(255),
Name STRING,
Description STRING,
dwh_deleted BOOLEAN)
USING DELTA LOCATION 'abfss://******@XXX.dfs.core.windows.net/Landing/ABC/Opportunity/'
via spark.sql().
Yet the auto-generated table uses 'varchar(8000)' as datatype for the Name and Description column. We would like to use varchar(max) as especially the Description column can have a lot of text.
Are we missing some settings or is this a bug?
Help would be appreciated,
best, Sven
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,357 questions
Sign in to answer