Error when creating view on pipeline (problem with BULK path)

Mateo Estrada Bazan 81 Reputation points
2023-01-04T04:52:23.7+00:00

Good morning everybody!

Me and my team managed to create part of a pipeline which selects the database and creates a data source named 'files'.
Now we want to create a view in the same Azure Synapse pipeline using a Script activity. However, this error comes up:

{ "errorCode": "2011", "message": "Content of directory on path '01_landing/test/year=*/month=*/day=*/*.parquet' cannot be listed.", "failureType": "UserError", "target": "CREATE VIEW", "details": [] }

This is our query. If we run it manually on a script in the Develop section everything works smoothly:

CREATE OR ALTER VIEW our_view AS SELECT *, res.filepath(1) AS [year], res.filepath(2) AS [month], res.filepath(3) AS [day] FROM OPENROWSET( BULK '01_landing/test/year=*/month=*/day=*/*.parquet', DATA_SOURCE = 'files', FORMAT='PARQUET' ) AS [res]

Is there another way to pass wildcard characters such as '*' to the BULK path for our pipeline to work correctly?
Thanks in advance!

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,150 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,161 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.