Cannot convert a Azureml.Core.Dataset type object into a pyspark dataframe.

Ameya Bhave 0

I have created a azureml dataset and want to convert it into a pyspark dataframe. I am not able to do it because I am getting a ParseException

import mltable
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
from azureml.core import Dataset, Workspace, Datastore
from pyspark.sql import SparkSession


spark = SparkSession.builder.appName("MySparkApp").config("spark.some.config.option", "some-value").getOrCreate()

datastore_name = 'datastore_name'
workspace = Workspace.from_config() #if we have existing work space.
datastore = Datastore.get(workspace, datastore_name)
df = Dataset.Tabular.from_parquet_files(path=(datastore,'path/file_name.parquet'))
spark_df = df.to_spark_dataframe()

When I do this, I get an error saying

AzureMLException: AzureMLException: Message: Execution failed unexpectedly due to: ParseException InnerException Syntax error at or near '['(line 1, pos 276) > == SQL ==> all the column names are written here> -----------------^^^ ErrorResponse { "error": { "message": "Execution failed unexpectedly due to: ParseException" } }

Why is this happening?
Please help

Ramr-msft 17,736 Reputation points

2024-02-28T08:39:28.4966667+00:00

Thanks for the question, The from_parquet_files method expects a string or a list of strings as the path argument, not a tuple. You can fix this by providing the path as a string: df = Dataset.Tabular.from_parquet_files(path=f"{datastore}/path/file_name.parquet")
Ameya Bhave 0 Reputation points

2024-02-28T15:31:45.9133333+00:00

Hi @Ramr-msft
Thanks for your reply. I was following this link where I got that syntax: https://learn.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_factory.tabulardatasetfactory?view=azure-ml-py#from-parquet-files-path--validate-true--include-path-false--set-column-types-none- It can be found under the from_parquet_files() section. I tried your syntax and I got this error:

"Invalid argument type for path. It can be single value or a list of: string starting with "http://" or "https://", azureml.data.datapath.DataPath object, or tuple of azureml.core.Datastore object and string for relative path in the datastore.Url of blob can't contains SAS token, please use datastore instead."

Share via

Cannot convert a Azureml.Core.Dataset type object into a pyspark dataframe.

Your answer