How can we convert these funcation in Pyspark from Scala in Azure Databricks

manish verma 516 Reputation points
2022-06-09T11:14:19.227+00:00

Hi All,

i have below code that is working fine in Scala, but due to requirement we need to change this in Pyspark. we are new in Pyspark unable to convert in Pyspark

Scala code

import org.apache.spark.sql.types.{DataType, StringType, StructField, StructType}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser;

def toSparkType(inputType: String): DataType = CatalystSqlParser.parseDataType(inputType)

def getSchema(inputType: String) : StructType = StructType(inputType.split(",").map(line ⇒ line.split(" ")).map(field => StructField(field(0),toSparkType(field(1)),true)))

val NewFileSchema = getSchema(Source_Schema_V)
print(" newfileschema print "+NewFileSchema)

val NewFileDF = spark.read.option("header", "true").schema(NewFileSchema).csv(Source_File)

value of Source_Schema_V="ID int,FirstName string,LastName string,Department string,Created_Date string,LastUpdated_Date string"

Please help on this

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,514 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.