How can we convert these funcation in Pyspark from Scala in Azure Databricks
Hi All,
i have below code that is working fine in Scala, but due to requirement we need to change this in Pyspark. we are new in Pyspark unable to convert in Pyspark
Scala code
import org.apache.spark.sql.types.{DataType, StringType, StructField, StructType}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser;def toSparkType(inputType: String): DataType = CatalystSqlParser.parseDataType(inputType)
def getSchema(inputType: String) : StructType = StructType(inputType.split(",").map(line ⇒ line.split(" ")).map(field => StructField(field(0),toSparkType(field(1)),true)))
val NewFileSchema = getSchema(Source_Schema_V)
print(" newfileschema print "+NewFileSchema)val NewFileDF = spark.read.option("header", "true").schema(NewFileSchema).csv(Source_File)
value of Source_Schema_V="ID int,FirstName string,LastName string,Department string,Created_Date string,LastUpdated_Date string"
Please help on this