Use Apache Kafka and Azure Databricks for streaming

Cunwei zhao 0 Reputation points

Use Apache Kafka and Azure Databricks for streaming,The code is as follows

# Read data from Apache Kafka
from pyspark.sql.functions import *
from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import *

#import pdb; pdb.set_trace() 
kafka_df_zcw = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "").option("", "PLAINTEXT").option("subscribe", "zcw").load()

schema = StructType([
    StructField("speed", IntegerType()),
    StructField("volkswagen", StringType()),   
    StructField("version", StringType()),
    StructField("ts", LongType())

kafka_df_zcw_temp = kafka_df_zcw.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")

                                .start() query_zcw.awaitTermination()

Always prompt as follows:


Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,596 questions
{count} votes