How to send compressed Json to azure event hub and read in fabric pyspark streaming ?
Kumar, Sudarshan
0
Reputation points
I am sending compressed data into Event Hub to overcome 1 MB hard limit in Azure Event Hub. I also have to read this in Py-spark and update delta table.
The compressed data send to Event hub is coming as null in Py-spark stream. How to read it?
This is how I am reading from Event Hub
df_stream_body = df_stream.select(F.from_json(F.col("body").cast("string"), message_schema).alias("Payload"))
This is how I am sending data to Event Hub
` async with producer:
# Create a batch.
event_data_batch = await producer.create_batch()
# Add events to the batch.
body = '{"id": "200", "firstName": "Sudarshan70","middleName": "Kumar2","lastName": "Thakur2"}'
# Compress the JSON string using the gzip algorithm.
compressed_body = gzip.compress(body.encode('utf-8'))
event_data_batch.add(EventData(compressed_body))
# Send the batch of events to the event hub.
await producer.send_batch(event_data_batch)`
```when I compress and send data to event hub and read in spark i get null in all filed .