How to get rid of Base64 encoding on event body

Question

I develop my own IoT device using Azure SDK for Embedded C IoT Hub Client.

When publishing MQTT events I do not use Base64 encoding on message body, something like this:

  StaticJsonDocument<255> doc;
  doc["Temp"] = String(data.Temp);
  doc["Hum"] = String(data.Hum);
  doc["Battery"] = String(data.Battery);
  char strJSON[255] = {0};
  serializeJson(doc, strJSON);        
  mqtt_client.publish(topic, strJSON, false);

Using cloud shell I can see events arriving on IoT Hub, and payload is a simple JSON formatted data:

I then set up event routing to forward all telemetry data to a CosmosDB endpoint. However, in CosmosDB I see item's body in base64 encoded format.

Why and where this encoding happens ? How to get rid of this ? I mean I understand that I can use logic apps, triggers, whatever, to decode, but I am more interested if I can somewhere "turn off " something that is encoding the body ?

Answer 1

Hi @Laszlo Frank The encoding is done as the message gets send to the IoT Hub, which is the PubSubClient step in this case. To overcome this, we need to specify the Content Type and Encoding Type before sending the message.

Since we are publishing the data using MQTT topic name, we can set these parameters at the end as follows.

  doc["Temp"] = "25";
  doc["Hum"] = "46";
  doc["Battery"] = "Full";
  char strJSON[255] = {0};
  //JsonObject jobject = doc.to<JsonObject>();
  serializeJson(doc, strJSON); 
  char topic[] = "devices/<IoTDevice>/messages/events/$.ct=application%2Fjson%3Bcharset%3Dutf-8";     
  mqtt_client.publish(topic, strJSON, false);
  //mqtt_client.beginMessage(telemetry_topic);
  //mqttClient.print(strJSON);
  Serial.println("OK");
  delay(2000);
  digitalWrite(LED_PIN, LOW);

Please note to replace the Topic with the appropriate device to which you would like to send the data. Once you set the parameters, you can view the message in the correct format. Please refer the below image for reference.

Here is the documentation on MQTT provided on IoT Hub that provides this explanation https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-mqtt-support#sending-device-to-cloud-messages

Kindly note that, the message format you see when you monitor the built in event end point in Azure cloud shell is not accurate depiction of the data received by the IoT Hub. If you route the data to blob container, you would see the message would still be in base64 format.

Hope this addresses the issue you are facing. Please let us know if you still need any further clarification on this.

---

If this answers your query, do click Accept Answer  as helpful. And, if you have any further query do let us know by commenting in the below section. Happy to help!

Answer 2

The answer on this question might help you. Let me know if that doesn't solve it.

Answer 3

Hello @Laszlo Frank ,

if you want to work with (Azure) IoT, you need to build up some basic knowledge about both the cloud and the way IoT is implemented in a particular vertical. Training for Azure IoT is available for free.

In this case, messages travel through Azure as 'Event data', see also this documentation.

There you see the body is a byte array. It's up to you what to put in there.

So the body is turned into base64 to prevent possible errors with other resources.

By convention, the IoT Hub and surrounding tooling assume it's just JSON, in Base64 encoding. It is encouraged to be explicit so eg. the IoT Hub routing can support routing on body properties.

But you can also send the body as a GZIPed or Deflated compressed message so a service like Azure Stream Analytics can automatically decompress it (using the right settings).

Back to Base64 messages in CosmosDB, check out this 1-year-old blob post:

The answer is indeed to be explicit about sending the format of the message you send.

I hope this more elaborate answer explains some of the design decisions Microsoft has made to serve all IoT usecases.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.