How exactly are existing TSI Preview versions, upgraded towards GEN 2?

Question

Hi,

We curently have a production environment running Time Series Insights Preview. We have been running it for over a year already and we are very satisfied with the performance it gives us to analyse data, even though it was always just a preview version.

However, support for that version is running to an end, at the end of this month. Upon initializing new TSI Gen 2 versions, we realised our current data model is not enough anymore, where it did work for the preview version just fine.

Here is an example of the data load ingested into TSI preview:

{  
	"iothub-connection-device-id": "deviceA",  
	"timestamp": "2020-10-07T13:03:41.707Z",  
	"sensors": [{  
			"sensor": "sensorA",  
			"unit": "mA",  
			"value": 42598  
		}, {  
			"sensor": "sensorB",  
			"unit": "mA",  
			"value": 36666  
		}  
	]  
}  
  
  

We could query this data with TSX filters as such '"tsx": "$event.sensors_sensor.String = 'sensorA' AND $event.sensors_unit.String = 'mA'".

This would work fine. However, the GEN 2 does not allow this anymore, and needs an extra timestamp property per sensor object. We can do that for our newer customers that will get the latest version (GEN 2) of TSI, but we can't do it that easely for our existing production preview environment.

My question is: what happens at the end of this month exactly for existing environments?

Will we have to change the api version towards "api-version=2020-07-31"?
Can we still query the data as currently resides on production without the extra timestamp properties in the sensor objects?
Or would we need to alter all data/ migrate to a new platform and make sure a timestamp property is added to each sensor object?

Currently we can query the data with the API version 2020-07-31 as well, as long as we make the changes described here:
https://learn.microsoft.com/nl-nl/azure/time-series-insights/how-to-api-migration

The main question is: will this still work after October 31st?

--- EDIT ---

An example of the payload is the JSON as described above. That is literally how we send it to an event hub from which it is reading.
We also send data from the devices to the IoT Hub, but this has a very similar structure (only the deviceId is omitted, since it will be added by the IoT Hub anyway):

 {  
     "timestamp": "2020-10-07T13:03:41.707Z",  
     "sensors": [{  
             "sensor": "sensorA",  
             "unit": "mA",  
             "value": 42598  
         }, {  
             "sensor": "sensorB",  
             "unit": "mA",  
             "value": 36666  
         }  
     ]  
 }  

Our Time Series ID is 'iothub-connection-device-id' as described on this page:
https://learn.microsoft.com/en-us/azure/time-series-insights/how-to-select-tsid

Now, for the GEN2 version, we'd just add a timestamp to each sensor object, but we can only easely do this for new environments.

Answer 1

Hello @RensGroen , Thanks for posting this question over this forum.

Below is the response from Microsoft's Time Series Insights Product Team. We hope this response helps with your initial query.

The previous API version is separate and orthogonal to the ingestion flattening rules. The preview API version will be retired at the end of Oct, and thus the syntax of $event['sensors_sensor'].String or $event.sensors_sensor.String but NOT $event.[sensors_sensor].String is accepted.

But the API and the ingestion are two different trains/processes. Any environment made during Preview can (must) use the new API syntax. At some point, the Preview flattening rules will be retired, but there is no timeline currently. You'll have ample time to prepare.

If we do create a new TSI environment with the GA flattening algorithm we might be able to select the composite TS ID of iothub-connection-device-id and sensors.sensor as shown in this example here rather than having to duplicate the time stamp.

In fact, maybe the IoT Hub device ID does not need to be part of the TS ID at all, only the sensor name? It depends on how they want to query and analyze their tags.

We might be able to remove "unit" from the payload and instead store that data as metadata--an instance field that would be a one-time set-up for each tag, but then the message over the wire could be reduced.

Please let us know if you need further help in this matter.

Thank you,
Satish.