Best way to copy (and tranform) records between two Azure TSI environments

Petja Belemezova 1

We need to create another TSI environment and transfer all the data from an old TSI instance with minor modifications (for example, convert true/false values to 1/0). What would be the best, most efficient and performant way to do it? Are there any helpful tools to use, or is a custom solution the only way?
The Query API does not seem a very good option as it provides the data in a very different format from the one that needs to be fed into the event hub for TSI.
An important consideration is the number of records that need to be transferred - there are billions at this point and increasing.
We are currently not looking at migration to Azure Data Explorer, this would be done at a later stage.

Thank you.

AshokPeddakotla-MSFT 27,126 Reputation points

2022-10-19T16:29:35.313+00:00

@Petja Belemezova Welcome to Microsoft Q&A forum!

The Query API does not seem a very good option as it provides the data in a very different format from the one that needs to be fed into the event hub for TSI.

Can you elaborate this with more details? We would like to understand more about your challenges and help you accordingly.

I hope you have already gone through the available resources for migration. Currently we only have two ways as you called out in your query, Migrating to new Azure Time Series Insights Gen2 API versions and Migrating Time Series Insights (TSI) Gen2 to Azure Data Explorer
AshokPeddakotla-MSFT 27,126 Reputation points

2022-10-25T04:38:52.733+00:00

@Petja Belemezova Did you get a chance to see my suggestions? Do let us know the requested details or have any further queries.
Petja Belemezova 1 Reputation point

2022-10-25T05:52:59.073+00:00

Hello @AshokPeddakotla-MSFT ,

Thank you for replying.
As I stated before, we are not interested in migrating from TSI Gen1 to TSI Gen2, or from TSI to Azure Data Explorer. What we want to do is transfer data from one TSI Gen2 instance to another TSI Gen2 instance. During this transfer, we want to apply minor modifications to the values of the records, for example, convert "true/false" strings to 1/0 values.

Can you advise on a way to do so? Is it a custom solution the only option?

Thank you!
AshokPeddakotla-MSFT 27,126 Reputation points

2022-11-02T11:01:13.017+00:00

@Petja Belemezova No other ways I'm aware of. Yes, custom solution would be an option in this scenario.
Petja Belemezova 6 Reputation points

2022-11-10T07:19:29.567+00:00

@AshokPeddakotla-MSFT
Thank you! We went with the custom solution and it worked fine.
I have another question, hopefully easier, if you can help me with it:

In TSI explorer, we see the option Event Count for a particular TSI instance. When you click it, you get a chart of the events for this TSI instance for the period selected above. Do you know which is the API endpoint behind this operation and can we use it in a custom solution (i.e. we need to count events for an instance for a given period of time)?

Note: We are not interested in the Aggregate count() operation, which basically counts events per intervals.

Thank you in advance!