True that Data Flow doesn't do HTTP connector. @Mohamed Abobakr (IBS)
I'm thinking maybe break this down into smaller bits, if you don't mind. Can we get clever with the query and the timespan?
For example, with the timespan, can we do multiple queries, each getting a different span of time, like the same query, but 1 month ago, another copying doing same query but 2 months ago, then 3 and so on?
Or perhaps do multiple queries, each retrieving a piece of the data, like one that does all even numbered events, and another doing all odd numbered events? If they do have numbers.
Copying the output of these to blob or lake, we can then operate on them, and perhaps unify at the end.
As I see it, we have several tasks:
- get ALL the data in ANY form
- massage the data into a form you like
- send it to final destination.
I'm poking around to see what transformations I can do now.