Data Flow Output Structure

Question

Hi,
I'm using the output of the Data Flow to capture the rows count by parsing the JSON. It works fine mostly and then suddenly fails saying the property doesn't exist. Is it that the structure of this output keeps changing? What could be a better way of capturing the rows count from data flow?

Answer 1

Hi @39861377 ,
Thankyou for using Microsoft Q&A platform and posting your query.
As per your query, It seems that you want to fetch the rowCount processed by the data flow . It was working fine till date but suddenly it has started throwing error saying 'Join2' property doesn't exist in the data flow output.

Looking at your expression that you are passing, I suspect that the Join2 property is missing in the Stages array. Could you please share the output of Dataflow by attaching it as a .txt file here or atleast share the snapshot of Streams array for the confirmation on the same like below:

Using @activity('dataflow1').output.runStatus.metrics.sink1.stages[1].streams.Join2.count would give the rowcount processed after the Join2 output . To fetch the rowCount processed in the Sink transformation, you can instead use @activity('dataflow1').output.runStatus.metrics.sink1.rowsWritten which would be independent of all the other transformations used in between source and sink transformations.

Please do let us know if the suggested response helped, otherwise please feel free to comment if you have any further question. Thanks !