Using cached data from one data flow activity as the source for another data flow activity

David Lang 40

On this reference page...

https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink

...MS says, "Write to activity output The cached sink can optionally write your output data to the input of the next pipeline activity. This will allow you to quickly and easily pass data out of your data flow activity without needing to persist the data in a data store."

This is exactly what I want to do but cannot figure out how to reference my cached data set from the first data flow in the second data flow.

I have been told to add a source transformation in the second data flow and select "Cached" as the source type (see Q&A's AI Assist answer) but when I do this there is no Cached option in the UI:

User's image

I have also been told to use a Lookup activity but when I add this to the second data flow and then reference the Cached lookup section in expression builder I get no options...

User's image

Can someone give me very specific steps on how this is done?

Harishga 5,990 Reputation points Microsoft Vendor

2024-05-16T11:01:39.0733333+00:00

@David Lang
Just checking in to see if the below answer provided by @Q&A Assist helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
David Lang 40 Reputation points

2024-05-16T17:50:29.9633333+00:00

@Harishga No, as I explained in my original post, the answer provided by @Q&A Assist has steps that are incorrect, In particular, step 2 says to add a source transformation and select "Cached" as the source type, but as I show in my screenshot in the original post, when I do this there is no option for "Cached". There are only options for "Dataset" and "Inline".
Harishga 5,990 Reputation points Microsoft Vendor

2024-05-17T02:22:24.78+00:00
Hi @David Lang

Thank you for reaching out with your question about using cached data from one data flow activity as the source for another data flow activity in Azure Data Factory.

If you're not seeing the "Cached" option in the Source settings of your Azure Data Factory data flow, there could be several reasons for this.
Here are a few things to check:

Service Version: Ensure that you are using the latest version of Azure Data Factory, as the “Cached” option may not be available in older versions.

Configuration: Double-check the configuration of your first data flow’s sink. It must be set to “Cache” and configured to write to the activity output.

UI Glitch: Sometimes, UI elements may not appear due to a temporary glitch. Try refreshing the page or restarting the browser.

Reference
https://stackoverflow.com/questions/71508220/adf-using-sinked-cache-in-source
https://www.youtube.com/watch?v=9d39Bw2uFLw

I hope this information helps you. Let me know if you have any further questions or concerns.
David Lang 40 Reputation points

2024-05-17T19:31:12.6933333+00:00

Hi @Harishga

I've rechecked all the items you listed above and I still do not see an option for "Cache" when I create a new source. Also, the two links you sent show how to use cache lookup for data in the SAME data flow...I want to be able to do a cache lookup in a DIFFERENT data flow.

Also, I'm going to abandon this investigation. It appears that using the cache is only suitable for smaller data sizes (approx 4MB) and the data we'll need to process is much larger than this, so using the cache is not suitable for our purpose.

Thanks for your reply.
Harishga 5,990 Reputation points Microsoft Vendor

2024-05-21T01:11:08.74+00:00

@David Lang
If you have found a solution to your problem, please share it with the community to help others who may be facing the same issue. If you haven't found a solution yet, feel free to ask here and we will help you.
Nirmal Chowdhary 5 Reputation points

2024-05-24T18:33:57.57+00:00

Hi @David Lang ,
I am also trying to implement a similar solution to pass the staged data from one dataflow activity to another. But I am not able to find a solution over it . Did you find any trick on this. If possible please can you share. I really appreciate you're support.

1 answer

Q&A Assist

2024-05-15T14:57:15.3566667+00:00

AI generated content. This question contains an answer created with AI from Q&A Assist. Learn more
To reference cached data from one data flow activity as the source for another data flow activity, you can follow these steps:

In the first data flow activity, add a sink transformation and select "Cache" as the sink type.

In the second data flow activity, add a source transformation and select "Cached" as the source type.

In the "Cached" source transformation, select the first data flow activity as the source data flow.

Map the columns from the cached data to the input of the second data flow activity.

This will allow you to use the output of the first data flow activity as the input for the second data flow activity without persisting the data in a data store.

References:

Data flow sink

Data flow source
Please sign in to rate this answer.
David Lang 40 Reputation points

2024-05-16T17:51:00.2666667+00:00

This AI generated answer has steps that are incorrect.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Using cached data from one data flow activity as the source for another data flow activity

1 answer

Your answer