From what I understood you need to do the following :
- Create the First Data Flow (Data Flow 1):
- Design your data transformation logic as required
- Add a Sink transformation to write the output
- In the Sink settings, instead of writing to an external storage, configure it to use the Cache option
- Go to the Settings tab of the Sink transformation
- Enable Staging and select Cache as the staging type
- Assign a unique name to the cache (
MyDataFlowCache
)
- Create the Second Data Flow (Data Flow 2):
- Add a Source transformation
- Configure the source to read from the cache created in the first data flow
- Select Cache as the source type
- Provide the cache name used in the first data flow (
MyDataFlowCache
)
More links :
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-cached-lookup-functions
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-activity