"No result available for stream" error when performing join and previous step uses cached lookup

Question

I have a data flow where I look up and cache rows that have been updated as part of an ETL pipeline. I then want to join that updated data with the existing data in the target database. The problem seems to be that when doing the join operation the part of the data flow with the cached sink is only evaluated up to the point where the stream that's part of the join exists. The cached sink is not evaluated even though it's needed in the source before the join. How can I get around this? It seems like this is a bug in ADF with evaluation order.

I've watched some tutorials doing similar things but in all of them they fetched the entire target table. Which is very inefficient.

The top section in the screenshot below is composing the updated data and writing to the cache. The bottom section uses the cache output to select data from the target database and then attempts to join with one of the streams from the top section.

Answer 1

I figured it out. I just needed to create a source and sink just for caching updated Ids and then use that cached data to fetch updated data from both the source and target databases.