Creating a dynamic Avro Dataset in Azure Data factory

Question

I've been exploring options on how to create a dynamic source to pull the most recent data from my blob storage and into azure data factory but I can't seem to find any tutorial explaining how to do that. Most point to using concat or a SQL database, but I'm unsure if that would work with Avro files.

To summarize what I'm looking for is: Avro Container Storage --- Transfers most recent file---> Dataset in Data Factory

Answer 1

Hello @Keith Springs ,

Welcome to the Microsoft Q&A platform.

A similar requirement has been discussed in this post on how to pick only the most recent file and process it through ADF copy activity to copy data to desired destination data source. Please feel free to try it out and let me know if you have any specific questions.

• ADF: copy last modified blob
• Here is a video demonstration from one of my colleagues on similar requirement. Please feel free to leverage it and enhance the flow design as per your need - Get Latest File from Folder and Process it in Azure Data Factory

Hope this will help. Please let us know if any further queries.

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Answer 2

I apologize for the extremely delayed answer, but I was able to find the answer to my problem. To handle copying all the data and transforming all at once I had to avoid importing the schema in the copy data tool. For some reason the copy data tool doesn't like Avro's schema.

I flattened the hierarchy and allowed it to copy dynamically. The only issue now is transforming the data properly back to the body being a dynamic column.

Thank you for your answers.