Creating a dynamic Avro Dataset in Azure Data factory

Keith Springs 21 Reputation points

I've been exploring options on how to create a dynamic source to pull the most recent data from my blob storage and into azure data factory but I can't seem to find any tutorial explaining how to do that. Most point to using concat or a SQL database, but I'm unsure if that would work with Avro files.

To summarize what I'm looking for is: Avro Container Storage --- Transfers most recent file---> Dataset in Data Factory

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,137 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 38,597 Reputation points Microsoft Employee

    Hello @Keith Springs ,

    Welcome to the Microsoft Q&A platform.

    A similar requirement has been discussed in this post on how to pick only the most recent file and process it through ADF copy activity to copy data to desired destination data source. Please feel free to try it out and let me know if you have any specific questions.

    Hope this will help. Please let us know if any further queries.

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Keith Springs 21 Reputation points

    I apologize for the extremely delayed answer, but I was able to find the answer to my problem. To handle copying all the data and transforming all at once I had to avoid importing the schema in the copy data tool. For some reason the copy data tool doesn't like Avro's schema.

    I flattened the hierarchy and allowed it to copy dynamically. The only issue now is transforming the data properly back to the body being a dynamic column.

    Thank you for your answers.

    0 comments No comments