This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 57.00000000000001%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAE%3C/text%3E%3C/svg%3E)
We are capturing data through an event hub every 15 minutes and consolidating it in a data lake. With a view to data mining we would like to copy these avros into a SQL that acts as a data warehouse. My question is:
-How to treat this avro in order to keep a part of the body and copy it into the SQL table created, is it possible through DataFactory or should it be done in another way?
An Apache Spark-based analytics platform optimized for Azure.
An Azure service for ingesting, preparing, and transforming data at scale.
Yes, this is possible in ADF. Make sure to use data flows in ADF for this because event hub uses complex data types in Avro and you'll need the data flow Spark environment in order to process that data.
So within data flow is it necessary to incorporate some kind of script or something similar? I find this part a bit complicated as it doesn't allow me to preview the avro either. I'm sorry if this question is a bit silly but I've been stuck on this issue for a few days. Do you have any example?
Thank you in advance
No script, you just design your transformation workflow in the ADF designer. Add an Avro source and make sure that your dataset does not include any schema. In the Source Projection tab, click "import projection".
Hello again,
I have set up the schematic as you suggested and imported the projection, however I have the following problems:
As you can see in the image there are different parameters and the idea is that each desired parameter is stored in a column of the SQL table previously created. However, in the Data Factory preview, the AVRO has the following structure:
Why does this happen, is it normal, how could this AVRO be transformed in order to keep only the necessary values, e.g. temperature, and consolidate them in each column of the PostgreSQL table? What language is used, or how could it be done?
-For the other hand, Event Hub consolidates the data sent in an Azure Data Lake Gen 2. The consolidation is done in AVRO format in a nested folder structure, i.e. each time a new event arrives a new folder is generated. Each event can contain multiple messages from IoT devices, i.e. there is not one event per message.
How can it be done so that all existing AVRO folders are scrolled through and not just one AVRO file is displayed?
Is it possible that only new data is copied into the final table?
Thnak you very much in advance