Query Data From Data Files in Azure Storage

Question

In AWS cloud we have S3 bucket which contains CSV or Parquet files. Our customers require the catalog information of data files there and query data inside those files. Since downloading and querying is not an option as CSV file can be large. We used AWS Glue and Athena for this purpose, now we can catalog data inside the CSV files and query them as well.

Is it possible to achieve this in Azure Cloud with Azure Storage?

Storage can be Azure Blob Storage or Data Lake Storage, whichever gets the job done.

Answer 1

@Muneeb Mirza Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.

I see you have posted the similar query in SO forum, Kindly try the follow the steps mentioned over-there, kindly let us know, if you still have any questions on this.

For example, the CSV might look like…

Server,schema,database,table,tableDescription
Server1,dbo,mydb,table0001,”This is a really great table!”

If so, you’ll have to write some code on your own to parse the data and conform it to the Atlas Entity REST API calls.
You might take inspiration from the work I’ve done on PyApacheAtlas: pyapacheatlas/reader.py at master · wjohnson/pyapacheatlas (github.com)

Alternatively, you might work with them to restructure the data (in a Python program) to conform to the templates that I provided. The required headers are:
• "typeName"
• "name"
• "qualifiedName"
• "classifications"

If you have that information in a list of dictionaries in Python, you could use Reader.parse_bulk_entities(list_of_dicts) and then take the results and call client.upload_entities(results) and bulk upload the entities inside that csv.

Hope this helps!
Kindly let us know if the above helps or you need further assistance on this issue.

---------------------------------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.