Distinct from blob storage

Question

Distinct from blob storage

Vineet S 1,390

Hi, how to read distinct data from blob storage without using data flow ie table1---- company code(columnname) p1, p2, p1,p5, p2.. Pls share screenshot

phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-08-12T17:54:15.8366667+00:00
@Vineet S

Thanks for using MS Q&A platform and posting your query.

To read distinct data from blob storage without using data flow, you can use Azure Databricks or Azure Synapse Analytics. Here's an example using Azure Databricks:

First, you need to read the data from the blob storage into a DataFrame:

from pyspark.sql.functions import col # Read data from blob storage df = spark.read.format("csv").option("header", "true").load("abfss://<container_name>@<storage_account_name>.blob.core.windows.net/<blob_name>")

Then, you can use the distinct function to get the distinct values of the company code column:

# Get distinct values of company code column distinct_df = df.select("company code").distinct()

Finally, you can collect the distinct values into a list:

# Collect distinct values into a list distinct_values = distinct_df.collect()

from pyspark.sql.functions import col # Read data from blob storage df = spark.read.format("csv").option("header", "true").load("abfss://<container_name>@<storage_account_name>.blob.core.windows.net/<blob_name>") # Get distinct values of company code column distinct_df = df.select("company code").distinct() # Collect distinct values into a list distinct_values = distinct_df.collect()

Note: You need to replace <container_name>, <storage_account_name>, and <blob_name> with your actual blob storage container name, storage account name, and blob name.

Hope this helps. Do let us know if you any further queries.
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-08-13T12:24:31.33+00:00

@Vineet S We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Vineet S 1,390 Reputation points

2024-08-13T12:46:13.2866667+00:00

This ticket is for adf
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-08-14T14:48:06.48+00:00

@Vineet S Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 answer

Your answer

phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-08-12T17:54:15.8366667+00:00

@Vineet S

Thanks for using MS Q&A platform and posting your query.

To read distinct data from blob storage without using data flow, you can use Azure Databricks or Azure Synapse Analytics. Here's an example using Azure Databricks:

First, you need to read the data from the blob storage into a DataFrame:

from pyspark.sql.functions import col # Read data from blob storage df = spark.read.format("csv").option("header", "true").load("abfss://<container_name>@<storage_account_name>.blob.core.windows.net/<blob_name>")

Then, you can use the distinct function to get the distinct values of the company code column:

# Get distinct values of company code column distinct_df = df.select("company code").distinct()

Finally, you can collect the distinct values into a list:

# Collect distinct values into a list distinct_values = distinct_df.collect()

from pyspark.sql.functions import col # Read data from blob storage df = spark.read.format("csv").option("header", "true").load("abfss://<container_name>@<storage_account_name>.blob.core.windows.net/<blob_name>") # Get distinct values of company code column distinct_df = df.select("company code").distinct() # Collect distinct values into a list distinct_values = distinct_df.collect()

Note: You need to replace <container_name>, <storage_account_name>, and <blob_name> with your actual blob storage container name, storage account name, and blob name.

Hope this helps. Do let us know if you any further queries.
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-08-13T12:24:31.33+00:00

@Vineet S We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Vineet S 1,390 Reputation points

2024-08-13T12:46:13.2866667+00:00

This ticket is for adf
phemanth 15,765 Reputation points Microsoft External Staff Moderator

2024-08-14T14:48:06.48+00:00

@Vineet S Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Hello @Vineet S

To read distinct data from blob storage without using data flow, you can use Azure Stream Analytics. Here's an example query that reads data from a CSV file in blob storage and selects distinct values from the "company code" column:

SELECT DISTINCT [company code] INTO output FROM input

Here's a screenshot of how you can create a Stream Analytics job and configure it to read from blob storage: Stream Analytics job configuration

And here's a screenshot of how you can create an input for the job and specify the path to the CSV file in blob storage: Stream Analytics input configuration

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.

Share via

Distinct from blob storage

1 answer

Your answer