How to delete rows from a hybrid streaming dataset in Python?

Question

How to delete rows from a hybrid streaming dataset in Python?

Zephaniah, Sirivella 0

Is it possible to delete all data from a push streaming dataset (with Historical data enabled) at regular intervals (e.g., at the start of each day) using Python?

Saideep Anchuri 9,500 Reputation points Moderator

2025-05-16T10:41:48.98+00:00
Hi Zephaniah, Sirivella

To delete rows from a hybrid streaming dataset in Python, you can utilize the Delta Lake features in Databricks. However, it's important to note that structured streaming does not handle deletions directly. Instead, you can manage deletions by using specific strategies.

One approach is to periodically delete data based on a condition. For example, you can set up a scheduled job that deletes all rows from the dataset at the start of each day. You would typically achieve this by executing a delete command on the Delta table using the DeltaTable API.

how you might implement this in Python:

from delta.tables import * from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder.appName("DeleteRowsExample").getOrCreate() # Reference your Delta table deltaTable = DeltaTable.forName(spark, "your_database.your_table_name") # Delete all rows (you can modify the condition as needed) deltaTable.delete("true") # This will delete all rows in the table

Make sure to adjust the delete condition according to your requirements.Kindly refer below link: https://learn.microsoft.com/en-us/azure/databricks/delta/tutorial#delete-from-a-table

Thank You.
Zephaniah, Sirivella 0 Reputation points

2025-05-16T10:48:40.97+00:00

Hi, Saideep thanks for the answer. perhaps I was not clear in my question. I was looking specifically 'how to delete rows from PowerBI push streaming dataset'
Saideep Anchuri 9,500 Reputation points Moderator

2025-05-16T15:30:15.87+00:00
Hi Zephaniah, Sirivella

To delete rows from a Power BI push streaming dataset, you can use the Power BI REST API. Specifically, the DELETE request allows you to remove all rows from a specified table within the dataset.

Here are some steps:

Use the API Endpoint:

DELETE https://api.powerbi.com/v1.0/myorg/datasets/{datasetId}/tables/{tableName}/rows

2.You need the Dataset.ReadWrite.All permission to execute this API call.

3.This method only works for push datasets.

You cannot delete specific rows, only all rows in the table

Kindly refer below link: https://community.fabric.microsoft.com/t5/Service/How-can-we-delete-rows-in-power-bi-push-dataset-using-rest-api/m-p/339994

https://learn.microsoft.com/en-us/rest/api/power-bi/push-datasets/datasets-delete-rows

Thank You.

Your answer

Saideep Anchuri 9,500 Reputation points Moderator

2025-05-16T10:41:48.98+00:00

Hi Zephaniah, Sirivella

To delete rows from a hybrid streaming dataset in Python, you can utilize the Delta Lake features in Databricks. However, it's important to note that structured streaming does not handle deletions directly. Instead, you can manage deletions by using specific strategies.

One approach is to periodically delete data based on a condition. For example, you can set up a scheduled job that deletes all rows from the dataset at the start of each day. You would typically achieve this by executing a delete command on the Delta table using the DeltaTable API.

how you might implement this in Python:

from delta.tables import * from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder.appName("DeleteRowsExample").getOrCreate() # Reference your Delta table deltaTable = DeltaTable.forName(spark, "your_database.your_table_name") # Delete all rows (you can modify the condition as needed) deltaTable.delete("true") # This will delete all rows in the table

Make sure to adjust the delete condition according to your requirements.Kindly refer below link: https://learn.microsoft.com/en-us/azure/databricks/delta/tutorial#delete-from-a-table

Thank You.
Zephaniah, Sirivella 0 Reputation points

2025-05-16T10:48:40.97+00:00

Hi, Saideep thanks for the answer. perhaps I was not clear in my question. I was looking specifically 'how to delete rows from PowerBI push streaming dataset'
Saideep Anchuri 9,500 Reputation points Moderator

2025-05-16T15:30:15.87+00:00

Hi Zephaniah, Sirivella

To delete rows from a Power BI push streaming dataset, you can use the Power BI REST API. Specifically, the DELETE request allows you to remove all rows from a specified table within the dataset.

Here are some steps:

Use the API Endpoint:

DELETE https://api.powerbi.com/v1.0/myorg/datasets/{datasetId}/tables/{tableName}/rows

2.You need the Dataset.ReadWrite.All permission to execute this API call.

3.This method only works for push datasets.

You cannot delete specific rows, only all rows in the table

Kindly refer below link: https://community.fabric.microsoft.com/t5/Service/How-can-we-delete-rows-in-power-bi-push-dataset-using-rest-api/m-p/339994

https://learn.microsoft.com/en-us/rest/api/power-bi/push-datasets/datasets-delete-rows

Thank You.

Share via

How to delete rows from a hybrid streaming dataset in Python?

Your answer