How to delete batch transcription results automatically stored in Azure Storage?

Question

How to delete batch transcription results automatically stored in Azure Storage?

Ulhas Hulyal, Nilesh 35

For Batch transcription as suggested in document https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-transcription-create?pivots=rest-api, we are storing transcription results in an Azure container by providing "destinationContainerUrl" also used TimetoLive parameter so that transcription results will be automatically deleted.

but we noticed that transcription results are not getting removed from Azure container, we are providing value in TimetoLive as "timeToLive": "PT0H10M", so expecting transcription will be deleted post 10 min after the transcription job is created, but we noticed that transcription is still present in the container. Can anyone please guide if there are any different / additional setting needs to be done while creating transcription request.

Please refer to below transcription payload

{
    "displayName": "TranscriptionDemo1",
    "locale": "en-US",
    "properties": {
        "diarizationEnabled": true,
        "wordLevelTimestampsEnabled": true,
        "timeToLive": "PT0H10M",
        "destinationContainerUrl": "SAS URL"
    },
    "contentUrls": [
        "https://www.signalogic.com/melp/EngSamples/Orig/male.wav"
    ],
    "contentContainerUrl": "",
    "customProperties": {
        "diarizationV3Enabled": true,
        "diarizationMinSpeakers": 1,
        "diarizationMaxSpeakers": 10
    }
}

dupammi 8,615 Reputation points Microsoft External Staff

2024-01-04T13:28:50.04+00:00

Hi @Ulhas Hulyal, Nilesh ,

Thank you for using the Microsoft Q&A.

It's important to note that the "timeToLive" parameter is not a guarantee that the transcription results will be deleted exactly after the specified duration. The actual deletion time may vary depending on various factors such as the permissions (delete, write, permanent delete etc.) of the file, load on the Azure storage service, network latency, etc.

If you want to ensure that the transcription results are deleted immediately after the transcription job is completed, you can call the DeleteTranscription API to delete the transcription file. However, this approach requires additional coding effort and may not be suitable for all scenarios.

I hope this will be helpful for your scenario. Thank you!
dupammi 8,615 Reputation points Microsoft External Staff

2024-01-05T07:02:22.59+00:00

Hi @Ulhas Hulyal, Nilesh ,

Following up to see if the above (OR) below suggestions were helpful.

Thank you!
Ulhas Hulyal, Nilesh 35 Reputation points

2024-01-05T08:23:47.5333333+00:00

Hi @dupammi

As requested in below we provided delete permissions delete permission to SAS URL, but transcription was still available after specified duration provided in timeToLive parameters.

If there is no certainty of file getting deleted, we might try to trigger scheduler to remove transcription from storage container.
Else please advise if any alternative approach can be used which can delete transcription file automatically post transcription completion from container.

We are using DeleteTranscription API as well to remove file post transcription completion, but there might be possibility that script will fail due to some server issues at our end and DeleteTranscription API didn't execute as expected and file might remain in container forever.

dupammi 8,615 Microsoft External Staff

Hi @Ulhas Hulyal, Nilesh ,

Yes, it would be a better approach to trigger a scheduler to remove transcription from the storage container.

Another workaround would be to implement a custom script, that checks for a 'timeToLive' metadata attribute and calculates the time difference between the blob's creation time and last modified time. If more than 2 minutes have passed, it sleeps for the remaining time and then deletes the blob, ensuring that the blob is deleted only after 2 minutes have elapsed since the last modification. This approach can help ensure that transcription files are deleted.

Kindly refer below code snippet for clean-up.

Note: You may need to modify the approach to suit your specific requirements.

import time
from azure.storage.blob import BlobServiceClient
from datetime import datetime, timedelta

def cleanup_old_blobs():
    blob_service_client = BlobServiceClient.from_connection_string(storage_connection_string)
    container_client = blob_service_client.get_container_client(container_name)

    for blob in container_client.list_blobs():
        blob_client = container_client.get_blob_client(blob.name)
        blob_properties = blob_client.get_blob_properties()

        # Check if 'timeToLive' metadata is present
        if 'timeToLive' in blob_properties.metadata:
            upload_time = blob_properties.creation_time.replace(tzinfo=None)
            expiration_time = upload_time + timedelta(minutes=2)

            print("Upload Time:", upload_time)
            print("Expiration Time:", expiration_time)

            # Get last modified time from blob properties
            last_modified_time = blob_properties.last_modified
            last_modified_time = last_modified_time.replace(tzinfo=None)

            # Calculate the remaining time until expiration
            remaining_time = (expiration_time - last_modified_time).total_seconds()

            # Sleep for the remaining time
            time.sleep(max(0, remaining_time))

            blob_client.delete_blob()
            print(f'Deleted blob {blob.name} because more than 2 minutes have passed since upload.')

cleanup_old_blobs()

I hope this helps. Thank you!

1 answer

Your answer

dupammi 8,615 Reputation points Microsoft External Staff

2024-01-04T13:28:50.04+00:00

Hi @Ulhas Hulyal, Nilesh ,

Thank you for using the Microsoft Q&A.

It's important to note that the "timeToLive" parameter is not a guarantee that the transcription results will be deleted exactly after the specified duration. The actual deletion time may vary depending on various factors such as the permissions (delete, write, permanent delete etc.) of the file, load on the Azure storage service, network latency, etc.

If you want to ensure that the transcription results are deleted immediately after the transcription job is completed, you can call the DeleteTranscription API to delete the transcription file. However, this approach requires additional coding effort and may not be suitable for all scenarios.

I hope this will be helpful for your scenario. Thank you!
dupammi 8,615 Reputation points Microsoft External Staff

2024-01-05T07:02:22.59+00:00

Hi @Ulhas Hulyal, Nilesh ,

Following up to see if the above (OR) below suggestions were helpful.

Thank you!
Ulhas Hulyal, Nilesh 35 Reputation points

2024-01-05T08:23:47.5333333+00:00

Hi @dupammi

As requested in below we provided delete permissions delete permission to SAS URL, but transcription was still available after specified duration provided in timeToLive parameters.

If there is no certainty of file getting deleted, we might try to trigger scheduler to remove transcription from storage container.
Else please advise if any alternative approach can be used which can delete transcription file automatically post transcription completion from container.

We are using DeleteTranscription API as well to remove file post transcription completion, but there might be possibility that script will fail due to some server issues at our end and DeleteTranscription API didn't execute as expected and file might remain in container forever.

Answer 1

Sedat SALMAN 14,185 MVP

The timeToLive countdown may not start until the transcribing task achieves a terminal state (such as "Succeeded" or "Failed"). It is also important to make sure the SAS URL you gave has the required rights, including delete permissions.

or as an alternative according to the documentation

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-transcription-create?pivots=rest-api

you can call Transcriptions_Delete regularly after you retrieve the transcription results.

Share via

How to delete batch transcription results automatically stored in Azure Storage?

1 answer

Your answer