PowerShell script at azure storage account getting failed

Question

Hello everyone,

I've developed a PowerShell script that copies appended blocks into a block blob in a new container. It worked well in my POC environment and even ran successfully from a local PowerShell client in production.

However, when I tried it on a production storage account with thousands of append blobs, the script took hours to run on my local system and sometimes got stuck due to internet issues or system hang etc . To resolve this, I decided to run the script via an Azure Automation Account runbook, as I have done with other tasks before.

This time, the runbook keeps failing with unknown errors—such as a variable value not being found. Can anyone help me troubleshoot this issue and suggest another method, perhaps using Azure Data Factory or an App Service? Please note : all permission level , storage account network , RBAC etc is correct . its is working when container has few blobs . but not when Blob count is high.

Answer 1

@Tabish Ahmed Shaikh upgrading to Az.Storage 8.2 was a smart move, and exploring Hybrid Runbook Worker should help bypass those execution limits.

For Configuring Azure Data Factory (ADF) for Large Blob Transfers

To ensure your copy operation is robust and trackable, you can follow this structured approach in ADF:

1️⃣ Set Up Your Pipeline

• Create an ADF pipeline with a Copy Data activity.
• Define the source as Azure Blob Storage and the destination as your target container.

2️⃣ Optimize for Performance

• Enable Parallel Copy: In the Copy Data activity settings, set Enable Parallelism to handle large volumes efficiently.
• Use Blob Chunking: Configure chunk sizes to split large blob transfers into manageable parts.

3️⃣ Implement Failure Recovery & Continuation

• Retry Policy: Set a retry limit for transient failures.
• Fault Tolerance: Enable skip incompatible files so your process doesn’t halt due to one failed file.
• Incremental Copy: Use LastModifiedDate filters to copy only new blobs, reducing unnecessary processing.

4️⃣ Track Progress and Monitor Execution

• Enable Logging: Store logs in an Azure Blob container or SQL DB to track processed blobs.
• Incremental Copy: Use LastModifiedDate filters to copy only new blobs, reducing unnecessary processing.

4️⃣ Track Progress and Monitor Execution

• Enable Logging: Store logs in an Azure Blob container or SQL DB to track processed blobs.
• Enable Logging: Store logs in an Azure Blob container or SQL DB to track processed blobs.
• Use Lookup Activity: Periodically fetch a list of remaining blobs and compare against completed ones.
• Azure Monitor Alerts: Set up alerts to notify when failures occur or transfers slow down.

5️⃣ Automate and Scale

• Trigger Execution: Schedule the pipeline to run incrementally, avoiding long execution times.
• Use Data Flow for Custom Logic: If needed, implement Data Flow transformations for more control over blob processing.

Next Steps

• Try setting up an ADF Copy Data pipeline with parallel processing.
• Monitor progress using logs and alerts.
• If execution limits persist, consider breaking transfers into multiple runs.

I hope these helps. Let me know if you have any further questions or need additional assistance.

Also if these answers your query, do click the "Upvote" and click "Accept the answer" of which might be beneficial to other community members reading this thread.

Answer 2

Hi @Tabish Ahmed Shaikh welcome to the Microsoft Q&A.

It sounds like you're facing challenges with scaling your PowerShell script in Azure Automation for a large number of append blobs. Here are some troubleshooting steps and alternative approaches:

Troubleshooting Azure Automation Runbook

1. Variable Scope: Ensure that all variables used in the script are properly initialized and scoped. Sometimes, variables might not persist across runbook executions due to scope issues.
2. Runbook Logging: Enable detailed logging in your runbook to identify where the script is failing. You can use Write-Output or Write-Verbose to capture more information.
3. Timeouts: Azure Automation has limits on execution time. If your script is long-running, consider breaking it into smaller tasks or using a Hybrid Runbook Worker for better performance.
4. Modules and Dependencies: Verify that all required PowerShell modules are imported into your Automation Account. Missing modules can cause unexpected errors.
5. Error Streams: Check the error streams in the Automation Account to pinpoint specific issues.

Alternative Approaches

Azure Data Factory: This is a great option for handling large-scale data movement. You can create a pipeline to copy blobs from one container to another efficiently. Data Factory is optimized for such tasks and can handle large datasets without the limitations of a runbook.

Azure Functions: If you prefer a serverless approach, you can write an Azure Function to process the blobs. Functions can scale automatically and handle high blob counts effectively.

App Service: Deploy your script as a web job in an App Service. This provides more control over execution and scaling compared to Azure Automation.

Given the scale of your task, Azure Data Factory might be the most efficient solution.

See: https://learn.microsoft.com/en-us/azure/automation/troubleshoot/runbooks

I hope these helps. Let me know if you have any further questions or need additional assistance.

Also if these answers your query, do click the "Upvote" and click "Accept the answer" of which might be beneficial to other community members reading this thread.

Answer 3

It looks like your script doesn't perform direct, server-side copy between the two containers - but rather copies files first to the local compute resource and then back to the container.

Try the following instead (adjust accordingly):

# Define variables
$sourceAccount = "<your-storage-account-name>"
$containerSrc = "<source-container-name>"
$containerDest = "<destination-container-name>"
$blobPrefix = "<optional-prefix-or-blank>"
$sasToken = "<your-sas-token>"  # Include leading "?" if needed
# Construct the source and destination URLs
$srcUrl = "https://$sourceAccount.blob.core.windows.net/$containerSrc/$blobPrefix$sasToken"
$destUrl = "https://$sourceAccount.blob.core.windows.net/$containerDest$sasToken"
# Perform the server-side copy
azcopy copy `
    $srcUrl `
    $destUrl `
    --recursive=true

---

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin