How can I run the following Python script in VS Code to store a file in Azure Blob Storage? I'm facing an issue where the file isn't stored when using an Azure Function

Question

How can I run the following Python script in VS Code to store a file in Azure Blob Storage? I'm facing an issue where the file isn't stored when using an Azure Function with http trigger / Blob trigger, give me the steps for it.

but the same task works locally through Spyder.
What could be the cause?

import requests
import csv
from azure.storage.blob import BlobServiceClient
# Your API key for News API (replace with your own key)
api_key = 'xyz'  # Replace with your API key
# Host key for Azure Function
host_key = 'xyx'
# Define the endpoint and parameters for News API
url = f'https://newsapi.org/v2/everything?code={host_key}'
parameters = {
    'q': 'technology',  # Search query
    'language': 'en',
    'sortBy': 'publishedAt',
    'apiKey': api_key
}
# Fetch the news data from the API
response = requests.get(url, params=parameters)
data = response.json()
# Define the file path for the CSV file
csv_file_path = 'news_data.csv'
# Save the data to a CSV file
with open(csv_file_path, 'w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    # Write the header
    writer.writerow(['Source', 'Author', 'Title', 'Description', 'URL', 'Published At'])
    # Write the data
    for article in data['articles']:
        writer.writerow([
            article['source']['name'], 
            article.get('author'), 
            article['title'], 
            article['description'], 
            article['url'], 
            article['publishedAt']
        ])
# Azure Storage account details
account_name = 'iii'  # Replace with your account name
container_name = 'ik'  # Replace with your container name
# Create a connection string
connection_string = f"xyx"
# Create a BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
# Get the container client
container_client = blob_service_client.get_container_client(container_name)
# Name of the blob
blob_name = 'news_data.csv'
# Create a blob client
blob_client = container_client.get_blob_client(blob_name)
# Upload the file to Azure Blob Storage
try:
    with open(csv_file_path, "rb") as data:
        blob_client.upload_blob(data, overwrite=True)
    print(f"File '{blob_name}' uploaded to {blob_client.url}.")
except Exception as e:
    print(f"An error occurred: {e}")

Thanks in advance for any assistance.

Answer 1

Here are the steps to run your script in your Azure Function :

1. Set Up the Development Environment in VS Code:
   • Ensure you have the Azure Functions extension installed in VS Code.
   • Install the Azure CLI if you haven't already, which you can use to manage Azure resources directly from the terminal.
   • Install Python on your local machine and make sure the Python extension for VS Code is installed.
2. Create an Azure Function App in VS Code:
   • Open VS Code and select "Azure" from the sidebar, then click on "Create New Project."
   • Select Python as the language, and choose an HTTP trigger or Blob trigger, depending on your requirement.
   • Follow the prompts to set up the function app. This will create a local Azure Function project in your workspace.
3. Modify the Function Code:
   • Replace the default function code in __init__.py with your script.
   • Update the function.json file if necessary, to match your function's triggers and bindings.
   • Ensure that any sensitive data such as API keys, host keys, or connection strings are not hardcoded in the script. Instead, use Azure Function's environment variables or Azure Key Vault.
   For example, you can set environment variables in the local.settings.json file:

   
      {
   
        "IsEncrypted": false,
   
        "Values": {
   
          "AzureWebJobsStorage": "UseDevelopmentStorage=true",
   
          "FUNCTIONS_WORKER_RUNTIME": "python",
   
          "API_KEY": "your_api_key_here",
   
          "HOST_KEY": "your_host_key_here",
   
          "CONNECTION_STRING": "your_connection_string_here"
   
        }
   
      }
   
   

   Then, access these variables in your script:

   
      import os
   
      api_key = os.getenv('API_KEY')
   
      host_key = os.getenv('HOST_KEY')
   
      connection_string = os.getenv('CONNECTION_STRING')
   
   

4. Test Locally:
   • Open the terminal in VS Code and run the Azure Function locally using the command:

     
          func start
     
     

   • This will start the function app locally, and you can trigger it via an HTTP request or the appropriate trigger depending on your setup.
   • Verify that the file is being uploaded to Azure Blob Storage. If you face issues locally, check the terminal for errors and troubleshoot as needed.
5. Deploy to Azure:
   • After testing locally, deploy the function to Azure using the Azure Functions extension in VS Code. Right-click on the function app in the Azure pane and select "Deploy to Function App."
   • Follow the prompts to deploy your function app to Azure.
6. Set Up Environment Variables in Azure:
   • After deployment, go to the Azure portal and navigate to your Function App.
   • Go to "Configuration" under "Settings" and add the necessary environment variables (API_KEY, HOST_KEY, CONNECTION_STRING...).

Answer 2

Hi @Vaibhav Patil Greetings! Thank you for posting this question here.

Your function app fails because the file system in Azure Function App is read-only, which means that you cannot write to the file system directly. The code errors out when you try to create and write a file open(csv_file_path, 'w' command.

Here is what you can do to write a file to Azure blob storage container without needing to create a local file. Declare a variable and copy the data from your end point similar to below

import azure.functions as func
import logging
import csv
from azure.storage.blob import BlobServiceClient

app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)

@app.route(route="<funcapp_name>")
def funcapp_name(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        #test data for CSV file
        #Define the data to be written to the CSV file
        data = {
    "status": "ok",
    "totalResults": 3,
    "articles": [
        {
            "source": {
                "id": "cnn",
                "name": "CNN"
            },
            "author": "John Doe",
            "title": "Article Title 1",
            "description": "Article Description 1",
            "url": "https://www.example.com/article1",
            "urlToImage": "https://www.example.com/article1/image.jpg",
            "publishedAt": "2022-01-01T00:00:00Z",
            "content": "Article content goes here."
        },
        {
            "source": {
                "id": "bbc-news",
                "name": "BBC News"
            },
            "title": "Article Title 2",
            "description": "Article Description 2",
            "url": "https://www.example.com/article2",
            "urlToImage": "https://www.example.com/article2/image.jpg",
            "publishedAt": "2022-01-02T00:00:00Z",
            "content": "Article content goes here."
        },
        {
            "source": {
                "id": "reuters",
                "name": "Reuters"
            },
            "author": "Jane Doe",
            "title": "Article Title 3",
            "description": "Article Description 3",
            "url": "https://www.example.com/article3",
            "urlToImage": "https://www.example.com/article3/image.jpg",
            "publishedAt": "2022-01-03T00:00:00Z",
            "content": "Article content goes here."
        }
    ]
}

        
        

        #with open('/home/site/wwwroot/data/data.txt', 'r') as file:
            # Read the contents of the file
            #content = file.read()
            # Print the contents
            #print(content)

        # Azure Storage account details
        account_name = 'iii'  # Replace with your account name
        container_name = 'appcontainer'  # Replace with your container name
        # Create a connection string
        connection_string = f"ConnectionString"
        # Create a BlobServiceClient
        blob_service_client = BlobServiceClient.from_connection_string(connection_string)
        # Get the container client
        container_client = blob_service_client.get_container_client(container_name)
        # Name of the blob
        blob_name = 'parsed_data.csv'
        # Create a blob client
        blob_client = container_client.get_blob_client(blob_name)
        #Process data for upload
        csv_data = ''
        for article in data["articles"]:
            csv_data += ','.join([str(article['source']['name']), 
            str(article.get('author')), 
            str(article['title']), 
            str(article['description']), 
            str(article['url']), 
            str(article['publishedAt'])]) + '\n'         
        
        # Upload the file to Azure Blob Storage
        blob_client.upload_blob(csv_data, overwrite=True)
        print(f"File '{blob_name}' uploaded to {blob_client.url}.")
        return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.")
    else:
        return func.HttpResponse(
             "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
             status_code=200
        )

You can then directly upload the above data to the Azure storage account using the following command blob_client.upload_blob(csv_data, overwrite=True) without using open command.

---

We noticed your feedback that the answer on this thread was not helpful. Thank you for taking time to share your feedback. Kindly let us know what we could have done better to improve the answer and make your experience better.

Since an alternate approach has been shared, I request that you would kindly re-take the survey for your experience on this thread.

However, if your issue remains unresolved, please let us know how we can assist. We are here to help you and strive to make your experience better and greatly value your feedback. Looking forward to your reply. Much appreciate your feedback!