Azure Key Vault secret access intermittently fails from a Docker containerized Python application

ThierryL 141 Reputation points
2022-11-01T06:33:14.76+00:00

Hello,

I have deployed a Python application in a Docker container in which the script fetches a secret stored in Azure Key Vault in order to connect to a database.
The Azure access tokens are set as environment variables in the Dockerfile, and I have both libraries 'azure-identity' and 'azure-keyvault-secrets' imported in the script.
However, the script execution intermittently hangs at client.get_secret("DBPassword").value and throws an exception after a few tries.

While I have never experienced this kind of problem in my local enviromnent, the application intermittently fails to retrieve the secret once it runs in a container on the server (I am not entirely certain that it will never fail in my local environment though).
The problem seems to occur and disappear every few hours : once it stops working it won't work even if I relaunch the container or rebuild the image. And then after some time it starts working again.
I can't figure out whether this is a network problem on our side or something else.
I tried to execute the containerized application with or without the host server being logged in to Azure portal, but that doesn't seem to change anything.

If you have experienced the same problem, thank you for pointing me in the right direction.

Below is the application code (some parts are redacted) :

Python script : 

from azure.keyvault.secrets import SecretClient  
from azure.identity import DefaultAzureCredential  

# Retrieve password from Azure Key Vault  
KVUri = "https://xxxx.vault.azure.net"  
credential = DefaultAzureCredential()  
client = SecretClient(vault_url=KVUri, credential=credential)  
password = client.get_secret("DBPassword").value

Dockerfile : 

FROM python:3.8-slim  
  
RUN apt-get update && apt-get install --no-install-recommends -y libaio1 wget unzip libgomp1 && apt-get clean  
  
# Install Oracle instant client  
WORKDIR /opt/oracle  
RUN wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-basiclite-linuxx64.zip && \  
    unzip instantclient-basiclite-linuxx64.zip && rm -f instantclient-basiclite-linuxx64.zip && \  
    cd /opt/oracle/instantclient* && rm -f *jdbc* *occi* *mysql* *README *jar uidrvci genezi adrci && \  
    echo /opt/oracle/instantclient* > /etc/ld.so.conf.d/oracle-instantclient.conf && ldconfig  
RUN python -m pip install cx_Oracle  
  
# Set environment variables  
ENV TZ="Asia/Seoul"  
  
# Install dependencies  
COPY ./requirements.txt /app/requirements.txt  
RUN pip install --no-cache -r /app/requirements.txt  
  
# Copy project files  
COPY ./app/ /app/  
  
# Azure credentials for connection to Azure Vault  
ENV AZURE_TENANT_ID="45f64d4d-91b6-4285-8c07-[xxx]"  
ENV AZURE_CLIENT_ID="c7c24351-5cba-4889-ac94-[xxx]"  
ENV AZURE_CLIENT_SECRET="[xxx]"  
  
# Set working directory  
WORKDIR /app  

ENTRYPOINT python main.py

requirements.txt 

numpy==1.21.6  
pandas==1.3.5  
fastapi==0.85.0  
uvicorn==0.18.3  
scikit-learn==1.0.2  
sqlalchemy==1.4.32  
lightgbm==3.2.1  
xgboost==1.5.0  
azure-identity  
azure-keyvault-secrets

Logs when the execution works : 

2022-11-01 13:47:29.220 Environment is configured for ClientSecretCredential  
2022-11-01 13:47:29.221 ManagedIdentityCredential will use IMDS  
2022-11-01 13:47:29.232 Request URL: 'https://[xxxxx].vault.azure.net/secrets/DBPassword/?api-version=REDACTED'  
Request method: 'GET'  
Request headers:  
    'Accept': 'application/json'  
    'x-ms-client-request-id': '4a2ae7f6-59a0-11ed-b72b-[xxxxx]'  
    'User-Agent': 'azsdk-python-keyvault-secrets/4.6.0 Python/3.8.12 (Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.2.5)'  
No body was attached to the request  
2022-11-01 13:47:29.381 Response status: 401  
Response headers:  
    'Cache-Control': 'no-cache'  
    'Pragma': 'no-cache'  
    'Content-Length': '97'  
    'Content-Type': 'application/json; charset=utf-8'  
    'Expires': '-1'  
    'WWW-Authenticate': 'Bearer authorization="https://login.windows.net/45f64d4d-91b6-4285-8c07-[xxxxx]", resource="https://vault.azure.net"'  
    'x-ms-keyvault-region': 'koreacentral'  
    'x-ms-client-request-id': '4a2ae7f6-59a0-11ed-b72b-[xxxxx]'  
    'x-ms-request-id': 'b7b9ee5e-fb25-409a-a69c-[xxxxx]'  
    'x-ms-keyvault-service-version': '1.9.576.1'  
    'x-ms-keyvault-network-info': 'conn_type=Ipv4;addr=115.88.43.205;act_addr_fam=InterNetwork;'  
    'X-Content-Type-Options': 'REDACTED'  
    'Strict-Transport-Security': 'REDACTED'  
    'Date': 'Tue, 01 Nov 2022 04:47:29 GMT'

Logs when the execution fails : 

2022-11-01 11:22:04.238 Environment is configured for ClientSecretCredential  
2022-11-01 11:22:04.239 ManagedIdentityCredential will use IMDS  
2022-11-01 11:22:04.246 Request URL: 'https://[xxxxx].vault.azure.net/secrets/DBPassword/?api-version=REDACTED'  
Request method: 'GET'  
Request headers:  
    'Accept': 'application/json'  
    'x-ms-client-request-id': 'f9abb83c-598b-11ed-86ca-[xxxxx]'  
    'User-Agent': 'azsdk-python-keyvault-secrets/4.6.0 Python/3.8.12 (Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.2.5)'  
No body was attached to the request  
2022-11-01 11:22:37.181 Request URL: 'https://[xxxxx].vault.azure.net/secrets/DBPassword/?api-version=REDACTED'  
Request method: 'GET'  
Request headers:  
    'Accept': 'application/json'  
    'x-ms-client-request-id': 'f9abb83c-598b-11ed-86ca-[xxxxx]'  
    'User-Agent': 'azsdk-python-keyvault-secrets/4.6.0 Python/3.8.12 (Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.2.5)'  
No body was attached to the request  

[...]  

Traceback (most recent call last):  
  File "/usr/local/lib/python3.8/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 562, in _run_script  
    exec(code, module.__dict__)  
  File "/app/app.py", line 4, in <module>  
    import db  
  File "/app/db/__init__.py", line 1, in <module>  
    import db.auth_rcx  
  File "/app/db/auth_rcx.py", line 8, in <module>  
    password = client.get_secret("DBPassword").value  
  File "/usr/local/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer  
    return func(*args, **kwargs)  
  File "/usr/local/lib/python3.8/site-packages/azure/keyvault/secrets/_client.py", line 72, in get_secret  
    bundle = self._client.get_secret(  
  File "/usr/local/lib/python3.8/site-packages/azure/keyvault/secrets/_generated/_operations_mixin.py", line 1574, in get_secret  
    return mixin_instance.get_secret(vault_base_url, secret_name, secret_version, **kwargs)  
  File "/usr/local/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer  
    return func(*args, **kwargs)  
  File "/usr/local/lib/python3.8/site-packages/azure/keyvault/secrets/_generated/v7_3/operations/_key_vault_client_operations.py", line 694, in get_secret  
    pipeline_response = self._client._pipeline.run(  # pylint: disable=protected-access  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 211, in run  
    return first_node.send(pipeline_request)  # type: ignore  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send  
    response = self.next.send(request)  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send  
    response = self.next.send(request)  
  Stopping...  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send  
    response = self.next.send(request)  
  [Previous line repeated 2 more times]  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.py", line 158, in send  
    response = self.next.send(request)  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py", line 468, in send  
    raise err  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py", line 446, in send  
    response = self.next.send(request)  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py", line 118, in send  
    response = self.next.send(request)  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send  
    response = self.next.send(request)  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send  
    response = self.next.send(request)  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 71, in send  
    response = self.next.send(request)  
  [Previous line repeated 1 more time]  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 103, in send  
    self._sender.send(request.http_request, **request.context.options),  
  File "/usr/local/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 361, in send  
    raise error  
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7f39fc0b2880>: Failed to establish a new connection: [Errno 110] Connection timed out
Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,113 questions
Accepted answer
  1. JamesTran-MSFT 36,366 Reputation points Microsoft Employee
    2022-11-02T22:05:56.85+00:00

    @ThierryL-3166
    Thank you for the quick follow up on this!

    From the Key Vault side of things after reviewing the documentation and your logs when the execution fails, it looks like you're running into the below error message. This could be related to a connection issue, or as you pointed out in your workaround - the need to sign-in. The user credentials (or access token lifetime) could be expiring causing the connection to time out.

    Error Message when execution fails:
    azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7f39fc0b2880>: Failed to establish a new connection: [Errno 110] Connection timed out

    As mentioned in the Authenticate and create a client section of the doc you shared - "the logged in user is used to authenticate to key vault, which is the preferred method for local development". When your application is deployed to Azure, can you see if using a managed identity helps resolve your connection time out issue?

    I hope this helps!

    If you're still running into issues, can you share any error messages that you receive?
    Thank you for your time and patience throughout this issue.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest