Container Instance private IP changed unexpectedly in VNet

Question

Issue type: Technical
Resource Group: rg-congen-waf-dm1
Service: Container Instances (Compute)
Problem type: Networking, IP, and DNS issues
Problem subtype: Container group is unable to communicate with another resource in the same VNet

---

Summary

We have a customer-impacting outage on a demo environment caused by what we believe is a platform-initiated restart of a VNet-delegated Azure Container Instance that resulted in a private IP reassignment within the delegated subnet.

Symptom

• App Service (frontend) proxies /api/* to BACKEND_URL = http://10.0.4.4:8000 (the ACI's expected private IP at deploy time).
• All /api/* requests started returning HTTP 502 after ~2 minutes (consistent with Linux TCP SYN retry timeout) on or around <YYYY-MM-DD HH:MM UTC>.
• /.auth/me (App Service EasyAuth, no backend hop) remained 200 OK throughout.
• No customer-initiated restart, redeployment, scale, or config change was performed on the ACI in the preceding 7 days.

What We Need from Microsoft

1. Confirmation from platform telemetry of any restart, host migration, node replacement, planned/unplanned maintenance, or container group recreation events on this container group between the deploy date and the bug report date.
2. Exact timestamps of any such event, and the IP held by the container group before and after each event.
3. IP conflict check — confirmation of whether the IP 10.0.4.4 was held by any other resource in the subnet during that window (i.e., whether a transient conflict forced the new container group instance to allocate a different IP).

Answer 1

Azure Container Instances (ACI) container groups can be restarted by the platform for maintenance or other reasons, and when this happens the container group’s IP address can change. This behavior is documented and expected for ACI, especially when running in a virtual network.

From the available information:

• Container groups may restart due to platform maintenance events, which are performed to improve the underlying infrastructure. These restarts can occur without explicit user action.
• When a container group restarts, its IP address (private or public) is not guaranteed to remain the same. The documentation explicitly advises against relying on a hard-coded IP address for ACI container groups.
• For scenarios that require a stable endpoint, the recommended approaches are:
  • Use an Application Gateway in front of the container group to provide a static frontend IP and route traffic to the container group’s current private IP.
  • Use a NAT gateway for outbound connectivity when the container group is deployed into a virtual network, as this is the only supported configuration for outbound traffic from ACI in a VNet.

Regarding the specific requests:

1. Platform telemetry confirmation of restart/maintenance events
   Access to platform telemetry (host migration, node replacement, planned/unplanned maintenance, container group recreation) is not exposed through the public documentation or APIs referenced here. For confirmation of such events with exact timestamps, a support request must be opened so that the Azure support team can review internal telemetry for the specific container group and time window.
2. Exact timestamps and IP before/after events
   Exact event timestamps and the before/after IP assignments for a specific container group are not available in the provided documentation. These details can only be obtained by Azure support through internal logs. The documented guidance is that IP changes are expected when a container group is stopped, started, or restarted.
3. IP conflict check for 10.0.4.4 in the subnet
   There is no public mechanism in the referenced documentation to query historical IP ownership within a VNet/subnet or to confirm whether a specific private IP was temporarily held by another resource. An IP conflict investigation for a specific address and time window would also require an Azure support case so that internal networking logs can be examined.

Mitigation and design guidance based on the documentation:

• Do not hard-code the container group’s IP address in upstream services (such as App Service configuration). Instead, place the container group behind an Application Gateway or similar component that provides a stable endpoint and can be updated if the backend IP changes.
• Be aware that container groups may restart due to platform maintenance and that their IP addresses may change as a result.
• If the container group is in a VNet and requires outbound connectivity, ensure a NAT gateway is configured, as this is the only supported configuration for outbound traffic from ACI in a VNet.
• If the container group stops working or becomes unreachable, follow the ACI troubleshooting guidance first, and then open a support request if the issue appears to be platform-related (for example, suspected platform-initiated restarts causing IP changes).

For this specific incident (confirmation of restarts, timestamps, and IP conflict analysis), the next step is to open an Azure support request for the affected subscription and container group, referencing the time window and symptoms described, so that support can pull the internal telemetry and provide definitive answers.

---

References:

• Best practices and considerations for Azure Container Instances
• What is Azure Container Instances?
• Expose a static IP address for a container group
• Troubleshoot common issues in Azure Container Instances
• Azure container group intermittently restarts due to killing event interruptions

AI-generated content may be incorrect. Read our transparency notes for more information.