Troubleshoot Azure Sphere problems

Here are some troubleshooting steps for problems that may occur during installation and setup.

Device communication problems

A failure to connect to the device from a host machine can occur for many reasons and may trigger any of several error messages, depending on which tools or applications encounter it. The following error messages may indicate a failed connection:

• An error occurred. Please check your device is connected and your PC has been configured correctly, then retry.
• Could not connect to the device. Check if your device is connected to the PC. The device may be unresponsive if it is applying an Azure Sphere operating system update; wait a few minutes and then retry. If this issue persists, try uninstalling and reinstalling the Azure Sphere SDK.
• An unexpected issue occurred. Please try again; if the issue persists, please refer to aka.ms/azurespheresupport for troubleshooting suggestions and support.
• Failed to retrieve device ID from attached device: 'Could not connect to the device; please ensure it is attached.'
• Failed to establish communication with device after recovery.

There are three components that may cause device communication issues: (1) the command line interface itself; (2) the device communication service (DCS) that facilitates communication with a device; and (3) the device.

To isolate the cause, perform the following diagnostic steps:

1. Verify you can communicate directly with the device. Navigate to https://192.168.35.2/status in a web browser. Ignore and dismiss any presented SSL warnings. You should see output similar to: {"uptime":56} in the web browser window.

Note

You can also perform this step from the command line using curl https://192.168.35.2/status --insecure.

Note

You can also perform this step from PowerShell (v6 or greater) Invoke-WebRequest https://192.168.35.2/status -SkipCertificateCheck.

2. Confirm the device communication service has detected a device. Navigate to http://localhost:48938/api/service/devices in a web browser and confirm at least one device is shown in JSON output.

Warning

The Linux DCS is only included in SDK release 23.05 and above. Follow the legacy troubleshooting steps for SDK versions 22.11 and earlier.

Note

You can also perform this step from PowerShell via Invoke-WebRequest http://localhost:48938/api/service/devices.

Could not communicate directly with device and device is not detected by the DCS

If a connection could not be established with the device (Step 1) and the device cannot be detected by the DCS (Step 2), then there is likely a problem with the physical device configuration.

Firstly, check the device has been detected by the machine.

From a terminal:

1. Run sudo lsusb. Verify that Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC appears in the output.

Using a PowerShell window:

1. Run Get-PnpDevice -PresentOnly | Where-Object { $_.InstanceId -match '^USB' }. Verify that MSFT MT3620 Std Interface appears in the output.

If the prior command does not return the expected output, then connect the device to a different USB port. If this still does not resolve your issue, check the device has its jumpers configured correctly. For example, Seeed MT3620 Development Kits are shipped from the factory with a jumper header across pins 2 and 3 of J3, which powers the clock from the main power supply. Check that the header has not been dislodged or removed.

Could not communicate with device directly but device is detected by the DCS

If a connection could not be established with the device (Step 1), but the device has been detected by the DCS (Step 2), then it is likely that the device or device communication service is in a bad state.

If you have just connected a device to the Internet, and the device's operating system is out of date, the device may be applying an OS update. There is also a small chance the device is taking an OS update if the time coincides a release window. During an operating system update, you will not be able to communicate with a device.

If you suspect the device is up to date and/or it is not connected to the Internet, then the next best option is to unplug and reconnect the device to your PC. This should cause the DCS to renew its connection to the device and the device to reset to an initial state.

If you cannot unplug and reconnect device, the next best option is to restart the DCS. From a terminal:

1. Run sudo azspheredctl restart. Verify the command completes successfully.
   1. If the command fails, run sudo systemctl restart azsphered.

If you cannot unplug and reconnect device, the next best option is to issue a rescan command for the DCS. Upon receiving this command, the DCS will cycle and reset the FTDI interface chip used to communicate with the MT3620. Using a PowerShell window:

1. Run the command az sphere device rescan-attached

Now re-run the command you initially tried. If the issue is still not resolved, the next best option is to try restarting the DCS. From an administrator level Powershell window:

1. Run Restart-Service -Name AzureSphereDeviceCommunicationService. Verify the command completes successfully.

Alternately, restart the DCS using a graphical user interface.

1. Press Start and enter Services. Right-click the Azure Sphere Device Communication Service and select Restart.

Now re-run the command you initially tried.

If the command continues to fail, or az sphere device list-attached has output similar to:

1 device attached:
--> Device ID: Cannot get device ID
  --> Is responsive: no
  --> IP address: 192.168.35.2
  --> Connection path: 213

Run az sphere device recover

Can communicate with device directly and device is detected by the DCS

This likely indicates a problem with the CLI. Please ensure you have the latest version of the SDK.

If you have the latest version of the SDK and your issue is still not resolved, post a question to the Q&A forum.

Dropped connections or missing devices

If your computer can connect to the Azure Sphere device, but often drops the connection, you may have a conflict in the IP subnet.

Azure Sphere uses subnet 192.168.35.*. If you have other software that uses the same subnet, either disable that software or limit the range of IP addresses that it uses. Currently, you cannot change the range of IP addresses that Azure Sphere uses.

Cannot apply device capabilities

If you receive errors when you try to apply a device capability to an Azure Sphere device, the problem might be that the OS on your device is out of date. This problem can occur if the device has been offline for an extended period, during which Microsoft updated its internal keys. The following errors are symptoms of this problem:

The az sphere device enable-development or az sphere device enable-cloud-test commands return an error similar to the following:

error: The device did not accept the device capability configuration. Please check the Azure Sphere OS on your device is up-to-date using 'az sphere device show-deployment-status'.

The az sphere device capability update command returns an error similar to the following:

error: Could not apply device capability configuration to device.

This is due to the trusted keystore on the device being out of date. To solve this problem follow the instructions here.

Failure to create four USB serial converters

After you set up an MT3620 development board, you should see four USB serial converters in Device Manager. If you see fewer than four, you may have a problem with the FTDI driver.

Note

If this board has previously been used for RTApp development, you may see three converters instead of four. This is normal and does not represent an error.

If the FTDI driver is not correctly installed, the converters might appear in the wrong location, such as Other devices, or might not appear at all.

To resolve this problem:

1. To open Device Manager, click Start and enter Device Manager.

2. Under Universal Serial Bus controllers, select USB Serial Converter A. Right-click the name, select Uninstall Device, and delete the driver if given the option:

   

   Repeat this step for USB Serial Converter B through USB Serial Converter D.

3. Unplug your development board from your PC and plug it back in again. "MSFT MT3620 Std Interface" should appear with a triangle warning icon, which indicates no driver is available.

4. Right-click one of the MSFT MT3620 Std Interface devices and select Update driver. Choose Search automatically for updated driver software. Updating one should fix them all. You should now see four USB serial converters in the Universal Serial Bus controllers section. If all four converters do not appear, repeat this step for each converter.

Failure to install FTDI drivers

The FTDI drivers should be downloaded and installed automatically by Windows when your Azure Sphere device is first plugged in to your PC. If the drivers are installed correctly, you will see four USB Serial Converters listed under Universal Serial Bus controllers in Device Manager, as described in Set up your dev kit.

Windows 10, version 2004, does not search for the drivers. In this case, the drivers are not downloaded and installed automatically and you will see the following items listed in Device Manager:

To install the drivers, manually download the drivers from Future Technology Devices International (FTDI). Choose the driver that matches your Windows installation (32- or 64-bit).

For availability dates and build numbers of Windows 10 versions, see Windows 10 release information. This information can help you determine whether your version of Windows 10 is earlier or later than version 2004.

Connection problems caused by TAP-Windows adapter configuration

Azure Sphere tools communicate with attached development boards by using an IP network over USB. This requires the TAP-Windows adapter from OpenVPN Technologies. The Azure Sphere SDK installation procedure installs this adapter on your PC if it is not already present.

Two distinct types of problems have been reported with the TAP-Windows adapter:

• If a different version of the TAP-Windows adapter is already installed, or if the Azure Sphere device is not connected to the first instance of the TAP-Windows adapter, the Azure Sphere tools may fail to connect to your device.

• If you are using the Cisco AnyConnect virtual private network (VPN), your TAP-Windows adapter may appear unplugged and device communication may not work. This is because the VPN can redirect traffic away from the TAP adapter's IP address.

Ensure Azure Sphere device is connected to the appropriate TAP-Windows adapter

To determine whether the problem is related to the TAP adapter, first find out how many TAP adapters are installed on your PC, and then modify the installation if necessary.

To determine how many TAP adapters are installed on your PC:

1. Open Windows Settings and select the Network & Internet group.

2. Under Advanced network settings, select Change adapter options. You should see only one TAP adapter, as shown in the following screenshot:

   

   If you see more than one TAP adapter, or if you see only one TAP adapter but its name is not Azure Sphere, follow these steps to uninstall all TAP adapters and reinstall the SDK. If you see no TAP adapters, reinstall the SDK.

To uninstall the TAP adapters:

1. Click Start and enter Device Manager.

2. In Device Manager, open Network Adapters and select TAP-Windows adapter:

   

3. Right-click TAP-Windows adapter and select Uninstall Device. In the dialog box, select Delete the driver software for this device, then click Uninstall.

4. Open a command prompt as Administrator and run the following Powershell installer script:

    powershell -ExecutionPolicy RemoteSigned -File "%ProgramData%\Microsoft\Azure Sphere\TapDriverInstaller\TapDriverInstaller.ps1" Install
   

5. If the installation succeeds, restart the Azure Sphere Device Communication Service:

   net stop AzureSphereDeviceCommunicationService

   net start AzureSphereDeviceCommunicationService

6. Reinstall the Azure Sphere SDK.

Correct the unplugged state when using Cisco AnyConnect VPN client

When you are using Cisco AnyConnect VPN client, the TAP-Windows adapter may appear unplugged and device communication may not be working. You can correct the problem as follows:

1. Make sure that you have administrator privileges on your computer.

2. Open Windows Settings and select the Network & Internet group.

3. Under Advanced network settings, select Change adapter options.

4. Right-click Azure Sphere TAP-Windows Adapter V9 and select Properties.

5. On the Networking tab, locate the Cisco AnyConnect Network Access Manager Filter Driver entry and clear the item.

   

6. Select OK to save the setting and exit the properties.

7. Check that communication with your Azure Sphere device is now working by opening a command prompt and entering the az sphere device show-attached command.

Device does not respond

One or more of the following errors from an azsphere command may indicate that the Azure Sphere Device Communication Service failed to start:

• warn: Device is not responding. Could not perform version check.​
• Device is not responding. Cannot get device ID.​
• error: Could not connect to the Azure Sphere Device Communication Service. If this issue persists, try uninstalling and reinstalling the Azure Sphere SDK.​
• error: The device is not responding. The device may be unresponsive if it is applying an Azure Sphere operating system update; please retry in a few minutes.

Ensure you have performed the device communication troubleshooting steps.

If your issue is still not resolved and your machine has recently taken a Windows update, the device communication service may fail to start after Windows update and in cases where one of the internal JSON settings files or config file has become corrupted.

Failure after Windows Update

These errors can occur after you've updated Windows on your PC. Sometimes Windows Update uninstalls the FTDI drivers required for the communication service.

To resolve the problem:

1. Unplug the Azure Sphere device from USB and plug it in again. Upon replugging the device, the correct drivers should reinstall.
2. If unplugging and replugging the device fails to fix the problem, uninstall and reinstall the Azure Sphere SDK.

JSON file

If you have not recently updated Windows, the cause of the error might be the restore.json file that is used for the service.

To resolve this problem:

1. Save a copy of the following file:

   c:\windows\serviceprofiles\localservice\appdata\local\Azure Sphere Tools\restore.json

2. Delete the file from its original location.

3. Stop and then restart the Azure Sphere Device Communication Service:

   net stop AzureSphereDeviceCommunicationService

   net start AzureSphereDeviceCommunicationService

Corrupted config file

If an error is reported when you try to run a command, the corrupted config file might be preventing your device from running correctly.

To resolve this problem, delete the corrupted config file located in .azsphere\config on Windows or ~/.azsphere/config on Linux.

Windows crashes when plugging or unplugging a device

The MT3620 developer board has a Future Technology Devices International (FTDI) FT4232HQ chip, which facilitates communication between the device and PC. The official FTDI driver, Combined Driver Model (CMD), contains two drivers: one provides access through D2XX API, and the other provides a Virtual Com Port (VCP) for the same device. Both drivers are installed by default if the FTDI chip has VCP mode enabled. This may cause Windows to crash when the chip is power cycled.

To resolve this problem, you can disable VCP mode for the FTDI chip. You'll need to use the FT_PROG tool to reprogram the FTDI chip's EEPROM.

1. See FTDI FT_PROG programming tool to find out how to download and install this tool.

2. Run FT_PROG and find your attached device, as described in FT_PROG GUI application.

3. In the Device Tree view, expand the Hardware Specific section. You should see four ports.

   

4. Select Port A and choose D2XX Direct instead of Virtual Com Port.

   

5. Repeat the previous step for Port C and Port D. Port B should be in the D2XX Direct mode already.

6. Select the Program Devices icon (resembles a lightning bolt) to enter programming mode.

   

7. Select Program to program the EEPROM, then wait for it to finish.

   

8. Unplug your device from the USB port, then reconnect it to power-cycle the device and cause the change to take effect. In Windows Device Manager, the Ports (COM & LPT) section should now show three fewer COM ports. The number of Universal Serial Bus devices should remain the same.

Lost connection to non-Azure Sphere FTDI devices after enabling RTApp debug

Some Azure Sphere users have reported that they can no longer communicate with other attached, non-Azure Sphere FTDI devices after they use the az sphere device enable-development --enable-rt-core-debugging command to develop and debug RTApps from their host PC.

For example, if you have both an Azure Sphere device and a different FTDI device attached to your PC, you may see two Universal Serial Controllers named USB Serial Converter B devices in Windows Device Manager before running the command.

After executing the command, both USB Serial Converter B devices disappear from the Universal Serial Bus controller section and two new devices appear in the Universal Serial Bus devices display in Device Manager.

Cause

This problem occurs because the az sphere device enable-development --enable-rt-core-debugging command installs a new driver for Port B of the FTDI chip on the MT3620; the port then becomes MSFT MT3620 Std Interface. However, installation of this driver inadvertently changes the driver for Port B of the other non-Azure Sphere device. Because of a limitation in the underlying library, all FTDI devices with the same VID (0x0403) and PID (0x6011) will have their Port B driver replaced.

Solution

Follow these steps to manually revert the driver for any non-Azure Sphere devices to its previous version:

1. In Device Manager, select the non-Azure Sphere device (Another FTDI Quad GZ in the example), then right-click and select Update driver.

2. In Update Drivers, select Browse my computer for driver software.

   

3. Select Let me pick from a list of available drivers on my computer.

   

4. Select USB Serial Converter B driver from the list, then click Next.

   

5. Click Close in the confirmation window.

6. Device Manager should show Port B for the other FTDI device as USB Serial Converter B, which indicates that it uses the official FTDI driver. The driver for the MT3620 remains MSFT MT3620 Std Interface.

   

Additional information

• If you plug in another new non-Azure Sphere FTDI device after running the azsphere device enable-development --enable-rt-core-debugging command, that device will be assigned the Azure Sphere MT3620's driver in the same manner. Repeat the steps above to revert the device to the official FTDI driver.

• If you unplug and re-plug a non-Azure Sphere FTDI device after you return it to the official FTDI driver, the device will keep the official FTDI driver.

• If you run the az sphere device enable-development --enable-rt-core-debugging command again after you revert the driver, the non-Azure Sphere FTDI device will once again have its driver replaced, and you'll need to follow the steps in Solution to revert to the official FTDI driver. This occurs regardless of whether the non-Azure Sphere FTDI device is attached to the PC when the az sphere device enable-development --enable-rt-core-debugging command is run.

Commands not recognized

If you see the following error when you enter an azsphere command, ensure that you're using PowerShell or a standard command prompt on Windows.

'azsphere' is not recognized as an internal or external command, operable program or batch file.

Installer hangs at 60 percent

The installer hangs at 60%, and you are told that the Device Communication Service has failed to start. This typically occurs when the TAP driver installation fails in a strange way, leaving the system in an indeterminate state.

Confirm that the problem is the TAP driver

1. Open the Windows Event Viewer to check the logs.

2. Look in the Application log and the Azure Sphere Device Communication Service log for the following error message:

   "SerialSlipToTun.TunInterfaceSetupException: Error access tun registry settings ---> System.Collections.Generic.KeyNotFoundException: Tun tap device not found ---> System.Security.SecurityException: Requested registry access is not allowed."

   

3. When checking the Application log, filter the log to avoid seeing the many unrelated messages. On the Action tab, select Filter Current Log.

4. Select Error, then select AzureSphereDeviceCommunicationService to list only error messages from the Azure Sphere Device Communication Service.

   

5. If you cannot find the error in either the Application log or the Azure Sphere Device Communication Service log, then this may not be a TAP driver problem.

To resolve the TAP driver problem, follow these steps:

1. Go to the Network and Sharing Center and select Change adapter settings.

   

2. In Network Connections under Azure Sphere, select TAP-WIndows Adapter V9 and open its properties.

   

3. In Azure Sphere Properties, select Internet Protocol Version 4 (TCP/IPv4), then select Properties to view the protocol settings.

   

4. Ensure that the IP address is set to 192.168.35.1 and the subnet mask is set to 255.255.255.0.

5. Try the installer again. If it still hangs, try resetting your network connections. To do a reset, go to Settings > Network & Internet > Status and select Network reset near the bottom of the page.

   Important

   Resetting your network will reset all network settings.

Stop the Azure Sphere Device Communication Service

If the installer hangs you can stop the Azure Sphere Device Communication Service using the taskkill command.

Follow these steps:

1. Copy the PID for the AzureSphereDeviceCommunicationService.

   • Go to Task Manager > Services tab and note down the PID for the AzureSphereDeviceCommunicationService service, or
   • In the command line, type tasklist /fi "SERVICES eq AzureSphereDeviceCommunicationService*" and copy the PID from the output.

2. Type the following in the command line to end the service. The taskkill command ends the process that corresponds to the process ID number. The /f option is used to forcefully end the process.

   taskkill /PID <DCS_PID> /f

3. Try the installer again.

Cannot stop Azure Sphere Device Communication Service

When upgrading an existing Azure Sphere SDK, you might see a message saying "Service 'Azure Sphere Device Communication Service' (AzureSphereDeviceCommunicationService) could not be stopped. Verify that you have sufficient privileges to stop system services." If you receive this message, reboot and run the installer again.

Device communication is broken when split tunneling is used

When a development computer is connected to the corporate VPN and split tunneling is disabled by the VPN configuration, the development computer may not be able to communicate with the Azure Sphere device.

To resolve this problem, do one of the following:

• Disconnect from the corporate VPN and try the connection to the Azure Sphere device again.
• Change the VPN configuration to allow split tunneling.

Communication problems may also be caused by the TAP-Windows adapter configuration. See the Connection problems caused by TAP-Windows adapter configuration troubleshooting section.

Cannot install or uninstall Azure Sphere SDK on Windows 11

After upgrading to Windows 11, users are unable to install or uninstall the Azure Sphere SDK. To resolve this problem, install the latest version of the Azure Sphere SDK for Windows. After installing 21.07 Update 2 or later, you will be able to uninstall the SDK.

Cannot compile or debug RTApps

If CMake reports errors finding the toolchains or openocd when you try to build or debug an RTApp, make sure that:

• You have installed the GNU Arm Embedded Toolchain for your development environment.
• You have specified the correct path to openocd as described in Tutorial: Build a real-time capable application.

Troubleshooting pre-23.05 device communication problems

Warning

These steps are for users with SDK version 22.11 or earlier. Verify the installed SDK version is 22.11 or earlier by running azsphere show-version.

On Linux systems running SDK version 22.11 or earlier, the azsphere_connect.sh script needs to be run each time you plug in the device or unplug or replug it. The script refreshes device connections and spawns a kernel driver that maps network traffic to the device.

If you are encountering communication problems, start with these basic steps:

1. Ensure that the device is connected by USB.

2. Run the azsphere_connect.sh script, if you haven't already done so.

3. Verify you can communicate directly with the device. Navigate to https://192.168.35.2/status in a web browser. Ignore and dismiss any presented SSL warnings. You should see output similar to: {"uptime":56} in the web browser window.

Note

You can also perform this step from the command line using curl https://192.168.35.2/status --insecure.

1. If the error persists, unplug the device from the USB port, plug it in again, and wait for it to restart. Then run the azsphere_connect.sh script.

2. If the error continues, ensure there are no subnet conflicts with other networking tools.