Troubleshoot Azure Sphere problems

  • Article

This topic provides troubleshooting steps for the following types of problems:

General device connection problems

A failure to connect to the device from a host machine can occur for many reasons and may trigger any of several error messages, depending on which tools or applications encounter it. The following error messages may indicate a failed connection:

  • An error occurred. Please check your device is connected and your PC has been configured correctly, then retry.
  • Could not connect to the device. Check if your device is connected to the PC. The device may be unresponsive if it is applying an Azure Sphere operating system update; wait a few minutes and then retry. If this issue persists, try uninstalling and reinstalling the Azure Sphere SDK.
  • An unexpected issue occurred. Please try again; if the issue persists, please refer to aka.ms/azurespheresupport for troubleshooting suggestions and support.
  • Failed to retrieve device ID from attached device: 'Could not connect to the device; please ensure it is attached.'
  • Failed to establish communication with device after recovery.

Failure to connect may occur for any of these reasons:

  • The board is not connected by USB to the host machine.

  • The board is not powered.

  • The device is currently busy applying an Azure Sphere OS update. During OS updates, the device may fail to respond.

To diagnose and resolve the problem, start with these basic steps:

  1. Ensure that the device is connected by USB to the host machine.

  2. Wait a few minutes. If OS update is progress, the device should again become responsive when the update completes.

  3. If the device is connected, press the Reset button on the device or disconnect and reconnect the USB cable. Wait ten seconds or so for the device to restart, and then try the failed command again.

  4. On Windows, run the azsphere device rescan-attached command. This command power-cycles the FTDI ports of all attached devices and can sometimes help in connection problems.

If none of these steps solves the problem, see Windows-specific problems or Linux-specific problems to determine whether the problem might be unique to your platform.

Device is unresponsive

If the device appears "dead" or unresponsive, the problem could be a lack of power.

Make sure that the real-time clock (RTC) is powered, by either the main 3V3 power supply or a coin cell battery. If your dev kit is compatible with the MT3620 reference development board (RDB), see the MT3620 development board user guide for details. If you're using a different dev kit, see the manufacturer's documentation.

In addition, make sure that the board itself is properly configured. For example, Seeed MT3620 Development Kits are shipped from the factory with a jumper header across pins 2 and 3 of J3, which powers the clock from the main power supply. Check that the header has not been dislodged or removed.

USB errors

If you encounter USB errors:

  1. Plug the device into a different USB port. If the error occurred when the device was plugged into a USB hub, plug it into a port on the machine instead.

  2. If the problem recurs, plug the device into a powered USB hub. In some cases, USB ports do not provide adequate power for the board, so a powered hub may solve the problem.

Dropped connections or missing devices

If your computer can connect to the Azure Sphere device, but often drops the connection, you may have a conflict in the IP subnet.

Azure Sphere uses subnet 192.168.35.*. If you have other software that uses the same subnet, either disable that software or limit the range of IP addresses that it uses. Currently, you cannot change the range of IP addresses that Azure Sphere uses.

If the subnet is not the problem, you may have a problem related to the FTDI driver used on Windows. Driver problems can cause PC-to-device connections to disappear intermittently, or can prevent the azsphere command from finding all attached devices. Run the azsphere device rescan-attached command to power-cycle the FTDI ports of all attached devices.

Cannot apply device capabilities

If you receive errors when you try to apply a device capability to an Azure Sphere device, the problem might be that the OS on your device is out of date. This problem can occur if the device has been offline for an extended period, during which Microsoft updated its internal keys. The following errors are symptoms of this problem:

The azsphere device enable-development or azsphere device enable-cloud-test commands return an error similar to the following:

error: The device did not accept the device capability configuration. Please check the Azure Sphere OS on your device is up-to-date using 'azsphere device show-deployment-status'.

The azsphere device capability update command returns an error similar to the following:

error: Could not apply device capability configuration to device.

This is due to the trusted keystore on the device being out of date. To solve this problem follow the instructions here.

Windows-specific problems

In addition to the problems listed in Device connection problems, failure to connect on Windows may occur for any of these reasons:

  • The Azure Sphere Device Communication Service has not yet started.
  • The TAP-Windows adapter is not installed or is not configured correctly.

To resolve the problem from Windows, start with these basic steps:

  1. Make sure the device is connected to the PC by USB and has power.

  2. Restart the Azure Sphere Device Communication Service. Press Start and enter Services. Right-click the Azure Sphere Device Communication Service and select Restart.

  3. If the error recurs after you restart the service, use the View Network Connections control panel to check that the adapter exists and is configured to use IP address 192.168.35.1. To access this control panel, press Start and enter ncpa.cpl. Right-click the Azure Sphere device to view its properties. If Azure Sphere is not visible or does not have the correct IP address, reinstall the Azure Sphere SDK.

  4. If the device is connected, press the Reset button on the device. Wait ten seconds or so for the device to restart, and then try the failed command again.

  5. If the error recurs, unplug the device from the USB port, plug it in again, and wait for it to restart.

If none of these steps solves the problem, try the additional solutions that follow.

Failure to create four USB serial converters

After you set up an MT3620 development board, you should see four USB serial converters in Device Manager. If you see fewer than four, you may have a problem with the FTDI driver.

Note

If this board has previously been used for RTApp development, you may see three converters instead of four. This is normal and does not represent an error.

If the FTDI driver is not correctly installed, the converters might appear in the wrong location, such as Other devices, or might not appear at all.

To resolve this problem:

  1. To open Device Manager, click Start and enter Device Manager.

  2. Under Universal Serial Bus controllers, select USB Serial Converter A. Right-click the name, select Uninstall Device, and delete the driver if given the option:

    Uninstall device and delete driver

    Repeat this step for USB Serial Converter B through USB Serial Converter D.

  3. Unplug your development board from your PC and plug it back in again. "MSFT MT3620 Std Interface" should appear with a triangle warning icon, which indicates no driver is available.

  4. Right-click one of the MSFT MT3620 Std Interface devices and select Update driver. Choose Search automatically for updated driver software. Updating one should fix them all. You should now see four USB serial converters in the Universal Serial Bus controllers section. If all four converters do not appear, repeat this step for each converter.

Failure to install FTDI drivers

The FTDI drivers should be downloaded and installed automatically by Windows when your Azure Sphere device is first plugged in to your PC. If the drivers are installed correctly, you will see four USB Serial Converters listed under Universal Serial Bus controllers in Device Manager, as described in Set up your dev kit.

Windows 10, version 2004, does not search for the drivers. In this case, the drivers are not downloaded and installed automatically and you will see the following items listed in Device Manager:

MT3620 items listed in Device Manager

To install the drivers, manually download the drivers from Future Technology Devices International (FTDI). Choose the driver that matches your Windows installation (32- or 64-bit).

For availability dates and build numbers of Windows 10 versions, see Windows 10 release information. This information can help you determine whether your version of Windows 10 is earlier or later than version 2004.

Connection problems caused by TAP-Windows adapter configuration

Azure Sphere tools communicate with attached development boards by using an IP network over USB. This requires the TAP-Windows adapter from OpenVPN Technologies. The Azure Sphere SDK installation procedure installs this adapter on your PC if it is not already present.

Two distinct types of problems have been reported with the TAP-Windows adapter:

  • If a different version of the TAP-Windows adapter is already installed, or if the Azure Sphere device is not connected to the first instance of the TAP-Windows adapter, the Azure Sphere tools may fail to connect to your device.

  • If you are using the Cisco AnyConnect virtual private network (VPN), your TAP-Windows adapter may appear unplugged and device communication may not work. This is because the VPN can redirect traffic away from the TAP adapter's IP address.

Ensure Azure Sphere device is connected to the appropriate TAP-Windows adapter

To determine whether the problem is related to the TAP adapter, first find out how many TAP adapters are installed on your PC, and then modify the installation if necessary.

To determine how many TAP adapters are installed on your PC:

  1. Open Windows Settings and select the Network & Internet group.

  2. Under Advanced network settings, select Change adapter options. You should see only one TAP adapter, as shown in the following screenshot:

    One TAP Adapter

    If you see more than one TAP adapter, or if you see only one TAP adapter but its name is not Azure Sphere, follow these steps to uninstall all TAP adapters and reinstall the SDK. If you see no TAP adapters, reinstall the SDK.

To uninstall the TAP adapters:

  1. Click Start and enter Device Manager.

  2. In Device Manager, open Network Adapters and select TAP-Windows adapter:

    Device Manager with TAP adapter

  3. Right-click TAP-Windows adapter and select Uninstall Device. In the dialog box, select Delete the driver software for this device, then click Uninstall.

  4. Open a command prompt as Administrator and run the following Powershell installer script:

     powershell -ExecutionPolicy RemoteSigned -File "%ProgramData%\Microsoft\Azure Sphere\TapDriverInstaller\TapDriverInstaller.ps1" Install

  5. If the installation succeeds, restart the Azure Sphere Device Communication Service:

    net stop AzureSphereDeviceCommunicationService

    net start AzureSphereDeviceCommunicationService

  6. Reinstall the Azure Sphere SDK.

Correct the unplugged state when using Cisco AnyConnect VPN client

When you are using Cisco AnyConnect VPN client, the TAP-Windows adapter may appear unplugged and device communication may not be working. You can correct the problem as follows:

  1. Make sure that you have administrator privileges on your computer.

  2. Open Windows Settings and select the Network & Internet group.

  3. Under Advanced network settings, select Change adapter options.

  4. Right-click Azure Sphere TAP-Windows Adapter V9 and select Properties.

  5. On the Networking tab, locate the Cisco AnyConnect Network Access Manager Filter Driver entry and clear the item.

    TAP-Windows adapter properties showing Cisco AnyConnect item unselected

  6. Select OK to save the setting and exit the properties.

  7. Check that communication with your Azure Sphere device is now working by opening a command prompt and entering the azsphere device show-attached command.

Device does not respond

One or more of the following errors from an azsphere command may indicate that the Azure Sphere Device Communication Service failed to start:

  • warn: Device is not responding. Could not perform version check.​
  • Device is not responding. Cannot get device ID.​
  • error: Could not connect to the Azure Sphere Device Communication Service. If this issue persists, try uninstalling and reinstalling the Azure Sphere SDK.​
  • error: The device is not responding. The device may be unresponsive if it is applying an Azure Sphere operating system update; please retry in a few minutes.

The service may fail to start after Windows update and in cases where one of the internal JSON settings files or config file has become corrupted.

Failure after Windows Update

These errors can occur after you've updated Windows on your PC. Sometimes Windows Update uninstalls the FTDI drivers required for the communication service.

To resolve the problem:

  1. Unplug the Azure Sphere device from USB and plug it in again. Upon replugging the device, the correct drivers should reinstall.
  2. If unplugging and replugging the device fails to fix the problem, uninstall and reinstall the Azure Sphere SDK.

JSON file

If you have not recently updated Windows, the cause of the error might be the restore.json file that is used for the service.

To resolve this problem:

  1. Save a copy of the following file:

    c:\windows\serviceprofiles\localservice\appdata\local\Azure Sphere Tools\restore.json

  2. Delete the file from its original location.

  3. Stop and then restart the Azure Sphere Device Communication Service:

    net stop AzureSphereDeviceCommunicationService

    net start AzureSphereDeviceCommunicationService

Corrupted config file

If an error is reported when you try to run a command, the corrupted config file might be preventing your device from running correctly.

To resolve this problem, delete the corrupted config file located in .azsphere\config on Windows or ~/.azsphere/config on Linux.

Windows crashes when plugging or unplugging a device

The MT3620 developer board has a Future Technology Devices International (FTDI) FT4232HQ chip, which facilitates communication between the device and PC. The official FTDI driver, Combined Driver Model (CMD), contains two drivers: one provides access through D2XX API, and the other provides a Virtual Com Port (VCP) for the same device. Both drivers are installed by default if the FTDI chip has VCP mode enabled. This may cause Windows to crash when the chip is power cycled.

To resolve this problem, you can disable VCP mode for the FTDI chip. You'll need to use the FT_PROG tool to reprogram the FTDI chip's EEPROM.

  1. See FTDI FT_PROG programming tool to find out how to download and install this tool.

  2. Run FT_PROG and find your attached device, as described in FT_PROG GUI application.

  3. In the Device Tree view, expand the Hardware Specific section. You should see four ports.

    FTProg Hardware Specific four ports

  4. Select Port A and choose D2XX Direct instead of Virtual Com Port.

    Select port A and D2XX direct

  5. Repeat the previous step for Port C and Port D. Port B should be in the D2XX Direct mode already.

  6. Select the Program Devices icon (resembles a lightning bolt) to enter programming mode.

    click program devices

  7. Select Program to program the EEPROM, then wait for it to finish.

    program devices

  8. Unplug your device from the USB port, then reconnect it to power-cycle the device and cause the change to take effect. In Windows Device Manager, the Ports (COM & LPT) section should now show three fewer COM ports. The number of Universal Serial Bus devices should remain the same.

Lost connection to non-Azure Sphere FTDI devices after enabling RTApp debug

Some Azure Sphere users have reported that they can no longer communicate with other attached, non-Azure Sphere FTDI devices after they use the azsphere device enable-development --enable-rt-core-debugging command to develop and debug RTApps from their host PC.

For example, if you have both an Azure Sphere device and a different FTDI device attached to your PC, you may see two Universal Serial Controllers named USB Serial Converter B devices in Windows Device Manager before running the command.

device manager with two serial converter B

After executing the command, both USB Serial Converter B devices disappear from the Universal Serial Bus controller section and two new devices appear in the Universal Serial Bus devices display in Device Manager.

device manager two USB devices

Cause

This problem occurs because the azsphere device enable-development --enable-rt-core-debugging command installs a new driver for Port B of the FTDI chip on the MT3620; the port then becomes MSFT MT3620 Std Interface. However, installation of this driver inadvertently changes the driver for Port B of the other non-Azure Sphere device. Because of a limitation in the underlying library, all FTDI devices with the same VID (0x0403) and PID (0x6011) will have their Port B driver replaced.

Solution

Follow these steps to manually revert the driver for any non-Azure Sphere devices to its previous version:

  1. In Device Manager, select the non-Azure Sphere device (Another FTDI Quad GZ in the example), then right-click and select Update driver.

  2. In Update Drivers, select Browse my computer for driver software.

    browse my computer for drivers

  3. Select Let me pick from a list of available drivers on my computer.

    let me pick

  4. Select USB Serial Converter B driver from the list, then click Next.

    usb serial converter b

  5. Click Close in the confirmation window.

  6. Device Manager should show Port B for the other FTDI device as USB Serial Converter B, which indicates that it uses the official FTDI driver. The driver for the MT3620 remains MSFT MT3620 Std Interface.

    Port B has reverted

Additional information

  • If you plug in another new non-Azure Sphere FTDI device after running the azsphere device enable-development --enable-rt-core-debugging command, that device will be assigned the Azure Sphere MT3620's driver in the same manner. Repeat the steps above to revert the device to the official FTDI driver.

  • If you unplug and re-plug a non-Azure Sphere FTDI device after you return it to the official FTDI driver, the device will keep the official FTDI driver.

  • If you run the azsphere device enable-development --enable-rt-core-debugging command again after you revert the driver, the non-Azure Sphere FTDI device will once again have its driver replaced, and you'll need to follow the steps in Solution to revert to the official FTDI driver. This occurs regardless of whether the non-Azure Sphere FTDI device is attached to the PC when the azsphere device enable-development --enable-rt-core-debugging command is run.

Commands not recognized

If you see the following error when you enter an azsphere command, ensure that you're using PowerShell or a standard command prompt on Windows.

'azsphere' is not recognized as an internal or external command, operable program or batch file.

Installer hangs at 60 percent

The installer hangs at 60%, and you are told that the Device Communication Service has failed to start. This typically occurs when the TAP driver installation fails in a strange way, leaving the system in an indeterminate state.

Confirm that the problem is the TAP driver

  1. Open the Windows Event Viewer to check the logs.

  2. Look in the Application log and the Azure Sphere Device Communication Service log for the following error message:

    "SerialSlipToTun.TunInterfaceSetupException: Error access tun registry settings ---> System.Collections.Generic.KeyNotFoundException: Tun tap device not found ---> System.Security.SecurityException: Requested registry access is not allowed."

    Screenshot of the Windows Event Viewer.

  3. When checking the Application log, filter the log to avoid seeing the many unrelated messages. On the Action tab, select Filter Current Log.

  4. Select Error, then select AzureSphereDeviceCommunicationService to list only error messages from the Azure Sphere Device Communication Service.

    Screenshot of the Filter Current Log dialog box.

  5. If you cannot find the error in either the Application log or the Azure Sphere Device Communication Service log, then this may not be a TAP driver problem.

To resolve the TAP driver problem, follow these steps:

  1. Go to the Network and Sharing Center and select Change adapter settings.

    Screenshot of the Network Sharing Center.

  2. In Network Connections under Azure Sphere, select TAP-WIndows Adapter V9 and open its properties.

    Screenshot of Network Connections.

  3. In Azure Sphere Properties, select Internet Protocol Version 4 (TCP/IPv4), then select Properties to view the protocol settings.

    Screenshot of Azure Sphere Properties.

  4. Ensure that the IP address is set to 192.168.35.1 and the subnet mask is set to 255.255.255.0.

  5. Try the installer again. If it still hangs, try resetting your network connections. To do a reset, go to Settings > Network & Internet > Status and select Network reset near the bottom of the page.

    Important

    Resetting your network will reset all network settings.

Stop the Azure Sphere Device Communication Service

If the installer hangs you can stop the Azure Sphere Device Communication Service using the taskkill command.

Follow these steps:

  1. Copy the PID for the AzureSphereDeviceCommunicationService.

    • Go to Task Manager > Services tab and note down the PID for the AzureSphereDeviceCommunicationService service, or
    • In the command line, type tasklist /fi "SERVICES eq AzureSphereDeviceCommunicationService*" and copy the PID from the output.

  2. Type the following in the command line to end the service. The taskkill command ends the process that corresponds to the process ID number. The /f option is used to forcefully end the process.

    taskkill /PID <DCS_PID> /f

  3. Try the installer again.

Cannot stop Azure Sphere Device Communication Service

When upgrading an existing Azure Sphere SDK, you might see a message saying "Service 'Azure Sphere Device Communication Service' (AzureSphereDeviceCommunicationService) could not be stopped. Verify that you have sufficient privileges to stop system services." If you receive this message, reboot and run the installer again.

Device communications is broken when split tunneling is used

When a development computer is connected to the corporate VPN and split tunneling is disabled by the VPN configuration, the development computer may not be able to communicate with the Azure Sphere device.

To resolve this problem, do one of the following:

  • Disconnect from the corporate VPN and try the connection to the Azure Sphere device again.
  • Change the VPN configuration to allow split tunneling.

Communication problems may also be caused by the TAP-Windows adapter configuration. See the Connection problems caused by TAP-Windows adapter configuration troubleshooting section.

Cannot install or uninstall Azure Sphere SDK on Windows 11

After upgrading to Windows 11, users are unable to install or uninstall the Azure Sphere SDK. To resolve this problem, install the latest version of the Azure Sphere SDK for Windows. After installing 21.07 Update 2 or later, you will be able to uninstall the SDK.

Cannot compile or debug RTApps

If CMake reports errors finding the toolchains or openocd when you try to build or debug an RTApp, make sure that:

Linux specific problems

Device does not respond

One or more of the following errors from an azsphere command may indicate that the Azure Sphere Device Communication Service (azsphered) failed to start:

  • warn: Device is not responding. Could not perform version check.​
  • Device is not responding. Cannot get device ID.​
  • error: Could not connect to the Azure Sphere Device Communication Service. If this issue persists, try uninstalling and reinstalling the Azure Sphere SDK.​
  • error: The device is not responding. The device may be unresponsive if it is applying an Azure Sphere operating system update; please retry in a few minutes.

The first step is to restart your computer. After restarting your computer, run azsphere device list-attached. If any of the error messages above are present or your device is not listed, then proceed to the following steps.

  1. Restart the Azure Sphere Linux Device Communication service: sudo azspheredctl restart

  2. Check that the service started successfully: sudo azspheredctl status

  3. Check the output resembles the following:

azsphered.service - Azure Sphere Daemon
     Loaded: loaded (/lib/systemd/system/azsphered.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-01-17 10:46:20 GMT; 11s ago
   Main PID: 53221 (azsphered)
      Tasks: 10 (limit: 19012)
     Memory: 2.9M
     CGroup: /system.slice/azsphered.service
             └─53221 /etc/azure-sphere/azsphered

Jan 17 10:46:20 systemd[1]: Started Azure Sphere Daemon.

Install script cannot start the Device Communication Service

If during installation you see the message: WARN: Could not automatically start the Device Communication service. Is systemd available and enabled? The install script was not able to start the Linux Device Communication service. This can be caused by a failed installation or an error with systemd.

To validate the issue is not systemd related, run: sudo systemctl status azsphered. If output shows the status of azsphered as inactive/dead, the problem is likely a failed installation.

The next step is to manually install the .deb file and its dependencies:

  1. Run sudo apt install libudev-dev.
    • If the installation fails with output similar to libudev-dev : Depends: libudev1 (= 245.4-4ubuntu3) but 245.4-4ubuntu3.17, open Software Updater, apply any pending updates, and re-run sudo apt install libudev-dev.
  2. Run sudo apt install /opt/azurespheresdk/DeviceConnection/azsphered_<version_number>.deb. Either tab complete or manually complete the path to the azsphered Debian package.
  3. Assuming the previous step completed successfully, you should now be able to list devices. Run azsphere device list-attached.