Cannot Import Update to Azure IoT Hub Device Update: Internal Server Error

Matthew Farstad 5

I'm attempting to import an update to device update service in IoT Hub. I'm getting an internal server error with this trace ID. What is the issue?

1a8f0229c7b24e0489ef64d4e9de397f

LeelaRajeshSayana-MSFT 13,791 Reputation points

2024-02-27T14:38:33.05+00:00

Hi @Matthew Farstad Thank you for posting the question on the community. Can you share more details on the error message you see in the logs to help us better understand the issue you are facing?
Matthew Farstad 5 Reputation points

2024-02-27T14:41:17.3733333+00:00

I attempt to import an update to device update in Azure IoT hub and it says "internal server error" with the trace ID to provide to MSFT support. This most recent round of me trying it said I was importing malware which is interesting since its two binaries that I produced that have never been flagged in any other update before and then text files or shell scripts (which have also never been flagged before).
Matthew Farstad 5 Reputation points

2024-02-27T14:43:01.7466667+00:00

ignore this comment can't delete
Matthew Farstad 5 Reputation points

2024-02-28T17:14:11.55+00:00

Apparently the issue is that Golang binaries are often flagged as malware. Can I disable this check?
LeelaRajeshSayana-MSFT 13,791 Reputation points

2024-02-28T19:34:13.2933333+00:00
Hi @Matthew Farstad Thank you for sharing this information with us. We will review this scenario with the product team and share more findings on this with you.

I also appreciate it if you can share additional details such as the steps you have followed in creating the update files and the step handler through which you are testing the update. Following is a sample command that helps you create the update file

az iot du update init v5 \ --update-provider Microsoft \ --update-name AptUpdate \ --update-version 1.0.0 \ --compat manufacturer=Contoso model=Vacuum \ --step handler=microsoft/script:1 properties='{"installedCriteria": "1.0"}' \ --file path=/my/apt/manifest/file

I appreciate it if you can give this approach a try in creating the update file and try importing it into the IoT Hub Device Update. Let us know if you still encounter the issue after trying the above approach.

Matthew Farstad 5

Here's how I generate the import manifest

az iot du update init v5 \
    --update-provider company \
    --update-name "ota" \
    --update-version $__VERSION \
    --compat manufacturer=company model=$__MODEL \
    --step handler=microsoft/swupdate:1 properties="{\"installedCriteria\": \"$__VERSION\"}" \
    --file path=$1 > company_${__VERSION}.importmanifest.json

Here's how I make the .swu file

#!/bin/bash -e

CONTAINER_VER="$1"

echo "packaging $CONTAINER_VER"

PRODUCT_NAME="update"

mkdir -p archive
mv files/*.swu archive/ || true
mv *.importmanifest.json archive/ || true
rm files/sw-description.sig || true

__FILES="$(ls files | sed 's/sw-description//g' | sed 's/.sig//g' | sed 's/.*\.swu//g' | sed '/^\s*$/d')"

cd files

echo "software =
{
    version = \"$CONTAINER_VER\";
    hardware-compatibility: [\"1.0\"];

    stable = {
        update: {
            files: (" | tee sw-description

    for i in $__FILES; do
        __PATH=$(gum input --placeholder "where should $i be installed?" --value="/mnt/$i")
        echo "                {
                    filename = \"$i\";
                    path = \"$__PATH\";
                    sha256 = \"$(sha256sum $i | awk '{print $1}')\";
                    properties = {create-destination = \"true\"; atomic-install = \"true\";}
                }," | tee -a sw-description
    done

    sed -i '$ s/.$//' sw-description

    echo '            );
        }
    };
}' | tee -a sw-description

FILES="sw-description sw-description.sig $__FILES"

openssl cms -sign -in  sw-description -out sw-description.sig -signer public.cert \
        -inkey ../mycert.key.pem -outform DER -nosmimecap -binary

for i in $FILES; do
    echo $i;
done | cpio -ov -H crc -F ${PRODUCT_NAME}_${CONTAINER_VER}.swu

az storage blob upload --account-name account --container-name updates --name ${PRODUCT_NAME}_${CONTAINER_VER}.swu --file ${PRODUCT_NAME}_${CONTAINER_VER}.swu --auth-mode login --overwrite

I then import the importmanifest and .swu file as an update in Device Update.

LeelaRajeshSayana-MSFT 13,791 Reputation points

2024-02-28T20:37:25.8533333+00:00

Hi @Matthew Farstad Thank you for sharing the requested information. I am checking with the product team to get more feedback on this. Please stay tuned to this thread for more updates in the next few days.
Matthew Farstad 5 Reputation points

2024-02-28T20:46:17.66+00:00

I've submitted a copy of one of the afflicted binaries to Microsoft Security Inteligence Malware Scanner with submission ID 8bc22fb3-cd12-49fb-a285-71f2fb0d6efd

1 answer

Andrew Brown (W+D SERVICES) 5 Reputation points Microsoft Employee

2024-02-29T16:48:19.4633333+00:00

Thank you very much for reaching out regarding this issue. We had seen this week in telemetry that one of our customers had been encountering import failures due to anti-malware scanning, but we by design do not collect information that allow us to directly identify or contact customers. So I'm glad we can investigate this with you and hopefully resolve this issue.

If possible, it would be great to get a copy of the update you are submitting which is causing what appears to be a false positive. I'd like to provide that directly to the anti-malware scan team so they can investigate in more detail (and, if benign, ensure that it no longer is flagged as malware). , is there a recommended approach for file sharing like this, if @Matthew Farstad consents?

Another helpful thing would be to get additional error information. Can you take a look at the last entries on this documentation page: https://learn.microsoft.com/en-us/azure/iot-hub-device-update/device-update-error-codes#device-update-content-service, and in particular, the part about the error message containing the description of the malware signature, and a file hash for each file where the signature was detected? The file hash is intended to help our customers figure out which file(s) are being flagged, so I don't need that part, but the description of the malware signature can help us narrow down where the potential false positive is occurring on our side. That information should be present in the Azure portal UI in the Import History for any import attempts that fail due to anti-malware scan.

One final note: we have considered in the past making anti-malware scan optional, for cases such as this. Our initial implementation biased towards being more secure at the potential cost of occasional false positives blocking import. However, we are now actively planning for how we could make anti-malware scan optional, though I don't yet have a timeline to share.
Please sign in to rate this answer.

1 person found this answer helpful.
Matthew Farstad 5 Reputation points

2024-02-29T16:58:14.05+00:00

Malware was detected in source file. (ScanFailedWithMalware: Status=MalwareFound; ErrorCode=BlockingDetectionFound; FileManagementId:redacted; Operation:FileScan; ErrorCode:ScanFailedWithMalware; InnerException(none):null )

Whenever we know how its best for me to provide files, I'm happy to give both the .swu files and the individual binaries.

I narrowed it down by slowly adding files to the CPIO until it failed. Two go binaries both raise the flag, but they have a lot of code overlap.

Once we fix this false flag, how do we make sure that as I make changes to binaries we don't have to do this every time? I'm happy to just ping some MSFT employee, but that seems like not a good use of time for anyone involved.

JPMO 0 Reputation points

2024-02-29T17:01:30.52+00:00

I submitted the affected update package for analysis and got the following response from an analyst:

https://www.microsoft.com/en-us/wdsi/submission/ff956f2b-17d1-430f-a88f-921df7df2272

Matthew Farstad 5 Reputation points

2024-03-02T00:48:18.23+00:00

Another thing: sometimes it says its malware. Sometimes I just get something like this:

Operation cannot be completed due to internal server error.

Trace ID

e11aeefcd9524063b1e1a7016017c344

Provide to your support person for toubleshooting.

Matthew Farstad 5 Reputation points

2024-03-04T21:12:36.5+00:00

So on top of all of this, updates which were previously working without the malware binaries are now taking upwards of 30 minutes for a 15MB .swu file + manifest.

Andrew Brown (W+D SERVICES) 5 Reputation points Microsoft Employee

2024-03-04T21:46:02.1+00:00

Let me see if I can summarize current state:

You have been able to determine which files were generating the false positive and remove them.

You have been able to successfully import an update by omitting the files from #1.

Separately, there is an indication from JPMO's post that those files will no longer generate false positives. Curious if you have verified that your original updates can now be imported (without removing any files)?

You are still seeing other issues with import:

500 internal server error - these should be rare, and they fire alerts to our on-call team who works to address them quickly. If you are seeing one for more than a few hours, please let us know.

General slowness when importing a file - the import duration in Device Update for IoT Hub is primarily the anti-malware scan time, which is one reason we're considering making scan optional. The scan time can be variable based on the size and also the composition of an update. So if you have an update with a large number of nested directories, for example, that would be expected to take longer than an update of similar (or even larger) size which has a simpler folder structure. Compression of the files is also an important factor as well. If you are seeing significant variability when importing and re-importing an identical file over time, that would be interesting to learn more about.

To answer your question about what to do in the future to avoid a similar false-positive scenario: this is the first time we have seen a customer encounter this over a period of several years. Based on that, it should be highly unlikely that you encounter this again. If this does end up becoming a more common occurrence, our team will work with the anti-malware scan team on mitigations; it would also increase the priority of our work to make anti-malware scan optional, which would be the preferred long-term solution.

Matthew Farstad 5 Reputation points

2024-03-04T22:04:16.2466667+00:00

I don't recall if I have encountered the malware result since JPMO got the all clear back. Since then I have just been getting internal server errors (sometimes several an hour) with maybe 1-2 successful imports of trivial files in a .swu file. Before JPMO submitted file for scan I would repeatedly get either internal server errors OR the malware flag for the exact same file. Now I just get errors and my imports seem to either time out or have some other issues on backend. Latest attempt was 45 minutes of "Running" before I got the error.

Matthew Farstad 5 Reputation points

2024-03-06T14:37:11.71+00:00

I got another import attempt that timed out after 45 minutes with an internal server error.
Sign in to comment

Share via

Cannot Import Update to Azure IoT Hub Device Update: Internal Server Error

1 answer