Synapse SQL Serverless error: 'WaitIOCompletion call failed. HRESULT = 0x8007000C'

Question

Hello,

we get the following error when running a query to our Synapse SQL Serverless database:

Error handling external file: 'WaitIOCompletion call failed. HRESULT = 0x8007000C'. File/External table name: '<redacted: path to file on storage account>'.

This error occurs when querying a view which uses OPENROWSET, e.g.:

SELECT *
FROM OPENROWSET(
        BULK 'PATH/ON/STORAGE/ACCOUNT/**',
        DATA_SOURCE = 'AzureDataLakeStoreV2',
        FORMAT = 'PARQUET'
) t

The folder in the storage account contains several parquet files in subfolders, all files with the same structure.
Managed identify authentication is used for the data source.

I am not able to find any information about when 'WaitIOCompletion call failed' can occur.

Following discovering have been made:

• When recreating the view the query runs fine afterwards
• Still, after updating data in the storage account, the error occurs again
• Other views/queries work just fine
• We have different identical environments on which the problem does not occur (obviously there has to be a difference somewhere, still Synapse environments are configured in the same manner)
• When running this query outside of Synapse Studio with a user authenticated by AD Authentication the query also works fine -> seems like there is something going on with the managed identity authentication
• Recreating external data source and scoped credential did not solve the problem.
• Recreating the folder structure in the storage account was also of no prevail

I can provide more information about our setup if required.
Thanks for your help.

Answer 1

We resolved the issue by recreating the Synapse Serverless database inlc. recreation of the database MASTER KEY. Since then the issue did not occur again, but the underlying reason is still unclear.

It may be the case that recreating the MASTER KEY is sufficient to get rid of the issue, we just recreated the database using our infrastructure scripts and did not test it independently. As mentioned in the original issue description, recreation of other elements (e.g. scoped credential) was not sufficient.

While debugging we also discovered that the error message sometimes changed to

Error: handling external file: 'Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.'. File/External table name: <path to file>

when running the query in Synapse. It was not possible to make out a deterministic behaviour for when which error message will be displayed.

Answer 2

Sorry, this is not a reply...I created a new question (synapse-sql-serverless-error-39waitiocompletion-ca-1.html) referencing this question.

Hi,

We have exact the same error (location West Europe) but the solution above does not work in our case.
The same environment which executes the same procedure every day suddenly stopped working.
Friday morning November 12th (CET) was the last time it was ok, since then always a WaitIOCompletion error is raised.
Even the parquet files it processed the days before without problem now generate the same error.
I can open the parquet files from the ADLSv2 without problem using Storage Explorer.
Recreating the database, masterkey and data source does not help.

Any help is welcome :)

Answer 3

We had the exact same issue in the West-Europe region since friday.

In the end I sent in a Microsoft Support ticket and the issue was mitigated by MS about an hour ago. The issue was on their side.
Still waiting for the RCA atm.

Answer 4

I am just trying to read a CSV file using a serverless pool in synapse. I did not create a database, I just loaded a file and am trying to read. Is a database required to query a csv file in the data Lake from synapse?