I'm creating a ADF data flow to import data from CosmosDb. I have created a linked service and a dataset to connect with CosmosDb; these connections run successfully: Linked Service connection Dataset connection When i try to do a data preview the following error appears: I would like to mention that this data flow was working correctly 10 days ago. I'm using the same connection (same LS and same DS) on a copy data activity and it's working correctly. Do you have any idea on how can i resolve this issue? Thanks for your help.

Hello @Bhargava-MSFT , thank you for your reply.I'm not using a schema while importing the data on the data flow, neither on the dataset. Even when i try to import the schema from the "Projection" tab i have the same error About the other suggestion i don't think that the firewalls could be a problem because the copy data activities are running correctly and the test connections of Linked service and dataset run successfully. It would be very helpful if you had any other ideas. Thank you in advance!

Error Data Flow connection vs CosmosDb

Izaura Domi 0

I'm creating a ADF data flow to import data from CosmosDb.

I have created a linked service and a dataset to connect with CosmosDb; these connections run successfully:

Linked Service connection
Dataset connection

When i try to do a data preview the following error appears:

User's image

I would like to mention that this data flow was working correctly 10 days ago.

I'm using the same connection (same LS and same DS) on a copy data activity and it's working correctly.

Do you have any idea on how can i resolve this issue?

Thanks for your help.

Bhargava-MSFT 28,956 Reputation points Microsoft Employee

2024-07-16T17:29:29.4533333+00:00

Hello Izaura Domi,

From the error message, it looks like the mapping is trying to access a field that does not exist in the source data.

Please check the schema of cosmos db and make sure it matches the schema defined in the source dataset of the data flow. You can do this by going to the"Projection" tab. Make sure that the columns in the cosmos db dataset match the columns in the source data.

also please check If there are any changes in network settings, firewall, or IP whitelisting.

I hope this helps.

1 answer

Izaura Domi 0 Reputation points

2024-07-17T09:08:50.18+00:00

Hello @Bhargava-MSFT , thank you for your reply.I'm not using a schema while importing the data on the data flow, neither on the dataset. Even when i try to import the schema from the "Projection" tab i have the same error

About the other suggestion i don't think that the firewalls could be a problem because the copy data activities are running correctly and the test connections of Linked service and dataset run successfully.

It would be very helpful if you had any other ideas.

Thank you in advance!
Please sign in to rate this answer.
Bhargava-MSFT 28,956 Reputation points Microsoft Employee

2024-07-18T00:13:24.22+00:00

Try using the copy activity in ADF to preview the data in Cosmos DB. This will help determine if the problem is specific to the data flow.

Verify that the data types and structure in Cosmos DB have not changed. Any changes in the schema could potentially cause issues in the data flow.

Ensure that the JSON documents stored in Cosmos DB are well-formed and consistent. Any malformed JSON or variations in the document structure can lead to issues in the data flow processing.

Izaura Domi 0 Reputation points

2024-07-18T10:43:12.8533333+00:00

Hi @Bhargava-MSFT

Thank you for your reply.

I will answer your propositions in bullet points:

I have created a new copy data activity that imports data from Cosmos DB, and it works correctly every time.

The data types and the structure in Cosmos DB have not changed, and the schema has remained the same. The data flow is not using any schemas or projections, and I can't even import the schema because I always get the same error. So, I think it must be an error in the connection phase. For some reason, the data flow cannot connect to Cosmos DB and read the schema.

The JSON documents have not changed and are well-formed and consistent.

The connection that the linked service is using is public and doesn't have any type of firewall or VNET in use.

I don't know if you have any other ideas I can try; if so, it would be very helpful.

Thank you very much!

Bhargava-MSFT 28,956 Reputation points Microsoft Employee

2024-07-19T20:02:26.4566667+00:00

Izaura Domi

Thank you for the details.

As the next steps, please open a support case so that a support engineer can help troubleshoot the issue in a remote sharing session. If you don't have a support plan, please let me know. I can enable a one-time free support request on your subscription.

I look forward to hearing from you.

Izaura Domi 0 Reputation points

2024-07-23T07:53:03.2866667+00:00

Hi @Bhargava-MSFT

Thank you for your suggestion. I will proceed to open a support case with Microsoft Azure and see if a support engineer can help troubleshoot the issue. As soon as I have any news, I will comment here again to provide the solution.

Bhargava-MSFT 28,956 Reputation points Microsoft Employee

2024-07-23T14:10:27.7266667+00:00

Izaura Domi

Thank you for the update.
Sign in to comment

Share via

Error Data Flow connection vs CosmosDb

1 answer