MongoDB Atlas connection in ADF fails

fortoajit 5

I am trying to connect to MongoDB atlas in ADF to ingest data from collection.

I am getting an error like below;

Cannot get required property with key 'collections' in Bson: { "db" : "workbench", "collections" : NumberLong(5), "views" : NumberLong(0), "objects" : NumberLong(61723), "avgObjSize" : 3262.6839427765985, "dataSize" : 201382641.0, "storageSize" : 33226752.0, "indexes" : NumberLong(8), "indexSize" : 3747840.0, "totalSize" : 36974592.0, "scaleFactor" : NumberLong(1), "fsUsedSize" : 14534578176.0, "fsTotalSize" : 21464350720.0, "ok" : 1.0, "$clusterTime" : { "clusterTime" : Timestamp(1703249788, 5), "signature" : { "hash" : new BinData(0, "GKRN0HThTvrUTK7KvSwXTw5oE7E="), "keyId" : NumberLong("7269158910697668610") } }, "operationTime" : Timestamp(1703249788, 5) }.

connection string

mongodb+srv://<USR>:<PWD>@<CLUSTER>.8ddkv.mongodb.net/

Atlas have open connection to the internet. means below like this added in network access

IP AddressCommentStatusActions0.0.0.0/0 (includes your current IP address)

User's image

Azar 19,645 Reputation points

2023-12-21T16:12:35.4+00:00

Hi fortoajit

can you provide a lil more about the error so that we can help you better.
phemanth 6,555 Reputation points Microsoft Vendor

2023-12-22T07:15:26.2266667+00:00

@fortoajit

Thanks for reaching out to Microsoft Q&A.

IP Access List: Before connecting to your Atlas database deployment, check that you’ve added your host’s IP address to the IP access list for your database deployment’s project. MongoDB Atlas allows client connections only from IP addresses and CIDR address ranges in the IP access list.

Authentication: To connect to Atlas, you must authenticate with a MongoDB database user. Check that you’re using the correct username and password for your database user, and that you’re connecting to the correct database deployment. If you have a special character in your password, see Special characters in connection string password.

Too many open connections: Atlas sets limits for concurrent incoming connections to a database deployment. If you’re at this limit, MongoDB displays an error stating connection refused because too many open connections. You can close any open connections to your database deployment not currently in use, scale your cluster to a higher tier to support more concurrent connections, or restart your application.

UUID Representation: In the MongoDB connection string, add the uuidRepresentation=standard option.

reference: https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-mongodb

https://www.mongodb.com/docs/atlas/troubleshoot-connection/

kindly provide more details about the error you are getting.
fortoajit 5 Reputation points

2023-12-22T12:59:49.7+00:00

Thank you guys for looking into it. Updated the description of error in the post again.
phemanth 6,555 Reputation points Microsoft Vendor

2023-12-22T15:46:33.6366667+00:00

@fortoajit does information provided by me helped?
fortoajit 5 Reputation points

2023-12-26T19:33:21.89+00:00

@phemanth unfortunately, not yet. It does not connect. do you know if there are any specific roles In user creation which must be also added. I am trying to connect my org cluster and it rejects, although it shows database but test connection does not work. I tried with my personal atlas account and it does work. wondering what could be missing.
phemanth 6,555 Reputation points Microsoft Vendor

2023-12-29T04:14:14.94+00:00

@fortoajit

MongoDB Atlas User Roles: MongoDB Atlas user roles define the actions that users can perform in organizations, projects, or both. The Project Owner role grants privileges to perform actions such as creating a Database Deployment and managing project access and project settings. Make sure the user you’re using to connect to MongoDB Atlas has the necessary permissions.

Azure Data Factory Permissions: To create Data Factory instances, the user account that you use to sign into Azure must be a member of the contributor role, the owner role, or an administrator of the Azure subscription.

Database Users vs Atlas Users: Database users have access to MongoDB databases, while Atlas users have access to the Atlas application itself. You can create a database user to access your Atlas database deployment.

Connection String: Ensure that your connection string is correctly formatted.
Travis Canady 5 Reputation points

2024-01-31T01:36:22.9866667+00:00
Wondering if you ever found an answer to this?

I have a Mongo DB server setup and no matter what I do I get this same connection issue.

Cannot get required property with key 'collections' in Bson: { "db" : "Harris", "collections" : NumberLong(1), "views" : NumberLong(0), "objects" : NumberLong(1738028), "avgObjSize" : 6829.3874005482076, "dataSize" : 11869666525.0, "storageSize" : 3629322240.0, "indexes" : NumberLong(1), "indexSize" : 36794368.0, "totalSize" : 3666116608.0, "scaleFactor" : NumberLong(1), "fsUsedSize" : 7438942208.0, "fsTotalSize" : 266206101504.0, "ok" : 1.0 }.

Server is MongoDB Community Edition on Ubuntu 22

I whitelisted all IPs (0.0.0.0)

I have tried the following and they do not work:

Mongo DB Connector

Added Params

authSource=admin

directConnection=true

uuidRepresentation=standard

uuidRepresentation=csharpLegacy

Changed the User Password to something simpler

Changed User Permissions to superuser

Turned authentication off on the MongoDB server so no user required

Use Atlas Connector got the same error

I have confirmed the server is open to the public and not blocking IPs

Spun up a self hosted Integration Runtime in case the issue was related to the default Azure one.

I don't think this is a connection issue or a auth issue because when I put the connection string in, the drop down shows the databases that are on the server so it is connecting and authenticating in order to get that list.

It could be a role thing though I would think turning auth of would allow it to get past that but it doesn't. In Mongo this user has the following roles.

roles: [ { role: 'readWriteAnyDatabase', db: 'admin' }, { role: 'root', db: 'admin' }, { role: 'userAdmin', db: 'Harris' }, { role: 'userAdminAnyDatabase', db: 'admin' } ],

I am at a loss on how to get this to work and this Q&A is the only result in google that matches my issue. Any help is greatly appreciated.
phemanth 6,555 Reputation points Microsoft Vendor

2024-02-01T09:24:18.8933333+00:00

Travis Canady could you please create a new thread regarding your issue.
fortoajit 5 Reputation points

2024-02-01T09:36:59.32+00:00

@Travis Canady I exactly did what you did and i can confirm that it still doesn't work. I have a instinct that maybe ADF doesn't connect very well to latest version of Mongodb. Btw, i was connecting to MongoDB atlas.
phemanth 6,555 Reputation points Microsoft Vendor

2024-02-02T15:27:46.7766667+00:00

@fortoajit checking with the internal team will get back to you shortly
Travis Canady 5 Reputation points

2024-02-05T16:34:54.2+00:00

@fortoajit I submitted a azure support ticket and was told that MongoDB < 6.1 is supported my server was setup as MongoDB 7.0. Because this was a new VM that was not yet in use I scrapped it and did a fresh install of MongoDB 6.0. Everything works fine for me now. ADF has zero issues connecting and moving data now. The support person mentioned that MongoDB 7.0 changes the way the drivers work which can cause some issues if using an older driver on a 7.0 server. Seems like ADF hasn't updated their driver and so when hitting my 7.0 server I got strange errors. Hope this helps.
DavidEdwards-9083 0 Reputation points

2024-05-01T07:11:59.85+00:00

I came across this same issue and after contacting Azure support, it is still the case that MongoDB v7 is not supported and there is no ETA for adding that support. I have submitted a feature request for MongoDB v7 support in ADF here if anyone else comes across this issue and would like to upvote it https://feedback.azure.com/d365community/idea/0a6d74a6-ff06-ef11-a73c-6045bd7e8707

Share via

MongoDB Atlas connection in ADF fails