Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,624 questions
MongoDB Atlas connection in ADF fails
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 73%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EF%3C/text%3E%3C/svg%3E)
fortoajit
5
Reputation points
I am trying to connect to MongoDB atlas in ADF to ingest data from collection.
I am getting an error like below;
Cannot get required property with key 'collections' in Bson: { "db" : "workbench", "collections" : NumberLong(5), "views" : NumberLong(0), "objects" : NumberLong(61723), "avgObjSize" : 3262.6839427765985, "dataSize" : 201382641.0, "storageSize" : 33226752.0, "indexes" : NumberLong(8), "indexSize" : 3747840.0, "totalSize" : 36974592.0, "scaleFactor" : NumberLong(1), "fsUsedSize" : 14534578176.0, "fsTotalSize" : 21464350720.0, "ok" : 1.0, "$clusterTime" : { "clusterTime" : Timestamp(1703249788, 5), "signature" : { "hash" : new BinData(0, "GKRN0HThTvrUTK7KvSwXTw5oE7E="), "keyId" : NumberLong("7269158910697668610") } }, "operationTime" : Timestamp(1703249788, 5) }.
connection string
mongodb+srv://<USR>:<PWD>@<CLUSTER>.8ddkv.mongodb.net/
Atlas have open connection to the internet. means below like this added in network access
IP AddressCommentStatusActions0.0.0.0/0 (includes your current IP address)
Azure Data Factory
{count} vote
-
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth 15,755 Reputation points Microsoft External Staff Moderator2023-12-22T07:15:26.2266667+00:00 Thanks for reaching out to Microsoft Q&A.
IP Access List: Before connecting to your Atlas database deployment, check that you’ve added your host’s IP address to the IP access list for your database deployment’s project. MongoDB Atlas allows client connections only from IP addresses and CIDR address ranges in the IP access list.
Authentication: To connect to Atlas, you must authenticate with a MongoDB database user. Check that you’re using the correct username and password for your database user, and that you’re connecting to the correct database deployment. If you have a special character in your password, see Special characters in connection string password.
Too many open connections: Atlas sets limits for concurrent incoming connections to a database deployment. If you’re at this limit, MongoDB displays an error stating connection refused because too many open connections. You can close any open connections to your database deployment not currently in use, scale your cluster to a higher tier to support more concurrent connections, or restart your application.
UUID Representation: In the MongoDB connection string, add the uuidRepresentation=standard option.
reference: https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-mongodb
https://www.mongodb.com/docs/atlas/troubleshoot-connection/
kindly provide more details about the error you are getting.
-
fortoajit 5 Reputation points
2023-12-22T12:59:49.7+00:00 Thank you guys for looking into it. Updated the description of error in the post again.
-
phemanth 15,755 Reputation points Microsoft External Staff Moderator
2023-12-22T15:46:33.6366667+00:00 @fortoajit does information provided by me helped?
-
fortoajit 5 Reputation points
2023-12-26T19:33:21.89+00:00 @phemanth unfortunately, not yet. It does not connect. do you know if there are any specific roles In user creation which must be also added. I am trying to connect my org cluster and it rejects, although it shows database but test connection does not work. I tried with my personal atlas account and it does work. wondering what could be missing.
-
phemanth 15,755 Reputation points Microsoft External Staff Moderator
2023-12-29T04:14:14.94+00:00 MongoDB Atlas User Roles: MongoDB Atlas user roles define the actions that users can perform in organizations, projects, or both. The Project Owner role grants privileges to perform actions such as creating a Database Deployment and managing project access and project settings. Make sure the user you’re using to connect to MongoDB Atlas has the necessary permissions.
Azure Data Factory Permissions: To create Data Factory instances, the user account that you use to sign into Azure must be a member of the contributor role, the owner role, or an administrator of the Azure subscription.
Database Users vs Atlas Users: Database users have access to MongoDB databases, while Atlas users have access to the Atlas application itself. You can create a database user to access your Atlas database deployment.
Connection String: Ensure that your connection string is correctly formatted.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 88%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETC%3C/text%3E%3C/svg%3E)
Travis Canady 5 Reputation points2024-01-31T01:36:22.9866667+00:00 Wondering if you ever found an answer to this?
I have a Mongo DB server setup and no matter what I do I get this same connection issue.
Cannot get required property with key 'collections' in Bson: { "db" : "Harris", "collections" : NumberLong(1), "views" : NumberLong(0), "objects" : NumberLong(1738028), "avgObjSize" : 6829.3874005482076, "dataSize" : 11869666525.0, "storageSize" : 3629322240.0, "indexes" : NumberLong(1), "indexSize" : 36794368.0, "totalSize" : 3666116608.0, "scaleFactor" : NumberLong(1), "fsUsedSize" : 7438942208.0, "fsTotalSize" : 266206101504.0, "ok" : 1.0 }.
Server is MongoDB Community Edition on Ubuntu 22
I whitelisted all IPs (0.0.0.0)
I have tried the following and they do not work:
- Mongo DB Connector
- Added Params
- authSource=admin
- directConnection=true
- uuidRepresentation=standard
- uuidRepresentation=csharpLegacy
- Changed the User Password to something simpler
- Changed User Permissions to superuser
- Turned authentication off on the MongoDB server so no user required
- Added Params
- Use Atlas Connector got the same error
- I have confirmed the server is open to the public and not blocking IPs
- Spun up a self hosted Integration Runtime in case the issue was related to the default Azure one.
I don't think this is a connection issue or a auth issue because when I put the connection string in, the drop down shows the databases that are on the server so it is connecting and authenticating in order to get that list.
It could be a role thing though I would think turning auth of would allow it to get past that but it doesn't. In Mongo this user has the following roles.
roles: [ { role: 'readWriteAnyDatabase', db: 'admin' }, { role: 'root', db: 'admin' }, { role: 'userAdmin', db: 'Harris' }, { role: 'userAdminAnyDatabase', db: 'admin' } ],I am at a loss on how to get this to work and this Q&A is the only result in google that matches my issue. Any help is greatly appreciated.
- Mongo DB Connector
-
phemanth 15,755 Reputation points Microsoft External Staff Moderator
2024-02-01T09:24:18.8933333+00:00 Travis Canady could you please create a new thread regarding your issue.
-
fortoajit 5 Reputation points
2024-02-01T09:36:59.32+00:00 @Travis Canady I exactly did what you did and i can confirm that it still doesn't work. I have a instinct that maybe ADF doesn't connect very well to latest version of Mongodb. Btw, i was connecting to MongoDB atlas.
-
phemanth 15,755 Reputation points Microsoft External Staff Moderator
2024-02-02T15:27:46.7766667+00:00 @fortoajit checking with the internal team will get back to you shortly
-
Travis Canady 5 Reputation points
2024-02-05T16:34:54.2+00:00 @fortoajit I submitted a azure support ticket and was told that MongoDB < 6.1 is supported my server was setup as MongoDB 7.0. Because this was a new VM that was not yet in use I scrapped it and did a fresh install of MongoDB 6.0. Everything works fine for me now. ADF has zero issues connecting and moving data now. The support person mentioned that MongoDB 7.0 changes the way the drivers work which can cause some issues if using an older driver on a 7.0 server. Seems like ADF hasn't updated their driver and so when hitting my 7.0 server I got strange errors. Hope this helps.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 27%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED9%3C/text%3E%3C/svg%3E)
DavidEdwards-9083 0 Reputation points2024-05-01T07:11:59.85+00:00 I came across this same issue and after contacting Azure support, it is still the case that MongoDB v7 is not supported and there is no ETA for adding that support. I have submitted a feature request for MongoDB v7 support in ADF here if anyone else comes across this issue and would like to upvote it https://feedback.azure.com/d365community/idea/0a6d74a6-ff06-ef11-a73c-6045bd7e8707
Sign in to comment
Sign in to answer