If you're referring to the recent list then it should be small.
Can you use the Registered Servers
view and add your servers (grouped if desired) instead? We have a bunch of DBs as well and I have separate registered servers grouped by my needs. The registered server has the credentials (if desired) and all the other connection info. The nice thing about it is you can copy the file it stores this in and put it on another machine (such as when you get a new machine) rather than having to rebuild everything.