I think the FAQ does a good job of explaining the differences here and which option you should choose. A quick summary:
- Per device means you may have 1 user connected for each license at the same time. It doesn't matter who that user is, they get 1 CAL. So if you paid for 30 CALs then up to 30 people can be connected at any one time. Basically you're leasing a parking place on the server. This doesn't require any AD membership.
- Per user means each user must have their own license. When they connect they use the license associated with their account. It doesn't matter how many people are connected provided each one has their own CAL. Basically each user has a designated parking spot that no one else can use. This requires AD so when the user logs in their login is tied to the CAL.
In a normal situation you'd use per user (which is generally cheaper) if you have a fixed set of users who all might be connecting. You would need 1 CAL per user. If you have users who only occasionally need to connect then you are wasting money.
Per device is better when you have a large number of users who may connect at some point but not all at once. This tends to be cheaper since you only need enough licenses for the max # of users who might connect at any one time. For example if you have 100 users who might need to connect but only 20 at any one time will then you can pay for 20 CALs. That saves you the cost of 80 CALs but the price is slightly higher per CAL (I believe).
In your case you said you don't have AD so per device is your only choice. You need to purchase enough CALs to cover the # of users who would be connected at any one time. If you suddenly need more CALS then you'd need to buy them which allows you to "ramp up" as needed.
There is no technical limit on RDS other than the physical limits of the server itself. Each RDS session takes up resources so if your machine has 2 cores and 8 GB of memory then it isn't going to be able to handle 100s of users at the same time. The recommendation is to monitor your server resources and if you are consistently at 80+% utilization then it may be time to throw more CPUs/memory/storage at it.