The cost for Arc-enabling a Windows Server or SQL Server is miniscule. I don't think Azure Arc itself comes with a cost, but I seem to recall that you need to set up some secrets in a key vault, and the key vault comes with a very small cost.
Once you have Arc-enabled you machine, you add various functionality to it, for instance Microsoft Defender, and these add-ons can be pricier. But they are not mandatory.
You will also need to set your license type; one option here is Pay-as-you-go. I seem to recall that some services are only available if you have Sofware Assurance; but I don't recall if this applies to Entra ID.
In any case, to use Entra ID with SQL Server instance, you need to Arc-enable it.