For some reason, specifying the exact version seems to be working now. This might be because I now installed preview version of VS which supports .NET 5 RC.
This code worked:
xml
<ItemGroup>
<PackageReference Include="Microsoft.EntityFrameworkCore">
<Version Condition="'$(TargetFramework)' == 'netstandard2.0'">[2.0.0,5.0.0)</Version>
<Version Condition="'$(TargetFramework)' == 'netstandard2.1'">5.0.0-rc.1.20451.13</Version>
</PackageReference>
</ItemGroup>