Bot service with user assigned identity

I'm currently trying to set up a new teams bot with user assigned identity and while I can retrive messages I can't reply back.
I have created a new Azure Bot service in azure, set it to UserAssignedMSI and I have managed to add it to teams. If I send something to the bot I can also see that the methods like OnTurnAsync and OnMessageActivityAsync are triggered so everything looks good so far.
But the moment I try to send something back, like for example:
protected override async Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken)
{
await turnContext.SendActivityAsync(MessageFactory.Text("hello"), cancellationToken);
await base.OnMessageActivityAsync(turnContext, cancellationToken);
}
It crash with the following:
System.ArgumentNullException: Value cannot be null. (Parameter 'clientSecret')
at Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential..ctor(String clientId, String clientSecret)
at Microsoft.Bot.Connector.Authentication.MicrosoftAppCredentials.<BuildAuthenticator>b__16_0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at Microsoft.Bot.Connector.Authentication.AppCredentials.<BuildIAuthenticator>b__36_0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at System.Lazy`1.get_Value()
at Microsoft.Bot.Connector.Authentication.AppCredentials.GetTokenAsync(Boolean forceRefresh)
at Microsoft.Bot.Connector.Authentication.AppCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Bot.Connector.Conversations.ReplyToActivityWithHttpMessagesAsync(String conversationId, String activityId, Activity activity, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Bot.Connector.ConversationsExtensions.ReplyToActivityAsync(IConversations operations, String conversationId, String activityId, Activity activity, CancellationToken cancellationToken)
at Microsoft.Bot.Builder.BotFrameworkAdapter.SendActivitiesAsync(ITurnContext turnContext, Activity[] activities, CancellationToken cancellationToken)
at Microsoft.Bot.Builder.TurnContext.<>c__DisplayClass31_0.<<SendActivitiesAsync>g__SendActivitiesThroughAdapter|1>d.MoveNext()
\--- End of stack trace from previous location ---
at Microsoft.Bot.Builder.TurnContext.SendActivityAsync(IActivity activity, CancellationToken cancellationToken)
at iPMC.Autotest.DevOps.Bots.Bots.AutotestBot.OnMessageActivityAsync(ITurnContext`1 turnContext, CancellationToken cancellationToken)
And I'm note sure why. According to the documentation this should be enough in my appsettings.json when using user assigned identity (AVALUE is of course my real values):
"MicrosoftAppType": "UserAssignedMSI",
"MicrosoftAppId": "AVALUE",
"MicrosoftAppTenantId": "AVALUE",
"MicrosoftAppPassword": "",
"ConnectionName": "AVALUE"
It's seems like most examples use password (single/multi tenant) as well so I can't really find anyone else that have used this.
I have also tried to both do it locally and deployed but I get the same exception on both places so I'm running out of ideas what I should test next.
Anyone else that have used UserAssignedMSI with teams bots and got it to work?