Hi Suruthi Iyyappan,
Thanks for reaching out.
Could you please try adding "new claim" set from "User Attributes & Claims" section from Azure AD Enterprise Application as shown below:
Steps:
1.Under User Attributes & Claims, click the edit symbol
2.Click Add new claim
3.Write displayname in the Name field
4.Leave Namespace as empty
5.Choose Attribute as Source
6.Select user.userprincipalname as Source attribute and Save it.
Here is an example of UPN claim in SAML token. Hope this helps.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.