@Matthias Straßer Thank you for reaching out to us, As I understand you are looking for Custom authentication extension option, you can see this option under the enterprise application blade, not under the application.
Also you had a query related to custom security attributes option being greyed out, refer to this troubleshooting guide for similar issue - https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/custom-security-attributes-troubleshoot
Let me know if you have any further questions, feel free to post back.
Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.