Delegates by definition are going to be slower than regular code. If you have to use them then go ahead but accept the perf issues. You should profile the code. Most likely the delegate calls are not slow at all compared to everything else. Unless you can identify an actual bottleneck then there isn't much you can optimize.
Personally I think you should consider an alternative design where you don't need a delegate. Delegates were useful in the past (and still can be) but honestly most code has moved on to better solutions including passing function arguments, using events or async/await. Delegates are not designed to be a catch all solution and other approaches may prove to be faster.
Without further understanding what problem you're trying to solve using delegates it is hard to recommend better approaches.