You are right storing a large number of versions (500+) can impact the performance of backup operations or getting an error but unfortunately it is not possible to keep last three(less or more) versions of one key vault secret.
It seems there are no plans to support deletion of individual versions of a secret in nearest future.
As you are planning to add new version of each secret once a week I would recommend you to recreate Azure KeyVault Secret once a year.
If you think your question has been answered, click "Mark as Accept Answer" if just helped click "Vote as helpful". This can be beneficial to other community members reading this forum thread.