System Refresh is a new feature available in Windows 8 and above, that lets you reset your computer to factory settings easily and quickly. The refresh feature in itself isn't harmful, but if you've had to refresh your PC lots of time, there could be other problems with the system.
System Refresh is just an advantageous feature that you can use as a last resort, or when the system is damaged beyond repair (in terms of software). So, it's not System Refresh that is harmful, but you'll have to consider the factors that are causing you to do a refresh so often.
If you've had to do it because the performance is slow, and you face the same issue even after refresh, it could be something that you're installing, like a software application or even a hardware device. Or it could be that there's some issue with your hard drive. Those are just a few possibilities based on assumptions. But System Refresh will just get you back to factory settings.