First, Movie Maker never touches your original source files, so it can never "delete" the audio from your video. All that Movie Maker can do is create a new movie from other videos, photos and music.
So if your question is can you remove (mute) the original audio and replace it with music and then save that as a new movie, the answer is yes.
To mute the original audio, double-click on the video clip to get to the Video Tools tab. Set the Video volume to zero. Now you can do whatever you want with added audio such as adding music, sound effects, etc.
Another potential way to do this, if you don't want the original audio from any of your original video clips is to go to the Project tab and set the Audio mix. Normally it is set in the middle between your original audio and the added audio. If you only want to hear added audio, set it all the way to the right.
The first approach is useful if you want to only mute the audio from certain clips in your project. The second approach is useful if you know you want to do it completely for all clips in the project, not just some of them.