Hello,
Welcome to Microsoft Q&A.
What you see is the style change of the Button in the PointerOver state.
In the default control template, it behaves as follows:
13122-defaultbuttonpointeroverstate.txt
If you only need Button not to display Border, you can set it like this:
<Button BorderThickness="0"
... />
If you need to modify the different behavior of Button in each state, you can refer to this document:
The complete Button default style is defined here:
13194-defaultbuttoncontroltemplate.txt
Thanks.