That means your graphics card/driver is having an issue. Standard suggestions apply here including making sure you are running the latest graphics driver and that you ensured the graphics card was seated properly. Note that for nVidia you should ensure you are running their Graphics driver software and not the version that Windows probably auto-installed.
You should also ensure that you aren't overclocking. If the system runs fine until the gfx card gets hit hard then it might be a cooling issue. Use a tool like CPUtemp to check the temps of your CPU. Gaming motherboards generally support doing this directly. If you install the nVidia Graphics Experience(?) then I believe it also has a tool to monitor GPU temps and whatnot. Right now it sounds like a cooling issue to me.
Lastly there is always the chance that your gfx card is bad. You'll need to work with the manufacturer and/or nVidia to get a replacement but I might recommend you start with their support forum. You should consider running a stress test on the gfx card instead of running a game. If it fails then it is almost certainly a cooling or bad gfx issue.