dialog box, dialog, dialogue
In general, avoid talking about UI. Instead, talk about what the customer needs to do.
When you need to refer to a dialog box, use dialog. Don't use pop-up window, dialog box, or dialogue box.
See also Describing interactions with UI, Formatting text in instructions