A custom image is a managed image. A formula is an image that you can configure with additional settings, and then save and reproduce.
A custom image might be preferable if you want to quickly create several environments by using the same basic, immutable image.
A formula might be better if you want to reproduce the configuration of your VM with the latest bits, as part of a virtual network or subnet, or as a VM of a specific size.
For a more in-depth explanation, see Comparing custom images and formulas in DevTest Labs.