Set this to 2x (720p) or 3x (1080p) depending on your GPU.
Make sure you are using the latest Beta version of Dolphin, as the older Stable 5.0 version has known bugs with this specific title. download super mario galaxy for dolphin emulator
Use Vulkan or Direct3D 11/12 for the best stability. Set this to 2x (720p) or 3x (1080p) depending on your GPU