Okay, at this point I would go into settings, and under General: Warnings, enable the Show internal error alerts option. If something breaks badly enough while trying to use colours, you’ll get a message with some diagnostic data to share.
So try it with the Labels tab in Project Settings, as you have been, and look for messages.
Next, does it open for other things that might want to “attach” themselves to the palette? A few similar places in Scrivener would be in the Project ▸ Project Keywords panel, and the various colour settings scattered throughout the Appearance settings tabs.