Hm, I haven’t tested it with annotations, but this sounds like the same bug I reported with the footnotes.
For the annotation color, you should be able to get something mid-range by choosing say a pale gray in the Appearance colors. Also you can change the “ghost” colors the same way.