I’ve been experimenting with making custom themes and have added a few dozen swatches to the color picker (Format → Color…). Now I would like to delete the swatches that I no longer need. Ideally, there’d be an option to delete all unused swatches, but I’d settle for removing them manually, if I could just figure out how. I’ve tried clicking-and-dragging the swatch out of the swatch menu, but that doesn’t work.
If you mean the little popover that has a grid of preselected colours on it, there is no way to edit that. But below the main grid, it does also show your system-wide custom colour swatches, as set up in the standard macOS colour palette (⇧⌘C) See figure 18.9 in the user manual PDF. That area at the bottom is where you can add and remove colours. And that is where clicking on the colour you want to remove and hitting the delete key works.