Mark word on double click marks smart quotes as well

If I want to edit or delete a single word, I use to double click it to mark it.
If this word has a smart quote before or after, the quotation mark is marked as well. If I use normal quotes, this doesn’t happen.
This is very annoying behaviour, because I have to use the mouse to move over it to mark it, which costs a lot more time than a simple double click.