It appears this exact thing happens also when viewing PDFs in the editor and as a copyholder. I first thought it was just in the inspector, but now it turns out it is a widespread problem. The only way to view a PDF without encountering this problem is when using quick reference.
I hope the developers will look at this. Viewing PDFs while writing is a big part of Scrivener. Since the zoom level you choose for a PDF remains, it should be possible to make the position also remain the same when you return to it.