Hiidden code in Scrivener text

Trying to compile for epub on the Nook. I keep getting some kind of weird text-code incompatibility, which produces small boxes with an X in it.

I’ve upload the compile to the Nook Press site, the translation is fine there. Problem is they don’t let you download your manuscript once it has been translated. When I drag the Scrivener-made ePub directly to the Nook is when I get the boxes with Xs in them.

I tried using Calibre with text encoding cp1252. No love.

At first I thought it might be a problem with smart quotes. I turned this off for the Scrivener document, as well as clicked the remove smart quotes in compile, that seemed to remove maybe 50% of the boxes with Xs. But they persisted elsewhere.

On closer scrutiny – finding the problem text on the Nook, and then examining it in Scrivener – I find that there are a series of invisible spaces with which must have some kind of code in them. In other words, I put the cursor in front of the problem area and use the forward arrow to step forward one character/space – at this problem space, the step forward skips a beat and doesn’t move forward, a second arrow-forward moves beyond the problem space/non-space (and its hidden code).

Not seeing a reveal hidden or formatting command in Scrivener, I copied a line on either side of the problem code and pasted it into Word. It immediately revealed what Scrivener had been hiding, every instance of my problem was a hidden smart quote or apostrophe mark (now visible in Word).

Hmmm… a legacy of some errant text from the Neolithic era? Scrivener hiding smart quotes in its editor, but not removing them? (Maybe from a much earlier version?)

I tried searching in Scrivener on two quotations marks. Didn’t work. I tried highlighting one of the errant smart quotes I had revealed in Word. Nada. I tried searching on unicode I found for smart quotes on the web. Again, failure.

Any suggestions on how to remove this hidden code before exporting to ePub?

I’ve never heard of such a thing, could you send a snippet to support so we can take a look at it (or attach it here)? To answer your question, Scrivener has never produced hidden punctuation to my knowledge, and I’ve never even heard of such a tactic being used by any software. But like I say, it’s hard to give any advice until I know what precisely is going on in the text. I would just export the file from Scrivener and trim it down if necessary in TextEdit. That should preserve the problem.

The errant quote/non-quotes are around “Murder Alley” in the snippet below:

People called it “Murder Alley” because of the number

Unfortunately that won’t work because your web browser (and maybe even the forum software) very likely deleted the formatting/unicode control characters/whatever glitch is going on, since the text box for typing in a response in plain-text. Like I say, the best way to preserve the glitch over the Internet will be to select a file in the binder that has the problem, hit File/Export/Files..., and then trim it down however much you want to using TextEdit (which uses the same text engine that Scrivener does, meaning it will likely pass on the error after saving, whereas another program like Word or OpenOffice might clean it out) and then attach that as a file to either a response here or to our support e-mail.

Well, I posted my last response to you, viewed my post in Safari, highlighted the snippet and copied it, then posted it in TextEdit and the problem was still there. Then I posted that same snippet in Word and it revealed the smart quotes around Murder Alley that were not there in the Scrivener version.

No matter, I will email a file to Support.

Ah, I see it now. I had to use Safari for some reason. Okay, since the invisible character can be selected, what I would recommend is putting your cursor next to it, and then holding down the Shift key and arrowing over to select it. You won’t see anything in the editor, but if you check the footer bar it should be blue and say “Chars: 1”. Hit Cmd-C to copy it and paste it into Edit/Find/Project Replace. For the replacement type in the curly quote you wish to use (Opt-[ or Opt-] on my keyboard) and replace all. That should wipe out the invisible character and replace it with the correct quotation mark. Then do the matching one on the other side in the same fashion.

I ran a test with Scrivener on the sample text, and it cleaned up all instances. Needless to say you should probably make a backup for doing this, just in case.

Curiouser, and curiouser… I tried the project replace as you suggested above, putting nothing in the (Replace) With box, as I wanted to simply remove the hidden smart quotes. When I clicked Replace it either hung and did nothing, or crashed the project. This was on a duplicate of my original project.

By mistake I then tried this process on my original project. It worked fine, and I did it three times to find and remove hidden open and close smart quotes, and smart apostrophes. I have just ported it to the Nook via ePub compile and a quick glance suggests it worked fine. I will later go through the whole book on the Nook to make sure there are no remaining glitches.

Thanks for your help… not sure what to say about the crashes, other than the reports I sent you.

Yeah, that’s weird that it would crash in the one project but not the other. Perhaps there was some other cascading problem that existed in the running version, and once it crashed that condition naturally vanished. Anyway, glad you got it sorted out.