Strange Code Block formatting and character problem upgrading to Scrivener 3, EPUB and MOBI

thomasalmy · November 27, 2017, 9:28pm

First, let me show what the output should look like and does in Scrivener 2. As shown in Kindle Previewer 3:

This was with Monaco 10pt and “Keep Formatting”.
In Scrivener 3, I made a test project with the same text + formatting as in Scrivener 2, a second copy using the Code Span style, and a third using the Code Block Style:

If I compile to EPUB 3, I get the following (viewed in iBooks):

Note that in the embedded case that Monaco 10pt was not used (easy to see not fixed width font), first line of underscores becomes a solid line, some lines have been unexpectedly indented, and multiple spaces have been collapsed into single spaces, So I fail to get portability from Scrivener 2. For the Code Span and Code Block case I get the correct font but backslashes have been inserted before most underscores. Note that Code Span inserts one more backslash than Code Block does. Code Span removes consecutive spaces and does an unexpected indent. Code block, which should indent, doesn’t.

If I compile to MOBI (Kindle KF8) and view in Kindle Previewer I get (continued in next post, there seems to be a three image limit.)…

thomasalmy · November 27, 2017, 9:33pm

Compiling to Kindle KF8 MOBI I get:

In this case the fixed width font isn’t used at all, even for the Code Span and Block.

I’m running on macOS Sierra 10.12.6 with the latest (downloaded today) Kindlegen, Kindle Previewer 3, and iBooks.

Bridey · November 27, 2017, 9:45pm

Have you styled the text as required by ePub3 and KF8?

thomasalmy · November 27, 2017, 11:58pm

I used the “E-Book” format. But upon further investigation I determined there are several sources of the problems I’m seeing:

My fault - I can’t do this with the code span style because I lose the extra space characters. I’ll accept this. It is interesting that if I Compile to HTML, proper spacing is maintained, but that seems to be a feature in the HTML generator which is different (Scrivener) code than for generating E-Books.
Kindle Previewer fault and perhaps Scrivener fault as well – doesn’t display monospaced fonts processed from Scrivener 3 (it does from Scrivener 2). If I view the output in the Kindle application (for Mac) the monospace font is used. Also works on a Kindle paperwhite.
Scrivener fault – the horizontal rule and inserted backslashes. These show up in the HTML source output

Here’s the pertinent part of the generated xhtml file:

[code]

EMBEDDED MONACO 10PT:

|_|_|_|_|_|_|_|_|_|_|_|_|

| | | | |

| op |i|z| addr |

As Code Span:

_ _ _ _ _ _ _ _ _ _ _ \_

|\_|\_|\_|\_|\_|\_|\_|\_|\_|\_|\_|\_|

| | | | |

| op |i|z| addr |

As Code Block:

 _ _ _ _ _ _ _ _ _ _ _ _
|\_|\_|\_|\_|\_|\_|\_|\_|\_|\_|\_|\_|
|     | | |             |
|  op |i|z|    addr     |

No style.

[/code]

Scrivener 2 generated this for the Preserve Formatting:

[code]

_ _ _ _ _ _ _ _ _ _ _ _

|_|_|_|_|_|_|_|_|_|_|_|_|

| | | | |

| op |i|z| addr |

[/code] Class p6 sets the monospaced font. Note that nothing like this is done in the Scrivener 3 output for the first "Preserve Formatting" case.

thomasalmy · November 28, 2017, 5:02pm

I discovered that underscore followed by any character other than an underscore or space becomes a backslash followed by an underscore when used in a Code Block or Code Span.

I must add to the bullet list of bugs. Some I forgot, others new after doing more investigation. Basically “Preserve Formatting” does nothing for the new EPUB and MOBI formats.

I forgot that I don’t get monospaced font when I use “Preserve Formatting” in the first section (Embedded Monaco 10pt). This is when I carry over the text from Scrivener 2. If this worked then at least I could get by without having to make any changes. If I compile Epub 2 it all does work, but I guess I lose the advantages of the new Scrivener 3 compiler.
I tried “Preserve Formatting” around the Code Span. It makes no difference. Why doesn’t it add the
```
 tag?
```

KB · December 4, 2017, 11:39am

On the last two: monospace fonts will only be added for styles when using ePub 3. Preserve Formatting will not preserve monospace fonts - you must use styles for this.

You need to use the Code Block style (the style set as the “Code blocks style” in the Compile settings) to get the

 tag inserted around a block. (Code Span is for code inside a paragraph.)

KB · December 4, 2017, 12:09pm

The bug with underscores being incorrectly escaped in code blocks is fixed for the next update.

All the best,
Keith

thomasalmy · December 5, 2017, 6:15pm

Keith,

Is this somehow related to Markdown? Is it trying to do some Markdown conversion even though I don’t use it and haven’t checked the boxes?

In the example I show in the first post in this thread, the very first sequence of underscores got converted to a horizontal rule. It seemed to take this as a Markdown command. This was in the text carried over from Scrivener 2 where I did a “code block” by setting the font and marking the text as “Preserve Formatting.”

KB · December 20, 2017, 9:13pm

Sorry, I missed this. But yes, it is to do with MMD. In order to generated HTML5 (necessary for ePub 3 and KF8), Scrivener has to convert all its rich text to MultiMarkdown format and then pipe that through MMD to generate HTML5 - Apple only provides HTML4 converters and there are no other converters out there that will turn Apple’s rich text into HTML5. This is a hugely complex process and is the source of most of the bugs in the ePub 3 and KF8 export, as edge case areas I had not anticipated show up and need to be fixed. (ePub 2 and Mobi in Scrivener 2 had similar issues at first, because even though they use HTML4 and Apple can export to HTML4 out of the box, the HTML4 support in the text system is not great for ebooks and so required a lot of hack-like patching up in code. Ultimately, the MMD > HTML5 solution is cleaner and will result in better, nicer HTML and ebooks.)

All the best,
Keith