Conversion problems from ePub3 to Mobi on Kindle Touch (4th gen)

Scrivener Scrivener for macOS
1

Hello,

First I want to say even though I am having to relearn everything in regards to compile settings, from what I’ve seen, the changes are for the best and once I get used to them I see that compile is much better.

However I am having problems with something, and I’m not sure if it is a bug with Scrivener 3’s encoding or of it’s just because I have an older kindle, or perhaps it is with Kindle Previewer 3 that I use to convert epub to mobi.

The main problem is that my smart quotes, apostrophes, and my copyright symbol are all appearing as odd characters in my MOBI. The steps I have are as follows:

  1. Compile to ePub 3
  2. Open epub in Kindle Previewer 3 (newest version as far as I can tell)
    At this point, everything looks fine in the kindle previewer!
  3. Export to mobi
    All good so far! I can even open this MOBI up in Kindle Previewer and it looks perfect.
  4. Transfer MOBI to Kindle Touch 4th Gen via Send To Kindle App
  5. Open on Kindle-- at this point is where the odd characters show up. They don’t show up on Kindle Previewer EVEN when opening the MOBI rather than the EPUB. But once it’s on my Kindle, it’s messed up. As far as I an tell from a cursory look, it is the apostrophes, quotes, and copyright symbol that show up as strange characters. Otherwise things seem to be in order, though I haven’t read all the way through, only skimmed for problems.

I know that there is a MOBI compile setting, but I had everything worked out so well on the epub3 and I didn’t want to have to switch. Now, I should note that I have converted other epubs to MOBI and sent to my kindle this way with no conversion problems before, but those epubs were not compiled by scrivener3. They may have been in epub2 format, though I’m not 100% (I got the files from a book formatter guy I contract out for my novels–doing a novelette right now and trying to save money by formatting myself). Is the problem just that I should be using epub2? If so, I thought that epub3 was backward compatible and I’ve read that the encoding should all be fine coming from scrivener3 to not have this problem. What’s up? What am I missing? Thanks!

2

I think this may be related to my problem - I’ve had an issue with the Amazon version of my book being screwed. My publisher’s just come back with something saying that I have quote characters in my code instead of HTML escapes and that I need to convert to HTML escape codes accordingly. I’ve literally just posted to my previous thread to ask if this is an issue with Scrivener and ePub3’s output rendering … I think we have the same problem.

3

Another thing I noticed that isn’t coming out quite right, I think are my em dashes (—). I noticed this because when I tried replacing smart punctuation with dumb punctuation, it did make my apostrophes and quotes appear straight without the weird characters, but it didn’t fix the issue with the copyright symbol and it changed my em dashes to double hyphens (–). This led me to recompile without changing smart punctuation to straight to see if that was also a problem, and that’s when I noticed that sure enough, the em dashes were also appearing as weird symbols.

While I might be tempted to settle for straight quotes and remove the copyright symbol, I can’t settle for double hyphens instead of em dashes.

4

Problem SORT OF fixed

Okay so I was trying to figure out the whole custom start page problem while waiting for an answer to this and accidentally stumbled on a fix for at least this individual book.

After using Sigil to edit a line of code in my epub’s content.opf file to add a text semantic to a specific page, I saved the epub, did the old kindle preview export, then send to kindle. Wonder of wonders, imagine my surprise to see that not only did the start page conform to my designation in the opf, but all the weird characters were gone! My only explanation for this is that somehow in slightly editing and saving the epub in Sigil, Sigil is re-encoding to the proper standard. Other than that I have no idea what happened. So the problem is sort of fixed, but not, as it was Sigil that saved me, not Scrivener.

5

Thanks - I’d never heard of Sigil - I’ll have a look at that. Perhaps it might resolve my issue too :confused:

6

Sigil is a free ebook creation tool. I find it very useful for editing little things but I don’t think I would write an entire ebook in it from scratch. However, something to keep in mind: I got Sigil because I was asking my previously mentioned formatter how I might tweak various things in my ebooks on my own without having to contact him and send files back and forth, and he recommended the program for me, but he made me do one thing before opening any of myprojects. I’ll just copy and paste those instructions below:

The main advice in there I think you should note is making sure that “Mend XHTML Source Code” is off. I don’t know why, but it sounded important, so I would heed that advice before opening up your ebook in it.

7

Thanks, noted. Heads off to check settings …

8

First, I would really, really strongly recommend exporting to KF8/Mobi rather than Epub 3 if your target is an Amazon device. I’m not sure what you mean about not wanting to do this because you already have the settings adjusted for Epub 3 - this shouldn’t be a problem given that Epub 3 and KF8/Mobi have nearly the same settings, so if you sent up one, there should be little (if anything) to change for the other.

It’s important to understand that Amazon Kindle’s rules for HTML and CSS, for instance, are slightly different from those of Epub 3. The Kindle supports fewer CSS rules, and Scrivener’s export takes this into account, altering the HTML as necessary to ensure that formatting that would not work when converted from Epub to Kindle will work when going through the KF8/Mobi route. So, that’s the first thing to bear in mind: if you are exporting for use on a Kindle, you should almost always export to KF8/Mobi format.

Also, most importantly: if you do things this way, you will not experience the garbled characters problem that you are seeing. This is because the problem you are encountering lies with Kindle Previewer - not with Scrivener.

Your information about Sigil seeming to “fix” the problem was very useful, because it gave me the necessary clue as to what was going on. From MIke’s previous report, I had already gathered that there must be some sort of encoding issue going on: HTML5 files (which are used in Epub and Kindle files) are UTF-8, which means that characters such as curly quotes and copyright symbols can just be entered into the HTML code as they are. They do not require escaped sequences such as ©, &ldquo, &rdquo and so on, as was once necessary. However, if you open a UTF-8 file in an ASCII reader, UTF-8 characters (such as curly quotes, copyright symbols and so on) come out as garbage characters - exactly what you have both encountered.

In other words, it seems that the Kindle is interpreting the HTML files in the Mobi file generated by Kindle Previewer as ASCII files when they are in fact UTF-8, and thus garbling them. This shouldn’t happen, given that Epub 3 requires HTML5 and HTML5 is UTF-8. And, in fact, it doesn’t happen if you just export directly from Scrivener to KF8/Mobi format - in that case, all characters look fine on an old Kindle device. So the problem seems to be down to a bug in the way Kindle Previewer converter Epub files (the bug does not seem to be present in Amazon’s KindleGen utility, which Scrivener uses to generate Kindle files, though).

Looking at the Epub file generated by Sigil when you open a Scrivener-generated Epub file in it and reserve, I noted that it puts the following line at the start of each HTML document:

<?xml version="1.0" encoding="utf-8"?>

Now, this shouldn’t be necessary, since HTML5 files are version 1.0 and UTF-8 by default - and any reader handling such files should know this. But it seems that Kindle Previewer’s conversion code isn’t building in the correct defaults and so requires this line in the Epub 3 file to convert correctly.

I have therefore added this line to the start of XHTML files inside Epub 3 files created by Scrivener for the next update (3.0.3). This will ensure that Epub files created by Scrivener and converted in Kindle Previewer will work fine on Kindle devices.

However… Despite this fix, it is still far better to use Scrivener’s own KF8/Mobi exporter. Because of the differences between Kindle and Epub HTML specs, you will always see a better result going this way. Also, if you export as Epub and convert, the book will always open at first to the front matter rather than to the first chapter page, because Epub has no concept of a “book start” marker as Kindle does.

All the best,
Keith

9

Keith,

Thanks for the info! I don’t know the real reason for my hesitation for using the KF8/mobi compile in Scrivener other than I guess up until now I have been dealing with epubs others have made for me and then converting them through kindle previewer, so in using Scrivener to compile myself I naturally assumed I should start with epub. No good reason, in other words, but ignorance and weird assumptions. In regards to what I set about having it already set up in epub3, like I said before I was struggling to re-learn how to use the Scrivener 3 compile, and it took me a long time to figure out how to get the format settings exactly how I liked them (regarding text separators and chapter headings mostly) and once I had it all done in my custom epub3 section layout settings, I thought (apparently mistakenly, now that I look at it) I would have to go back and do it all over again to compile as kf8/mobi, if that makes any sense. Sorry like I said I’m new to the ebook conversion capabilities of Scrivener, and new to S3’s compile settings in general. Anyways at least in all my weird workarounds it brought to light something to put in a future update, so I guess it wasn’t all that bad

10

Ah, yes, sorry, I forgot to take that part into account, because of course Scrivener’s Compile settings have changed so much since 2.x that they do take a lot of re-learning. I’m glad you are seeing the benefits of the changes, though, now that you are getting used to them!

Essentially, Epub 3 and KF8/Mobi use much the same code, so if you set up one there will be very few tweaks for the other. Likewise with Epub 2 and Mobi, whereas setting up Epub 3 won’t affect Epub 2 much because the settings for those are quite different. (In fact, I’ll be dropping support for Epub 2 and Mobi at some point and keeping only Epub 3 and KF8/Mobi, but I wanted to keep the older formats for a transitional period so that old Scrivener 2 settings could be imported for a while.)

So, Epub 3 and KF8/Mobi use a lot of the same code to generate them, but with some internal tweaks specific to the platform. (For instance, you cannot centre-align a table of contents for the Kindle using straight CSS, but have to use an HTML tag - there a bunch of little things like that.) I suppose I could add an option to the Epub settings to generate the files optimised for Kindle conversion, given that it seems a few publishers require Epub files…

All the best,
Keith

11

Thanks for the replies - really useful. In my case, I didn’t have the option to go Mobi or ePub - I can only provide Ingram with the ePub format as their upload portal rejects anything else.

In the interim (as the book’s live), is it just a case of unzipping, adding the encoding declaration and then re-zipping, or do I need to do something else too? (Or running through Sigil - I’ve run the files through and seen the inserted encoding directive)

Ingram asked for information about the generator used so that they can try to insulate against this problem in the future - I’ve pointed them to this thread.

Thanks for letting me hijack the thread too - but I think it’s the same issue, so I’ll live with my sinful ways …

12

Mike, all I did was opened the epub in Sigil, then I made a small change (one line of code) to a part of the book that had nothing to do with the problem, then saved it. So I’m assuming that Sigil automatically took care of it when I saved it. So if I were you, I would try opening the epub, going to the html code in part of it, then doing something inconsequential that will make you able to save changes, such as putting a space or empty line somewhere without disrupting the code. Then save changes. And if you are nervous you might have messed up the code that way, just undo what you did and save it again. My theory is that somewhere in the background, even though you aren’t doing anything yourself to change the epub’s html, Sigil is adding that utf thing to it when you save changes or something.

Keith, if you’re still following this, a minor thing I noticed, when I compiled into KF8 /Mobi from scrivener, regarding the start page, for some reason it opened up on my kindle with the end (like last line or two) of my copyright page at the top, which was really weird. Not sure what that means except that I know when I edit an epub with the right semantics tags where I want it and convert to mobi from there, it goes where I want. So I don’t know if that is something you want to know about but thought i would mention it.

13

This is most likely because you are creating an Epub file and then converting to Kindle format. Amazon has a way of specifying the start position for a book using a “Start Here” item in the OPF ( is actually deprecated in Epub 3 but is usually still provided for backwards-compatibility purposes). This is only output when you generate a Kindle file. So at the moment, if you go through Epub, then there is no way to set the start position (and thus the “Book begins after front matter” option is not available for Epubs as it is for KF8 files).

However, I note that Epub 3 does have a “bodymatter” setting for this for the landmarks element that can be added as a hidden element in the ToC navigation file. I’ve therefore added this for the next update. I’ve also added an “Optimize for Kindle conversion” checkbox to the Epub 3 options. If you tick this, Scrivener will use the same HTML and CSS tricks it uses for creating a Kindle file so that it stays within the (extra) limits of Amazon’s specs, and it also includes the “Start Here” tag in the . Also, the “landmarks” element has been added, although I’ve also added a checkbox to omit that, as I have read that even hidden landmarks can sometimes end up being shown in the “Look Inside” feature, thanks to Amazon not having updated the code for that for years.

All the best,
Keith