MultiMarkdown to e-book ... or not

[jmwzht479]

I was excited about Scrivener in part because it (1) supported MultiMarkdown (MMD) and (2) created beautiful e-books. It’s only now occurring to me that it doesn’t do both at once.

Yes, technically, it can. It can create HTML files, and there are lots of ways to create .epub files from HTML (including Fletcher Penney’s MMD-ePub “script”). One can then use kindlegen to turn the .epub file into a .mobi file. One could even automate this process, perhaps by hijacking one of the Contents/Resources/MultiMarkdown/bin/mmd* scripts. (I thought I stumbled across something about /Library/…/bin/ vs ~/Library/…/bin/, but I can’t find it now.) Sadly, this approach would bypass Scrivener’s ability to create beautiful e-books.

Out of the box, Scrivener only supports four formats to compile MMD into: LaTeX, Rich Text, HTML, and Flat XML. Even so, I’m not sure which MMD features are supported:

• bold and italics: yep, yep (Compile options)

• metadata and headers: nope, nope (I think Scrivener only uses its own built-in support for these things)

• (ASSUMING YOU’RE NOT USING LATEX) cross references, images, block quotes, code blocks and inline code, etc.: probably not (though I haven’t tried very hard)

For most manuscripts, it appears I’m better off using rich text (Scrivener’s internal format). If I wanted to write an e-book about MultiMarkdown, using MultiMarkdown, I guess I’d use the “technically, it can” options I listed above.

Am I missing something?

[AmberV]

MultiMarkdown itself does not support an e-book export out of the box, otherwise we would gladly add support for it. As you note, Fletcher has made a script for turning an MMD generated HTML file into an ePub, but last I checked it requires a functioning development environment on the system to function, so we can’t really integrate that.

Am I missing something?

Pandoc. It takes a MultiMarkdown file as input and produces a very nice ePub. Do a little searching around on the forums here, you’ll find some tips on the workflow (there is even an ongoing discussion about it), but most of all, the Pandoc website is the best source of information. You can create a staging area with your CSS, XML file for meta-data, and maybe set up a little script so you don’t have to type in all of the necessary command-line flags every time, then just compile to that folder from Scrivener with your .mmd file. Just make sure to append “_mmd” to the end of the folder name, that will cause Scrivener to update the folder rather than replace it with compile data.

metadata and headers: nope, nope (I think Scrivener only uses its own built-in support for these things)

Could you clarify what you are referring to here? There is a “Meta-Data” compile option pane for setting up the MMD meta-data block, or you can just place a document called “Meta-Data” at the top of your compile group (what appears in the Contents compile option pane) and type in the block yourself. These two will be added together if you use them both, making it possible to supply supplementary document-specific meta-data in addition to global meta-data (for cases where you have more than one document in the project).

What sort of headers do you mean? Page headers and footers? You’re right, Scrivener has nothing to do with that as this is something MultiMarkdown does, and then of course only with formats that support the concept of a page.

(ASSUMING YOU’RE NOT USING LATEX) cross references, images, block quotes, code blocks and inline code, etc.: probably not (though I haven’t tried very hard)

I don’t understand the LaTeX comment—all MultiMarkdown is the same up front, that’s one of the big goals of the format. A block quote is merely a sequence of paragraphs prefixed with a greater-than symbol, no messing about with rulers and line-heights and font sizes. What that ends up looking like, or being is a function of the export converter. Most of this stuff is MMD’s remit, though, and a lot of it isn’t even something Scrivener has a concept of. I think you might be thinking that Scrivener is a MultiMarkdown generator, but it’s really more of a hybrid. It has a few internal features that will output as MMD syntax, you can read up on those in the user manual, but for the most part the system is set up to let you author an MMD file in the text editor, using Scrivener’s organisational features. This workflow is for people that actually like writing in MMD. I use Scrivener more like a plain-text file organiser, myself. The integration is all about making the process of assembling a large-scale MMD file as seamless and easy as possible. The hard stuff: managing heading levels with hashmarks, not having to bother with image management and coming up with a thousand footnote anchors—we can help you with that. Use Scrivener’s built in features for these things if you will.

So in short, there are three primary ways to write in Scrivener:

1. With a bunch of palettes and buttons and widgets and fonts and paragraph attribute settings: you use all of the compile options not prefixed with “MultiMarkdown”.
2. You write using MultiMarkdown to assign semantic ranges to your text with visible symbols. You get to use the compile options prefixed with “MultiMarkdown”.
3. Anything using the “Plain-text” compile option—and with that you can use Scrivener to compose everything from an XML file to a LaTeX document to a complicated bash script. But it’s the same idea as MMD, so I hesitate to call it a true third. The only difference is that you are not automatically post-processing the output, and Scrivener isn’t injecting any MMD-specific syntax into the output file.

[jmwzht479]

Ioa, thanks for your reply.

@AmberV

MultiMarkdown itself does not support an e-book export out of the box, otherwise we would gladly add support for it. As you note, Fletcher has made a script for turning an MMD generated HTML file into an ePub, but last I checked it requires a functioning development environment on the system to function, so we can’t really integrate that.

So far as I can tell, the “script” is a makefile, but could trivially be turned into a Bash script using only the multimarkdown and xsltproc commands. (in this case, “trivially” means “I’m able and willing to write such a script.”)

@AmberV

Am I missing something?

Pandoc. It takes a MultiMarkdown file as input and produces a very nice ePub. Do a little searching around on the forums here, you’ll find some tips on the workflow (there is even an ongoing discussion about it), but most of all, the Pandoc website is the best source of information. You can create a staging area with your CSS, XML file for meta-data, and maybe set up a little script so you don’t have to type in all of the necessary command-line flags every time, then just compile to that folder from Scrivener with your .mmd file. Just make sure to append “_mmd” to the end of the folder name, that will cause Scrivener to update the folder rather than replace it with compile data.

Thanks!

Pandoc is 99% MMD-compatible. Crazy thought: To deal with 100% MMD-compatible markup, some script could use MMD to generate HTML, then invoke Pandoc to turn the HTML into EPUB (and then maybe kindlegen to turn EPUB into Kindle format).

@AmberV

metadata and headers: nope, nope (I think Scrivener only uses its own built-in support for these things)

… There is a “Meta-Data” compile option pane for setting up the MMD meta-data block, or you can just place a document called “Meta-Data” at the top of your compile group (what appears in the Contents compile option pane) and type in the block yourself. These two will be added together if you use them both, making it possible to supply supplementary document-specific meta-data in addition to global meta-data (for cases where you have more than one document in the project).

I’ll try that.

@AmberV

(ASSUMING YOU’RE NOT USING LATEX) cross references, images, block quotes, code blocks and inline code, etc.: probably not (though I haven’t tried very hard)

I don’t understand the LaTeX comment …

Probably because it was bogus.-) I was unreasonably focused book creation. I played with MMD → HTML, and verified that block quotes, code blocks (but not code fences with info-strings), and inline code all work.

Thanks again!

[AmberV]

So far as I can tell, the “script” is a makefile, but could trivially be turned into a Bash script using only the multimarkdown and xsltproc commands. (in this case, “trivially” means “I’m able and willing to write such a script.”)

True, but we’re probably just going to go with Pandoc’s support in the future, rather than cobble something together from this. Importantly, Pandoc supports ePub2, which has far more support at the existing hardware reader level out there in the wild. MMD is pure HTML5, and the ePub generator ePub3 only.

Pandoc is 99% MMD-compatible. Crazy thought: To deal with 100% MMD-compatible markup, some script could use MMD to generate HTML, then invoke Pandoc to turn the HTML into EPUB (and then maybe kindlegen to turn EPUB into Kindle format).

That could work, though I think the only major roadblock you’d run into with Scrivener in its current implementation is the image dimensions being tacked on if you resize figures in the editor (images that haven’t been resized in the editor will be Pandoc compatible). Yes, the meta-data block is different, but that’s easily solved by avoiding the meta-data pane in the compiler and using your own “Meta-Data” file in the Binder. You could even hot-swap between a Pandoc/MMD meta-data block as needed, if you wanted MMD for something else like ODT or LaTeX (I prefer its output to Pandoc’s for these).

I was unreasonably focused book creation. I played with MMD → HTML, and verified that block quotes, code blocks (but not code fences with info-strings), and inline code all work.

Do you have your own MMD installation? If not, do that, the version in Scrivener is old, pre-dating the introduction of fenced code blocks. Scrivener will automatically defer to using the system /usr/local/bin/mmd if it exists.

[jmwzht479]

Some steps forward, some steps back.

I installed MultiMarkdown and Pandoc. (They live in /usr/local/bin/, not /usr/bin/ as I think you suggested.)

I think I understand the workflow better:

• Write in Scrivener using MultiMarkdown syntax.
• Add a “Meta-Data” document in the top level folder (Draft or Manuscript or whatever it is for your project).
• Compile for plain MultiMarkdown (not MultiMarkdown → anything), creating a .txt file.
• After compilation, escape to a command line and run pandoc against the .txt file.

If that’s not right, please let me know.

I’m still unclear about:

@AmberV

You can create a staging area with your CSS, XML file for meta-data, and maybe set up a little script so you don’t have to type in all of the necessary command-line flags every time, then just compile to that folder from Scrivener with your .mmd file. Just make sure to append “_mmd” to the end of the folder name, that will cause Scrivener to update the folder rather than replace it with compile data.

So I manually create a folder with an _mmd suffix? I’m not sure which “.mmd file” you’re talking about; one created by Scrivener?

When I use the above workflow, nothing external gets invoked (which makes sense).

When I compile MultiMarkdown → Web Page, only the multimarkdown binary (the one in /usr/local/bin) is invoked. None of the mmd2* scripts get run. That sort of ruins the ability to hijack one of those. Play sad trombone. Oh well.

Thanks for all your help, Ioa.

[AmberV]

Thanks, I’ve fixed the original post’s path error.

After compilation, escape to a command line and run pandoc against the .txt file.

Yup, that all sounds good to me. That’s basically what I do for a few projects that require custom post-processing.

So I manually create a folder with an _mmd suffix? I’m not sure which “.mmd file” you’re talking about; one created by Scrivener?

Call it whatever you like, .txt, .mmd, .md, .markdown—it’s just the text file that you compiled. Like I say if you choose to compile into a folder ending in “_mmd” then Scrivener won’t overwrite the contents of that folder, it will just update any files (the MMD file, any images) it would otherwise ordinarily produce into a new folder. Everything else is left alone, such as the files you use to put together the ePub. I’m not sure how well this idea will blend with automation, as described below, however. A static working folder with your CSS files and whatever else you need to generate the ePub file may work better.

When I compile MultiMarkdown → Web Page, only the multimarkdown binary (the one in /usr/local/bin) is invoked. None of the mmd2* scripts get run. That sort of ruins the ability to hijack one of those. Play sad trombone. Oh well.

No sad trombones! You are right that all of the built-in integrated converters will use the binary by default as it is far faster than the older scripted method. However, you can still cause Scrivener to use external shell scripts, and from there you can go in and hijack the process, run Pandoc instead of MultiMarkdown, send commands to MySQL, whatever strikes your fancy.

It sounds like you have the support package installed in your ~/Library/Application Support/MultiMarkdown, but if not download and install that. If you look in that ./bin folder, you’ll find a pair of “xslt” scripts. To cause Scrivener to use those shell scripts instead of the binaries, check off the Use XSLT post-processing option in the Compatibility compile pane (available to HTML and LaTeX outputs).

The script that HTML with XSLT triggers is, from that support folder, ./bin/mmd-xslt. LaTeX with XSLT will use the ./bin/mmd2tex-xslt script.

So with all of this, you can strike the last task on your checklist. With a customised post-processing script you will be able to compile to “MMD->HTML”, but completely override the output so that you use Pandoc to generate an ePub file instead. Scrivener is going to send the raw MMD document to that script via STDIN, and will wait for STDOUT, printing the results into a file as chosen in the file dialogue during the compile process. So you could generate the ePub in /tmp then ‘cat’ that to STDOUT which Scrivener will save into the chosen file path. The only minor hitch is that it will have .html on the end of it no matter what—so you just have to rename the file to .epub after compile. Otherwise that technique works great!

[beaker]

I’m going with Scrivener very happily for some time. I first used Pandoc in combination with cmake for compilation to doc(x), tex and pdf with latex. But recently, I changed to Pandoc with waf (build system). It’s really easy to export from Scrivener to md and compiling to any Pandoc format via waf.

I think, to include something like waf into Scrivener would be a great opppurtunity to supply most users with a fixed tool chain (as before) and power users with a possibility to control nearly every build step.