Pandoc to ICML

The incorporation of Pandoc into Scrivener 3 is excellent, and I have used Pandoc in the past to export to ICML, will this be something that can be added to the Scrivener 3 export options? Or if it can already be done I would like to know how to do this.

If you create your own Compile Format there is the option to post-process the output on the command-line - presumably you could use this go generate ICML. You would choose MMD format, then, in the “Processing” options, choose to use Pandoc syntax and add the necessary command-line arguments to turn the output text file into an ICML file.

All the best,
Keith

Thanks Keith, I have followed your description, but as yet not got it working, but will perceiver.

Thank you

It does work, here are the settings you should enter in the processing panel of the compile format editor:

You may need to change environment depending on where you installed Pandoc. I use Homebrew so the default is /usr/local/bin.

You can also use a Pandoc template manager, where you use metadata written in the Scrivener front-matter to identify the conversion settings if these change often, or you want to output multiple formats from the same Scrivener compile…

Thank you nontroppo, I was just about to post the settings I have it working with. It appears that the InDesign paragraph styles are hierarchal on folder structure rather than from the section type styles, which is all okay too.

You can also make sure you capture any output from Pandoc (warnings/errors) to a log file if you put:

 2>&1 >> scompile.log

…at the end of the Arguments line.

I found this article helpful.

This is exactly what I need. Unfortunately, I can not get it to work. An ICML file is being generated, and it contains the exported content, but InDesign doesn’t recognize it as a valid file, as it only contain the body and not the XML header.

I’ve struggled for almost a week to figure out Pandoc, Pandocomatic, Scrivomatic et cetera, but at this point it seems unlikely that I will be able to get any of it to work. I just want to get the Scrivener document into InDesign with the styles intact - is that even possible?

If anyone here have seen any tutorial on the topic, a link would be very appreciated!

For just about every file format that can be expressed in pieces (i.e. not binary like PDF or docx), Pandoc will by default only export the content without any kind of boilerplate document structure around it. This of course makes it easier to copy and paste into such a boilerplate, or use native syntax to include content files if supported by the format. It will even do that for files we would not traditionally think of in that fashion, like RTF.

If you want it to assemble a generic document container for your content, thus making it something normal software can open, then use the -s flag in your arguments, or --standalone if you prefer self-documenting commands.

And if you do have some idea for how you want the total document to look and function, you might be interested in templates, which is the stuff it wraps around your content when using the -s flag. There are vanilla defaults of course, but review the Templates chapter in the Pandoc documentation for more information on that.

Thank you! :slight_smile:
Ok, I formated some text in InDesign, with styles named the same as in Scrivener, then exported an ICML file and replaced its guts with Pandoc template variables, and copied it to $HOME/.local/share/pandoc/templates/.
Added a path to it in pandocomatic.yaml

In Scrivener Compile, I added -s to the cryptic command in the screenshot that was posted above:

-t icml <$inputfile> -s -o <$outputname>.icml and hit compile.

To my surprise, I got a working ICML file! :slight_smile:
However, with depressing predictability, it didn’t make use of my template. The header came from somewhere else…

Any idea where the default ICML template can be found, and how to tell Pandoc to use my own template instead?

Hi ,if you are using pandocomatic, then remember it uses a “recipe” combining several pandoc settings for you, you don’t need to pass settings in the Scrivener compile panel directly. Here is an example recipe you can put in pandocomatic.yaml:

#-----------------------------------------------------------------------------
#:::::::::::::::::::::::::::::::::: ICML :::::::::::::::::::::::::::::::::::::
#-----------------------------------------------------------------------------
  icml-test:
    pandoc:
      from: markdown
      to: icml
      standalone: true
      template: templates/custom.icml
      verbose: true

To get the latest pandoc ICML tempate use pandoc -D icml > custom.icml and then edit that to your taste, then put it in your pandoc data directory (~/.local/share/pandoc/templates as the default). Here is a sample document:

---
title: Test ICML
subtitle: A test document
abstract: This is the abstract for a test file
pandocomatic_:
  use-template: [icml-test]
---

# Intro

Sint meis quo et, vis ad fæcete dolorem! Ad quøt moderatius elaboraret eum, pro paulo ridens quaestio ut! 

I can compile this using pandocomatic, notice it doesn’t need any command-line options as the metadata specifies the template which contains the requisite settings for us) and I verified that the template I edited was actually used for the final document (I added a comment in the template that was in the output):

➜ pandocomatic --debug test.md
pandoc  --from=markdown \
    --to=icml \
    --standalone \
    --template=/Users/ian/.local/share/pandoc/templates/custom.icml \
    --verbose
Pandocomatic needed 0.1 seconds to convert '/Users/ian/test.md' to 'test.icml'.

Pandocomatic in debug mode shows you exactly the command-line options it has used, you can see the template is specified using –template=path/to/file.

Once you are sure pandocomatic is working then scrivomatic should work just using the default invocation in Scrivener’s compile panel.

The metadata I keep as front-matter in Scrivener, then edit it as required to compile, allowing me to select multiple outputs and making sure only pandocomatic.yaml recipes needs to be edited if I want different outputs (you can add as many recipes as you need). If you don’t need the flexibility pandocomatic affords (i.e. you only ever output the same format), you can easily go back to specifying pandoc as the executable and everything as command-line options in Scrivener compiler directly, just don’t invoke pandocomatic/scrivomatic at all (all they do is help with the path and wrap the settings for you).

Thank you! It is a bit tricky to follow along, since I’ve been a Mac user my whole life. The “programmer’s interface” is unfamiliar to me, and quite far from the Apple HIG. The learning curve for using Scrivener is the steepest I’ve ever experienced… this software is way more difficult to learn than POV-Ray, Fusion 360, After Effects et cetera…
Though, I got a little bit on the way before I got stuck again.

I tried to edit pandoc -D icml > custom.icml to my taste and put it in my template directory, with no success since I didn’t know what I should change and what kind of file to put it in, Eventually the thought struck me that pandoc -D icml > custom.icml maybe referred to something else, and copied it as a command into the Process field in Scrivener’s compile section and hit Compile - and voila; an .icml file!

I’m not sure what to do with the above, as it doesn’t look like XML, but I’ve put it near the top of the .icml template, wrapped in an html comment tag. My guess is that the latin writing isn’t necessary, that it’s just a “Lorem ipsum” to symbolize template content?

The rest of the description is unfortunately way over my head. Should the following be entered in the Scrivener Processing field?

After reading about Scrivener, Pandoc and Pandocomatic, and trying to make it work for the last 8 days, it is clear that I know way too little about Ruby to figure it out.
So I just bought a subscription to Lynda, as they have a “Ruby Essential Training” course there. It is only in three parts though, so I worry if it is enough? I do not aim to become an expert, I just want to learn enough to be able to export text from Scrivener to InDesign.
If it turns out that I need more programming knowledge than a weekend’s study, I fear I have to give up on Scrivener and go back to copypasting between TextEdit and InDesign. Not ideal, but it is functional.

Why specifically are you trying to get to InDesign via Pandoc? There are lots of ways to get material out of Scrivener.

In particular, if your last resort approach is to copy and paste from TextEdit, you can either copy and paste from Scrivener directly, or Compile to RTF and then either import the result to InDesign or open in TextEdit and copy and paste from there. If you need styles, Compile to Word instead.

Katherine

I was under the impression that Pandoc was the only direct route from Scrivener to InDesign. InDesign is the layout software I’m comfortable with, and its XML-variation .icml is quite rubust and straightforward. Would like to avoid too many detours, especially those who goes via Word. I tried to import Word files (exported from ClarisWorks) into QuarkXpress in the 90’s, and it never went well. Learnt that the least troublesome way was to solely use unstyled .txt files, or write directly in Quark. And after a few books, I rarely questioned that approach. And that habit stayed with me when I transitioned to InDesign in 2006.

But for the last few years, I’ve more and more thought that I should update my work process to be more efficient. A couple of years ago, I saw a few videos about Scrivener and thought its interface looked attractive and useful… so last new year’s eve I decided to try it out for my next book, provided I could find a somewhat direct pipeline between it and InDesign. After some googling, there seemed to be such a path… I just wasn’t prepared for how complicated that path was, and how many things that needed to be installed through arcane procedures, and the lack of comprehensible documentation of it. I’ve never come across a piece of software before, where I couldn’t make at least a rudimentary test within 2-3 hours, so not having had a single successful attempt for 8 days is puzzling.

Using my old approach is of course a possibility, a Plan B. But it is somewhat redundant to learn the idiosyncrasies of a new piece of software if I still will end up working as in the old one.

Just a little frustrating to be so close. The software is outputting functional .icml files now… it’s just not using my template… But on Monday, I need to get started for real, and if the choice is between an inefficient procedure that have worked for 25 years, or an elegant procedure that I never got to work, I’ll have to go with the former.

1 Like

I’m not familiar with InDesign, but I would be somewhat shocked if they can’t import Word files, given how many editors demand Word submissions and how unlikely the average writer is to be able or willing to produce ICML.

How much mangling is necessary after that is another issue, of course.

If you want to avoid Word – I don’t blame you! – there are lots of ways to produce various flavors of “unstyled text” from the Scrivener side, and let InDesign take it from there. Tables 23.1 and 23.2 in the manual have a complete list of supported output formats.

Katherine

From Scrivener you could just export it as an RTF or other format that Word reads then import it into InDesign. For more details see this article:

creativepro.com/retaining-impor … documents/

Word import/export have probably improved since 1997 when I gave up on it.

InDesign has a whole support community.

community.adobe.com/t5/indesign … filter=all

After a good night’s sleep, I woke up realizing that I’ve been stupid. If I revert to what I had before AmberV’s suggestion, I’ll get an incomplete but correct .icml file. All I have to do then is to write an AppleScript to merge it with my template .icml to get a functional path to InDesign that automatically will get the same formatting and typography as my last two books. :slight_smile:

A big thank you to everyone who helped out! I’m very grateful!