Pandoc to ICML

Scrivener Scrivener for macOS
21

Learning how to use the POSIX Shell using Terminal is the single greatest thing I did when I switched from Windows to Mac many moons ago! macOS foundations are *nix, even with all the lovely icing on top! You really don’t need to learn Ruby (although it is a beautiful language), as pandocomatic is a command-line tool and never exposes and ruby for its users. BUT learning how to use the shell even minimally (terminal.app) and understand what markdown and YAML look like is probably much more useful! But even then it isn’t necessary, as long as you know enough to move a file from A to B. If I was sat next you you I could probably fix what you are doing in a minute and you would be like “duh”! It is hard to see the woods from the trees sometimes, especially if you’ve only ever lived in a desert!

My instruction were to break the problem down into smaller chunks and try only the pandocomatic on a simple test markdown file. Even simpler is to run pandoc directly on the command-line.

Sorry this was my fault!!! I naively assumed you had some terminal experience as you knew something about XML and would understand what the commands look like that you should enter in the terminal. That command runs pandoc to save the defaults ICML template it uses. So open terminal.app and run this in the command line: cd ~/Desktop; pandoc -D icml > custom.icml — it should create the icml on the desktop and you can move it to your templates directory. You can see the one I generated here (I uploaded it to Github after answering your question):

This is the contents of a simple markdown file, copy that text into a blank text document and save it as test.md. This is the file you pass to pandocomatic. The bit at the top is what tells pandocomatic which template or recipe to use. It is worth looking at the markdown scrivener produces as it will also be helpful for you to see the intermediate from

[quote=“TStone”]
The rest of the description is unfortunately way over my head. Should the following be entered in the Scrivener Processing field?[\quote]

No. This was the command pandocomatic --debug test.md to enter into terminal and the output you should see.

What I was trying to do was to help you by first ignoring Scrivener / scrivomatic, and simplifying the problem to the simplest procedure. That procedure is to edit your pandocomatic.yaml to make the icml-test template, then create the test.md markdown test file, then open up terminal cd to the same directory the test.md file is stored, and run pandocomatic directly on the test.md file you created. A test.icml file should be created in the same folder.

Overall it does sound as if this workflow is close to the limit of your current abilities. I would suggest dropping scrivomatic/pandocomatic and just trying to get pandoc to run, first try manually in the terminal, if that works then translate it into Scrivener. Small steps can climb mountains!

22

That sounds very likely, because I can’t believe that anyone would get time over to do any real work, if they had to do what it looks like in my eyes. Somewhere, I must have turned left where I should have turned right.

Hahaha, ok I see! Well, XML is easy - all tags are human-readable and the nesting is inituive. Just like AppleScript, PHP and similar; human-readable. You read a “Hello World” example, and then you can simply guess, and 9 out of 10 times you’ll be correct, as each command and operator tells you what it does in its very name. The letters and glyphs of this ruby/terminal/commandline apporach to writing might be powerful, but human-readable it is not. Who would be able to guess that a negative value “-s” means an addition? :slight_smile:

Awesome! That worked!

Aha! Since you wrote “Here is a sample document:” while describing the .icml template, I assumed it was for the .icml template. But it is supposed to go into a different separate .md document, not in the .icml nor in Scrivener? Ok, I’ll try that.

Ok, I’ll copypaste it into the terminal now, but change your template and template path for mine! Like this:

➜ pandocomatic --debug test.md pandoc --from=markdown \ --to=icml \ --standalone \ --template=/Users/tstone/.local/share/pandoc/templates/magictemp.icml \ --verbose
Unfortunately, that didn’t work. I guess I must have missed to install something during the long and arduous installation of all the extras (homebrew, rbenv, ruby, pandoc, pandocomatic…) because I got this error message:

Toms-iMac:Desktop TStone$ ➜ pandocomatic --debug test.md
-bash: ➜: command not found

Any idea what I need to install to get the ➜: command to work?

I believe this is the most accurate statement so far. :slight_smile:
This is, by far, the most difficult piece of software I’ve ever come across. Last month I taught myself to program Arduino microprocessors, and that was a piece of cake in comparison.
I’ll give myself 2 more days to figure out this ruby/terminal/commandline apporach, and if I haven’t solved it by Sunday evening, I’ll go the AppleScript route instead, because I’ll really need to get started writing on Monday morning, or my publisher will want to have the long-gone advance returned.

23

If I’m not mistaken, try it without that arrow. I think that’s just for emphasis, not meant to be a literal part of the command line.

24

Aha, It also struck me that if I need to specify a path to the template file, I probably need to specify a path to the .md file. Like this?

Hm… I think the arrow is necessary, because while something happened, it wasn’t what was expected. I got a .HTML file instead of an .ICML file. This in the Terminal:

Toms-iMac:~ TStone$ pandocomatic --debug /Users/TStone/Dropbox/Apps/Scrivener/test.md
pandoc	
Pandocomatic needed 3.1 seconds to convert '/Users/TStone/Dropbox/Apps/Scrivener/test.md' to 'test.html'.
Toms-iMac:~ TStone$ pandoc  --from=markdown \
>     --to=icml \
>     --standalone \
>     --template=/Users/TStone/.local/share/pandoc/templates/custom.icml \
>     --verbose
25

Maybe I missed it somewhere in the thread but why do you want to convert your Scrivener files to ICML (InCopy) in the first place? When it seems from what I read from InDesign users (I posted in a previous post on this thread) is that the norm is that they get MS Word files which are then imported into InDesign and converted into ICML.

Unless I missed something along the way (probably) then the work flow should be to export your Scrivener files to MS Word and then feed that to InDesign. Viola!

So why does that work flow not work for you? It seems like you are introducing an overly complex and unnecessary middle step?

Please clarify?

26

I don’t own Word, so it feels a bit odd to me to export to a thirdparty fileformat that I don’t have the software for. Way back when I used QuarkXpress for layout, which also was said to work well with Word files, I did extensive attempts with the .doc files that ClarisWorks exported… and it invariably ended with being forced to go via unstyled .txt files instead. So I gave up on the Word format in 1997, and haven’t used it since.

On the other hand, the ICML format is a native format for InDesign, and being based on XML, it is way more rubust and predictable than Word have ever been. Going that route would not add a middle step - it would remove the middle step. The path from ICML into InDesign is simple, smooth and flawless.

My intention is to export the InDesign styles from my last two books into a template ICML. And using that template in the Scrivener/Pandoc export would mean that 90% of all the formatting for my next book would happen automatically, making it a breeze to get the same look and appearance as the previous books.
That was the idea at least. I didn’t expect the route from Scrivener to ICML to be so troublesome.

27

InDesign doesn’t care if you don’t own Word. It just wants Word files no questions asked. You could just export from Scrivener to RTF or ,DOCX and from there feed it to InDesign using these guidelines:

creativepro.com/retaining-impor … documents/

And see what the result is. If it is dreck then back to where you were in your adventures with Pandoc and ICML. But if you get something approaching the look and feel that you desire then you just clean it up till it suites your aesthetic and get to bed early.

Worth a try.

28

It ask a lot of questions.
It is far from automatic, and the whole process have to be repeated every time you place a file. It isn’t dynamically linked in any way. I am familiar with the workarounds in the article you linked, and for shorter pieces it is often faster and easier to import/paste it as txt and style it while layouting it.
But you’re likely correct, it is probably there I’ll end up. But I haven’t given up yet. A direct pipeline will be worth a few day’s angst. :slight_smile:

29

Devin was correct, and the arrow was the implicit shell text entry point. Let me try to be more explicit. ONLY text in red is a command, and only that gets pasted or typed into the terminal.

By the way in the terminal, the ~ character has a special meaning, which is that it is the symbol representing your home directory, so where you see a ~ in the paths terminal will replace with e.g. /Users/ian/ in my case and /Users/TStone in your case.

So in the pandoc-data-directory (~/.local/share/pandoc/) I have added a templates/custom.icml file that I edited and in ~/.local/share/pandoc/pandocomatic.yaml I made the section to define the recipe called icml-test (I think you did this successfully before). Now in terminal:

  • change to desktop directory: cd ~/Desktop
  • remove any previous test.md or test.icml files: rm -f test.md test.icml
  • create a new test.md file: echo “—\ntitle: Test\npandocomatic:\n use-template: icml-test\n—\n\n# Title\n\nA paragraph.\n” > test.md
  • run pandocomatic: pandocomatic --debug test.md

The output of pandocomatic in the terminal should look like this:

pandoc  --from=markdown \
    --to=icml \
    --standalone \
    --template=/Users/ian/.local/share/pandoc/templates/custom.icml \
    --verbose
Pandocomatic needed 0.1 seconds to convert '/Users/ian/Desktop/test.md' to 'test.icml'.

Let us now check the output. In my case I had previously added a comment on the 4th line of custom.icml, lets see if our output document test.icml has used the correct template:

  • show the first 4 lines of our test.icml file: head -n 4 test.icml 
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?aid style="50" type="snippet" readerVersion="6.0" featureSet="513" product="8.0(370)" ?>
<?aid SnippetType="InCopyInterchange"?>
<!-- This template has been customised by Ian -->

The output should look like the above if it worked (see the comment I had added to the 4th line of ~/.local/share/pandoc/templates/custom.icml).

In your recent post, you say the command only outputs HTML. This happens when pandocomatic doesn’t find the use-template line in the markdown document. It suggests your test.md file is not well formed. The echo command above should create a correctly formed file for you.

Now let’s remove pandocomatic from the process and focus on a simpler workflow.

  • change to desktop directory: cd ~/Desktop
  • remove any previous test.md: rm -f test.md test.icml
  • create a new test.md: echo “—\ntitle: Test\n—\n\n# Title\n\nA paragraph.\n” > test.md
  • run pandoc: pandoc --standalone --to icml --template=custom.icml --output test.icml test.md

Now check our file using head -n 4 test.icml to see if the template was used (in my case it worked fine).

Pandoc itself does have a newish system that allows you to create default files with a bunch of settings. So we can simplify our pandoc command-line a bit:

  • create a new defaults folder: mkdir -p ~/.local/share/pandoc/defaults
  • create a new icml defaults file: echo “standalone: true\nfrom: markdown\nto: icml\ntemplate: custom.icml\noutput-file: test.icml” > ~/.local/share/pandoc/defaults/icml.yaml
  • check our defaults file looks OK, this will show you the contents of the file: cat ~/.local/share/pandoc/defaults/icml.yaml 
standalone: true
from: markdown
to: icml
template: custom.icml
output-file: test.icml
  • run pandoc using the defaults: pandoc --defaults icml test.md
  • check we used the template: head -n 4 test.icml

If you can get one of these working then you can use that in Scrivener. I use pandocomatic as it gives me much more control, I can run scripts to modify input or output files, and automate lots of features from a single compile. Pandoc defaults is a simpler way to not have to type in lots of command-line options and you can create many defaults files for different sets of options. Raw pandoc on the command-line is the fall-back and still works great. The point is to get the basics working.

And yes, I agree that terminal invocations can seems like some kind of arcane abstract magical language. But this abstract language is incredibly powerful, which is why computer geeks keep using it. You can do a series of complex actions in one command on the shell that can express many steps and lots of clicking here and there in the GUI…

30

Hi Orpheus, perhaps not related to what TStone wants to do, but to answer the question more broadly – why would we want to use Pandoc to create an ICML file, when we could use a DOCX?

There are several advantages to the Pandoc workflow. Pandoc supports a lot more of the tools that can greatly simplify a publishing workflow. As soon as you want to add in a bibliography, maths support, figure placement etc, Pandoc offers a document-agnostic way to represent all these and many more features. So in theory, you could add bibliography and maths outputs without having to perform any intermediate manual steps. As your requirements scale, Pandoc can support that. For example, say you are writing a technical book that depends on many plots, Pandoc can be extended to support plots that get dynamically generated at compile time. And all of this is document-agnostic. say you have been working on ICML output but someone asks you for a LibreOffice or Word version, or a PDF generated from LaTeX instead? Pandoc will output multiple versions simultaneously, no need to change anything in Scrivener or do lots of intermediate manual adjustment. Pandoc supports powerful filters than can modify the document structure, say you want to have bibliographies per section then a filter can do that automagically.

So yes, we can use DOCX, and for simple books this is perhaps sufficient. But investing in a Markdown/Pandoc workflow ensures extensibility, and if our needs may require more complex features Pandoc will largely cover those needs in the future.

The obvious downside is a requirement to learn how to manage the shell and set up the folders and files that a system like Pandoc uses. You don’t need to be a *nix wizard, but you do need to be able to cast some rookie spells with shell commands. This is in addition to learning Scrivener itself, and the hand-rail that supports people compiling to DOCX (the user manual etc.), is removed.

31

Thank you for taking your time and holding my hand through this. I am really trying to follow your advice as closely as I can. So to be clear, that my installation is missing the arrow command doesn’t mean that I’ve missed to install anything?

Seems something have already gone awry, as the procedure resulted in a HTML file.
This in the Terminal:

Toms-iMac:~ TStone$ cd ~/Desktop
Toms-iMac:Desktop TStone$ rm -f test.md test.icml
Toms-iMac:Desktop TStone$ echo "---\ntitle: Test\npandocomatic:\n use-template: icml-test\n---\n\n# Title\n\nA paragraph.\n" > test.md
Toms-iMac:Desktop TStone$ pandocomatic --debug test.md
pandoc	
Pandocomatic needed 5.7 seconds to convert '/Users/TStone/Desktop/test.md' to 'test.html'.
Toms-iMac:Desktop TStone$

I think something went wrong in generating the test.md, as its content looks a bit wonky. It seems the Line Feed character (\n) wasn’t escaped, but got outputted as any other characters.

Our output… If I also should have expected a comment on the fourth line of my output document, I have missed something in your directions.

Let me post a zip archive with my pandocomatic.yaml, custom.icml, test.md and test.html
[attachment=0]testing-icml.zip[/attachment]

32

This thing is quite finicky and particular…

I manually replaced the newline characters for new lines in the test.md and ran “pandocomatic --debug test.md” again.
…got a HTML file, again.

Then I spotted that the second set of [code]

[/code] in the .md file had a space before it, so I removed it and tried again, This time; success!
This in the Terminal:

Toms-iMac:~ TStone$ pandocomatic --debug test.md
Input does not exist: 'test.md'.
Toms-iMac:~ TStone$ fuck!
-bash: fuck!: command not found
Toms-iMac:~ TStone$ cd ~/Desktop
Toms-iMac:Desktop TStone$ pandocomatic --debug test.md
pandoc	--from=markdown \
	--to=icml \
	--standalone \
	--verbose \
	--template=/Users/TStone/.local/share/pandoc/templates/custom.icml
Pandocomatic needed 2.4 seconds to convert '/Users/TStone/Desktop/test.md' to 'test.icml'.

I’ll go back to your previous post and continue where I left off.

33

That’s because you are missing some of the parameters in the command to run pandoc. The code below is all a single command, just spread out over multiple physical lines for ease of reading:

pandoc  --from=markdown \
    --to=icml \
    --standalone \
    --template=/Users/ian/.local/share/pandoc/templates/custom.icml \
    --verbose

What you seem to have typed is below:

pandocomatic --debug test.md

It has been a long while since I have played with all of this, so I am not even sure that ‘pandoc’ and ‘pandocomatic’ are the same program (isn’t pandocomatic a wrapper around pandoc?) But the first example is explicitly telling pandoc that the source format is Markdown and the target format is ICML, whereas your command is relying on defaults.

34

Thank you! I couldn’t quite understand what you say my error was. Since I am, evidently, easily confused, I think it is best that I follow nontroppo - The Doofus Whisperer - and his lead for now, :slight_smile:

35

I must seem terribly stupid now, but I fear our definitions of “simpler” differs. To me, it looks like we are rapidly moving away from “simpler” now. :slight_smile: But I’ll try! I’ll keep the previous test.md though, as the Terminal, for some reason, is generating a malformed file.
And yes, this worked! I got a new test.icml.
The last command generated some XML in the Terminal window, and I’m not sure what to do with it. Should I save that XML somewhere, or is it just to be looked at?

I have no idea what we’re doing now. But I’ll try to follow the best I can…
…ok, something went wrong. This is the inputs and outputs in the Terminal window.

Toms-iMac:Desktop TStone$ cd ~/Desktop
Toms-iMac:Desktop TStone$ mkdir -p ~/.local/share/pandoc/defaults
Toms-iMac:Desktop TStone$ echo “standalone: true\nfrom: markdown\nto: icml\ntemplate: custom.icml\noutput-file: test.icml” > ~/.local/share/pandoc/defaults/icml.yaml
Toms-iMac:Desktop TStone$ cat ~/.local/share/pandoc/defaults/icml.yaml
“standalone: truenfrom: markdownnto: icmlntemplate: custom.icmlnoutput-file: test.icml”
Toms-iMac:Desktop TStone$ pandoc --defaults icml test.md
Error parsing /Users/TStone/.local/share/pandoc/defaults/icml.yaml line 1 column 22:
Expected start of line
Toms-iMac:Desktop TStone$ head -n 4 test.icml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<?aid style="50" type="snippet" readerVersion="6.0" featureSet="513" product="8.0(370)" ?>
<?aid SnippetType="InCopyInterchange"?>
<Document DOMVersion="8.0" Self="pandoc_doc">

I think that the Terminal echo things in a malformed way. Do I need to reinstall anything?

That does sound attractive. Although, so far, this apporach is more about copypasting random glyphs for days to do what could be done through a GUI in a minute. So far, I haven’t noticed the advantage. ;-)

36

Thank you for taking the time to point this out to me. If/when I get close to publishing a book I will carefully consider this option. I grew in computers starting with punch cards and then it was command line till Windows 3. So while it has been a long time since I used command lines I could probably do it if I saw the need. Especially with your help. :wink:

Would I be correct to assume that you write your documents in Markdown in order to take advantage of Pandoc?

37

I could get pandocomatic --debug test.md to work, but not the later suggestions, so maybe we can stick to the pandocomatic route, since trying to troubleshoot several variations simultaneously most likely will lead to confusion, hairloss and distraught neighbours.

38

I’ve tried that now.
I found a spot where it seemed reasonable to put the info for the .md intro:
[attachment=1]md-intro-meta.jpg[/attachment]
On compile, it seems to have ended up in the right spot - at the very top of the generated .md file:

---
title: Cyclone  
author: Tom Stone  
pandocomatic: 
 use-template: icml-test
---

I did try to enter the code that worked earlier, into the processing section, like this:
[attachment=0]Process.jpg[/attachment]

However, that did not work. Instead, an error log window popped up, saying:

Unknown option --debug. Try pandoc --help for more information.
I did try pandoc --help, though in Terminal, and cartload of stuff fell out, none of it helpful.

39

After looking at the Pandoc site pandoc.org/ I see that you have many choices not just Markdown to write in. That is good because I am already far into a project that doesn’t use Markdown.

40

Great that pandocomatic is working at least. I am surprised that the echo command does not escape characters on your system (which OS?). I think if you replace echo with printf is should render escape sequences reliably for sure:

printf “—\ntitle: Test\npandocomatic:\n use-template: icml-test\n—\n\n# Title\n\nA paragraph.\n” > test.md

I am using macOS 11.2 and zsh is the default shell, but testing with the previous default bash also escapes on my system.

OK, back to the problem at hand. Pandocomatic works in the terminal, which means scrivomatic should. You Scrivener post-processing panel is wrongly treating pandocomatic as an “option” for pandoc, it is not it is a command itself. Pandocomatic runs pandoc for you, so it should be the main command…

If you are using the Ruby which comes with macOS (run ruby -v in terminal to see) and at least Catalina and installed pandocomatic gem using system Ruby, then as far as I know, Scrivener includes the path to the folder where pandocomatic is installed (/usr/local/bin), so you can simply do this:

[attachment=1]Screenshot 2021-03-07 at 10.49.03.png[/attachment]

That gets me 3 files in my compile folder, test.md + test.icml + pandocomatic.log which contains the same output you saw on the command line.

[attachment=0]Screenshot 2021-03-07 at 10.50.00.png[/attachment]

The contents of pandocomatic.log shows the correct template was used:

pandoc  --from=markdown \
    --to=icml \
    --standalone \
    --template=/Users/ian/.local/share/pandoc/templates/custom.icml \
    --verbose
Pandocomatic needed 0.2 seconds to convert '/Users/ian/Desktop/Compile-mmd/Test.md' to 'Test.icml'.

If this doesn’t work in Scrivener, then there is a problem in the path and Scrivener cannot find the Pandocomatic command. If you used rbenv to install a newer version of Ruby and pandocomatic is installed there, then this is where getting scrivomatic running will help. But if you want to stick to pure pandocomatic, then you can use rbenv to revert to using the system ruby like so in terminal, and make sure pandocomatic is installed there:

  • set rbenv to prioritise default Ruby: rbenv global system; rbenv local system; rbenv shell system
  • check the ruby version should not be 2.6.3: ruby -v
  • install pandocomatic: gem install paru pandocomatic
  • check pandocomatic is installed: pandocomatic -v

That should make pandocomatic work with Scrivener even if you have rbenv or other ruby versions installed. Scrivomatic manages the paths for you correctly, so that is the other way to get it to work…