Pandoc to ICML

Yes, many formats can be ingested, and Pandoc does its best to translate the format-specific terminology into an agnostic one. But markdown is the most universal and easily translatable, and Scrivener can create the markdown for you, so you don’t need to write in markdown, just use the tools Scrivener has to make it at compile time… All my writing uses Scrivener styles, I very rarely write raw markdown in my documents (except for my front-matter)…

I’m on OSX 10.14.6
No luck with printf either…
The file “test.md” was created, but it is empty.
This is what the Terminal says:

Toms-iMac:Desktop TStone$ printf "---\ntitle: Test\npandocomatic:\n use-template: icml-test\n---\n\n# Title\n\nA paragraph.\n" > test.md
-bash: printf: --: invalid option
printf: usage: printf [-v var] format [arguments]

Let’s try.
The Terminal says I’m running: ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-darwin18]

Scrivener couldn’t find the Pandocomatic command…

I need to pause here, so I don’t make a mistake I can’t undo… It took considerable effort and hairpulling to get Ruby 3.0.0 installed, because I was led to believe that I needed that update. So all that was in vain? Ruby 3 isn’t compatible with Scrivener, and I should rollback to Ruby 2.6?

This approach is leading me to despair. Do people really suffer through all this in any professional setting?

You asked “…if you want to stick to pure pandocomatic”. I lack the qualifications to understand or answer that. I rather go through more arcane procedures now, if it means less angst later. A “set and forget” solution is preferable,

Don’t panic. Both Ruby 2.6 and Ruby 3 work with Scrivener. But Scrivener doesn’t add rbenv to the path when it runs as it doesn’t know what it is. That is why scrivomatic exists, it manages the path and searches to see if Ruby 3 has been installed via rbenv. You don’t have to roll-back and it isn’t an either-or, but if you also install pandocomatic using the system Ruby (a single command) you don’t need to use scrivomatic.

Or you can replace pandocomatic with scrivomatic and try again:

[attachment=0]Screenshot 2021-03-07 at 20.42.36.png[/attachment]

Thank you. Markdown is simple but it would be tiresome to have to write in it so it is good to know that Scrivener can convert my output to Markdown. So far I have only compiled relatively short articles with Scrivener which were destined for a website. So I have not played with all the compile features and don’t know what it is actually capable of. I later noted that my document can be converted to HTML but I doubt it would be useful for me since I am not using a CMS like WordPress but a site built with Dreamweaver loaded with custom CSS so I would have to convert , and possibly other tags by hand anyway. But I think I will give it a try anyway to see if it saves me any work.

Again, than you for that information.

Let’s do that.
I’ve got the scrivomatic script (this one)here:
[attachment=2]ThePath.jpg[/attachment]
I’ve entered the path to it here:
[attachment=1]TheProcessing.jpg[/attachment]
And this is the log from compiling:
[attachment=0]TheLog.jpg[/attachment]
…not sure why it complains over “-y”, since I didn’t add any “-y”.

Seems I, in a spell of fatalism, just assumed it didn’t work, based on what the log said.
But, lo and behold, in my Scrivener folder is both a .md and an .icml file - all fresh and new!

It worked! And it looks good in InDesign! Wohoo!!! Thankyouthankyouthankyou! :slight_smile:

Now to figure out how to make it use my own icml template, so it will pick up the correct formatting!

Actually they pay people to do it for them. Version management for scripting and programming tools is a significant part of what corporate IT teams do. (When they’re not tearing their hair out over people answering phishing emails.)

Katherine

Great. I’m not sure why you get the log you do either, the log appears to be the output of Pandoc, which is not what Scrivener is running at all according to your screenshot. You should perhaps delete all files including the log files and try again. Anyway, the point is that Scrivener is creating a compiled markdown file with the correct meta-data that pandocomatic can read (otherwise you wouldn’t get any ICML output). I know nothing about ICML itself, so no idea how to best customise it. You can read more about Pandoc templates here: github.com/iandol/dotpandoc/blo … .html#L100). But really you need to build up slowly so you don’t get overwhelmed. Perhaps the first steps should be to really make sure you understand what Scrivener is doing as it compiles a document, how to transfer Scrivener styles to markdown, and understand how named Scrivener styles can be converted into block and inline classes…

Just to be clear, almost none of what is happening in this thread is about Scrivener per se. Scrivener creates a plain text/Markdown file, and provides an interface to route it to the post-processing tool of your choice. What happens after that is entirely up to you. Scrivener doesn’t care about Ruby one way or the other, it just writes text to a file and sends a command to tell the operating system what to do with it.

Which is why Nontroppo is helping you troubleshoot via the command line. It’s easier to get your post-processing flow up and running with simple test files, rather than a full-fledged multi-thousand word manuscript. (And, conversely, it’s easier to troubleshoot what Scrivener is doing by inspecting its output file, before Pandoc and friends work it over.)

Katherine

I do not dare to touch anything now and risk that it stops working! :slight_smile:

ICML is just a flavor of XML, so customising it is pretty straightforward. I’ve already set up three different ICML templates in pandocomatic.yaml & the pandoc templates folder, and they work flawlessly. Changing between the templates, through the meta field in the compile window, is smooth. It is lovely to see InDesign live-update the text when I compile new versions. I am very grateful for your work and assistance! :slight_smile:

As far as I can see, everything that doesn’t involve command line instructions seems straightforward and well documented.

I mainly need to tag character styles, so I’ve set up the Scrivener styles as “bracketed_spans”. I am a bit hesitant on how to tag paragraph styles, as “fenced_divs” looks pretty weird, but fortunately, besides the default paragraph style, all that will be handled in InDesign during layout so there’s no need to deal with that in Scrivener.
As far as I can tell, I have everything set-up and solved now, except for three things.

  • Paragraph breaks…, The compiling is corrupting my carriage returns, and I have to decide on a workaround. I really don’t want to be forced to add double returns manually, but I haven’t figured out how to amend it automatically yet.

  • Replacements. I refer to a lot of drawings in my writing, and it is a bit annoying to write a lot of figure references like; [size=90](FIGURE 38) [/size], in a different character style, I’d like to find a shortcut for that, a quick marking of the number, maybe double parantheises ((38)), and then let compile (if possible) replace and expand it into [size=90](FIGURE 38) [/size] wrapped in the correct bracketed_span. It seems to be possible, but I haven’t quite figured out how.

-Fleurons. Each chapter have the same structure (Title, Intro, Outer reality, Prerequisites, Inner reality, End comments) so I’ve set up a template folder structure accordingly in Scrivener (see screencap). I’d like to have a fleuron (a decorative glyph) between each segment… While I could add it directly to the template file, it is a bit unwise as it might accidentally get deleted or changed, and being a decorative element, it should really not be a part of the process until after compiling, during the layout. It seems to be possible to have it automatically added during compiling, but I haven’t figured out how yet.
[attachment=0]templatestructure.jpg[/attachment]

1 Like

Yes, you are correct. Since all the new stuff, new installations, new procedures and Scrivener itself have been introduced for the sole purpose of getting a smooth pipeline out from Scrivener, I have placed it all in the same bucket, incorrectly and unfairly labeled “Scrivener”.

In theory, I do understand the logic in taking the detour via incremental steps and simple tests via the commandline, but in practice, it led me further and further away from what I wanted to do, and since few of the tests actually worked, the whole thing became quite discouraging. Fortunately, Nontroppos “scrivomatic” solution worked better than the Terminal detours, and what I have now is joyful to use. :slight_smile:

I thought the replacement of ((12)) to [size=90](FIGURE 12) [/size] would be difficult, but it was way easier than I thought. Solved in 2 minutes.

Fleurons turned out to be WAY more difficult than expected. Adding the glyph as a suffix was easy, but getting an empty row before and after the glyph is not a trivial problem. Right now I’m getting no empty row before and three empty rows after…

Automatic double returns… I have no idea so far.

Remember to use ALT+Enter if you want to add newlines in Scrivener’s edit boxes in the Compiler. I wish Scrivener made it easier to visualise and enter newlines, but for the moment you need to scroll within the edit box and use ALT+Enter.

For proper paragraph spacing, maybe this thread helps: viewtopic.php?f=2&t=65564

I’m trying to have Scrivener (Mac) output to an icml file. Unfortunately, part of the infos above are no longer accessible, but I could rebuild them somehow.

I started from the “Basic Pandoc” compile format, and added these arguments to the Processing pane:

-s -t icml -o <$outputname>.icml <$inputfile>

Pandoc is installed in my usr/local/bin folder.

I always get a folder with the name of the compiled Scrivener document, containing the linked and embedded images, plus an .icml and an .md file. The .icml file can be placed by InDesign.

But it doesn’t contain any of the original paragraph and character styles. They indeed aren’t in the .icml file. They are all reassigned to a few default styles.

image

Is there something else that can be set, to have the styles included when compiling?

EDIT: I matched the Scrivener styles with the compile format styles. But this still didn’t translated into styles in the .icml file.

Paolo

ICML is a native interchange format for InDesign. It should preserve things like linked images, paragraph and character styles, tables.

The only other way of exchanging data that I can think to is RTF, but this one wants to embed images, and always to something to the tables. Granted, Scrivener has no table styles, but when exchanging between, say, InDesign and Nisus they are lost.

EDIT: I just notices something about placing in InDesign an RTF file generated by Scrivener. I have all the text after a linked image broken and apparently replaced by infinite page break codes. This is how it looks in Scrivener:

And this the result in InDesign (with the text broken after “Zoom In”):

Paolo

I’ve the same issue. I see the paragraph break symbols in the Scrivener document, but then some of them do not translate into the .icml file. There is no particular recurring situation that I can see.

EDIT: Solved! If you check the “Convert rich text to MultiMarkdown” in the main convert format dialog, paragraph breaks seem to work fine. Don’t ask me why.

I was sure this option was checked by default, but it wasn’t here. As far as I can view, it also converts some default styles (like the table cells and headers ones) there weren’t before.

Paolo

You shouldn’t need to use “Convert rich text…”, something else probably is going on. If you can make a minimal project and zip it up we can take a look.

Just to be clear here, you do need to use the Styles panel of the compile format editor to map the styles to use the custom-style attributes required for both block and inline styles. My scrivomatic sample project does include ICML output you can read through: scrivomatic/workflow.icml at master · iandol/scrivomatic · GitHub

So for example, an Infobox Scrivener style:

is given a prefix of \n::: {.info custom-style="info"}\n and suffix of \n:::\n by the compile format (remember you need to use opt+enter to enter the newlines and Scrivener makes it had to see the content of the prefix and suffix given the ui uses a tiny single line edit box):

and generates this XML with the AppliedParagraphStyle attribute:

<ParagraphStyleRange AppliedParagraphStyle="ParagraphStyle/info &gt; Paragraph">
  <CharacterStyleRange AppliedCharacterStyle="$ID/NormalCharacterStyle">
    <Content>This is an info box, containing fascinating information.</Content>
  </CharacterStyleRange>
</ParagraphStyleRange>

Yes: not all the styles were defined, and not all of the defined one have newlines at the beginning and the end. When each paragraph is surrounded with newlines, after a style has been applied and converted, paragraphs correctly break.

It wouldn’t indeed be bad if the ‘\n’ was allowed in the compile format fields, since it would make reading the prefix/suffix much easier.

Paolo