Disappointed in "Convert Multimarkdown to Rich Text" Compile option

Scrivener Scrivener for macOS
1

I love Scrivener. There’s nothing else like it. No others software gives me as many ways to look at a large project and tackle it—and to work in with other software (Aeon Timeline, PWA, iThoughts) when I need to. Ordinary Compile handles everything I want from output—I have no interest in the various MMD->{strange postprocessing format} compile options.

I also love Markdown and Multimarkdown, enough that I’ve switched to using a Markdown-based iOS editor via External Folder sync.

With those preferences in mind, I have to say that the “Convert Multimarkdown to rich text in documents and notes” compile option is a disappointment. Not that I haven’t found ways to work around it, but there it is.

This works only if I use an “as-is” section layout. In general, I don’t.

MMD constructs that actually work with a general (read: built-in) Scrivener compile format (tested with the Modern format, Titled Section layout):

  • Italics
  • bold
  • Markdown images
  • external links
  • MMD Tables
  • Bulleted and numbered lists
  • Footnotes might work if I really needed them and I had time to play with them. (But Scrivener inline footnotes work with no problems if the double-braces format is used on sync export, so why bother?)

On the other hand, these MMD constructs don’t work with a general Scrivener compile format:

  • Inline code.
  • Code blocks.
  • Block quotes.
  • Heading levels

These features either change paragraph formatting, or font, or both. They are destroyed by any section layout other than “as-is”—even if there’s a corresponding named style in the compile format. (I’m looking at you, block quote. You too, Heading 1.)

In short, I’m forced to a mix of Scrivener styles and MMD. Block quotes are particularly painful. Needless to say, if I edit any styled paragraph with my external Markdown editor, the styling is removed on the return trip to Scrivener and must be re-applied.

What I’d like to see for this option in future is the ability to somehow connect a style in the compile format to an MMD construct. In the meantime—I put up with it to have my Markdown editor on iOS, and glorious Scrivener structuring on Mac.

Thanks for your time reading my rant.

2

Honestly, Scrivener is never going to be a Markdown editor so anyone determined to use Markdown in it will, I’m afraid, have to live with the fact that it’s not ultimately built for that, but just has some options provided for those who are happy to live with this and want to use Markdown regardless. If you’re using Markdown, you are always going to get the best results by exporting to one of the dedicated MultiMarkdown or Pandoc formats. The “Convert to rich text” options are convoluted: they have to pipe your text through MultiMarkdown to generate HTML, read that into rich text, and then add a bunch of tags to try and maintain various formatting not supported by Markdown at all.

Even with those options ticked, you are right that if you want code blocks, you will need to assign a code block style; Markdown formatting for code blocks will not be converted. The problem here is that, when reading the HTML back into rich text, the rich text reader has no idea about code blocks. I would essentially have to write some much more complex dedicated routines either to pre-process and post-process this stuff, or to examine the raw HTML and try to tease out information from that as well as using the standard HTML > rich text converters.

And I’m afraid I can’t say that any of that is really on my radar for development.

All the best,
Keith

3

I rather expected this answer, Keith. :smiley: Thanks for reading through, and for your reply.

4

That’s not to say that I won’t have a think about it. It’s just very tough, because in order for Scrivener to recognise those blocks, it essentially needs its own Markdown interpreter before it processes everything else. i.e.:

  1. Lots of pre-processing (as now), but with:
  2. Recognise Markdown code blocks, spans, block quotes and heading levels, and mark those up in a different way.
  3. Convert to HTML using MMD.
  4. Read HTML into rich text using standard Cocoa routines.
  5. Lots of post-processing (as now), including converting stuff from (2).

(2) is the tough step here.

Alternatively:

  1. During (4), analyse the HTML and look for code blocks, headings etc.

This is also a tough step.

In either case I either need my own Markdown or HTML parser - that’s the issue.

All the best,
Keith

5

I completely understand. Since (so far) I believe that I’m the only person who’s posted a comment about this in the forum, it would be a lot of work for a small audience.

For anyone else who might come across this thread, here are a few of the workarounds I’ve tested that work, in the sense that they flag styled text in the Markdown editor so that I can avoid editing it if possible (and find it easily in Scrivener to reapply formatting if needed):

  1. Tagging the front of any styled paragraph with an inline annotation describing the style. Generally, this works great, though all the styled paragraphs look alike in the Markdown editor. It has the virtue of simplicity—I delete inline annotations as part of compile, and everything works as expected, both sides.
  2. Using a special style (I call mine “soft delete”) which is deleted upon compile. I then use this style for Markdown tags (like Block Quote and the various headers) in Scrivener styled paragraphs. The Markdown editor displays the text as whatever I’ve tagged it, and Scrivener deletes the tags before processing. This is perhaps my favourite as I can easily see my styled text as styled in my Markdown editor (I’m very visually oriented.) It has the disadvantage of needing to edit my compile formats to include that “soft delete” style.
  3. Putting a styled paragraph inside a fenced code block, and tagging the code block with the name of the style (instead of the coding language.) This makes the styled text painfully obvious in the Markdown editor, but can’t be used if there’s any internal markdown in the paragraph (like italics, or a link.) No need to do any compile formatting editing for this approach, either.

I’ve toyed with the idea of splitting styled paragraphs off into separate documents, and writing new section layouts that apply the proper formatting. But this seems like a lot more work on the compile format editing front, and, well, the Markdown editor I’m using isn’t as happy about splitting documents into small chunks as Scrivener is. So I’ve shelved this idea for now (not tested.)

6

This in theory can be solved by using an abstract syntax tree (AST)where the specific syntax for markdown/html/latex is turned into an abstract set of inline and block elements, a semantic representation (in a way, a cross-document “style”). That is the core underlying technology used by Pandoc, a series of document-format specific readers and writers who speak via a common AST. It means you can link a markdown code block ↔ html code block (or latex, docx, odt etc.). In between, filters can modify or change any logic in the mapping: you can access or manipulate the AST as you see fit. The AST itself can be exported and imported via the command interface, i.e. Pandoc can give you the AST, and then consume it later.

I know you already use MMD, but perhaps Pandoc would give you a bit more flexibility in this regard. I even wonder if a custom writer could be made to generate RTF directly from the AST, that possibly may help in the final problematic step you have of HTML ↔ RTF…

➜ pandoc -f markdown -t native
This is a `test` of inline code.

[Para [Str "This",Space,Str "is",Space,Str "a",Space,Code ("",[],[]) "test",Space,Str "of",Space,Str "inline",Space,Str "code."]]
➜ pandoc -f native -t html
[Para [Str "This",Space,Str "is",Space,Str "a",Space,Code ("",[],[]) "test",Space,Str "of",Space,Str "inline",Space,Str "code."]]

<p>This is a <code>test</code> of inline code.</p>
➜ pandoc -f html -t native
<p>This is a <code>test</code> of inline code.</p>

[Para [Str "This",Space,Str "is",Space,Str "a",Space,Code ("",[],[]) "test",Space,Str "of",Space,Str "inline",Space,Str "code."]]
7

Maybe I would like to make this into a (very small) complaint: The actual experience of using “Convert Multimarkdown to Rich Text” does not live up to its “can be used with one of Scrivener’s own native formats” billing. It’s true only so long as I don’t use an MMD construct that changes paragraph format, or font, or both. The list of these is short, true, but it’s not what the manual description led me to expect. Rather than spending weeks of coding time to correct this in the app, perhaps just an edit of the manual’s description would be in order so that readers don’t expect block quotes, for example, to work.

Just a thought. :smiley:

8

@Silverdragon, out of curiosity what is it that makes you prefer this somewhat convoluted workflow to just using Scrivener on iOS. I obviously love markdown, and use it everywhere, but given the centrality of Scrivener to my workflow, using Styles and keeping within the ecosystem is a compromise I’m willing to make. But most of my writing is done on my Mac, and I only really use tablets for notes and ideas, not serious writing. I’d doubt another editor, markdown or not, would change this for me. I simply prefer a traditional computer as scientific writing requires a lot of resources and switching apps that is insufferable on a tablet…

9

Hmm… I was persuaded by your arguments for quite some time, but I finally got too frustrated with all the things that iOS Scrivener doesn’t have. If I’m going to have to switch apps anyway, to (for example) look at my research clippings or check my keywords or even check the date/time of the event I’m about to write, why not write in the editor I prefer? Then I got stuck in the quicksand of … “Now why does THIS mmd construct work, but not this OTHER mmd construct?” And since I researched it, I wrote some blog posts about it (in my preferred Markdown editor).

In short, I got sucked into the Shiny Technology Vortex, hence this thread.

Back B.I.S. (Before iOS Scrivener) I used an external folder sync workflow on iOS that used the Editorial markdown editor (now effectively abandonware). I even sometimes edited in the synced folders on Mac with BBEdit instead of Scrivener. I liked the workflow a lot, and miss it. I most especially miss the fact that in Markdown I can always tell exactly where a format begins/ends. (I like to have everything about my documents visible. iOS Scrivener doesn’t even let me show invisibles.) That was in the days of Scrivener 2 as well, so styles weren’t even a thing. There wasn’t a compile option to “Convert Multimarkdown to Rich Text in Text and Notes” (which is, as I’ve found, misleading.) Instead there was a compile option to convert italics and bold to rich text. Italics alone get me 95% of my non-default formatting needs in the body of any document. There’s block quote for the occasional memo/text/email that I want my character to receive/send. I had an editor demand small caps once. The rest of my use of non-default formatting happens only during compile (section titles, etc.) or in front/back matter—title pages, acknowledgement lists, and the like.

In short, I could so do without Multimarkdown. I’d be happier if there were an option to just translate italics and bold as a compile option, the way it was back in the Good Old Days of 2015. But that’s no longer possible. If I want italics and bold, I have to use MMD->Rich Text compile. To use MMD->Rich Text, I must double-space my paragraphs, and I must use MMD lists and tables, because Scrivener lists and tables are hosed by MMD conversion, and without double return characters my text becomes a uni-paragraph. I don’t mind much about having to tag my block quotes, etc. somehow—that’s a new innovation I’ve just come up with, and I don’t need it much because 95% italics. (Before, I just sucked it up if I accidentally hosed formatting, because it happened so seldom once I got italics and bold into their asterisks.)

But now that I’ve researched it, converted my paragraph breaks, and tagged the Scrivener styles I still need to use, it’s not as convoluted as it sounds. External folder sync is fast, and my iOS markdown editor swallows it without a burp. I can go back to my Markdown editor (even – gasp! – on Mac, with Scrivener in outliner or corkboard sharing the monitor) and the garish syntax marking scheme I’ve developed. I’m happy because i know exactly where each format begins and ends. I can use regex find and replace to change formatting, as I used to do. All my text is displayed in Victor Mono font and rainbow colours. What I see is not what I get, and I like it.

10

nontroppo - I’m afraid you’ve completely lost me. :slight_smile:

Silverdragon - now I’m confused. Weren’t you saying only recently that you would rather the macOS version be more like the iOS version? But now you’ve left the iOS version because it doesn’t have everything the macOS version does?

11

@Silverdragon: thanks for the explanation, and your blogposts provide a ton of detail for those interested in this route.

@KB: :laughing: yes, I think AST is a computer science term that can make ones head spin. But the underlying idea is really similar to the idea of styles: a named bucket to store something in. Part of your friction in going from RTF to MMD is how to translate all the constructs between them, made more complex by the fact you need to go via HTML given the weak interoperability that RTF has. You’ve hand-rolled this, and I imagine that is pretty hard to do! Pandoc offers a general mechanism based on the fancy computer sciencey stuff to do this. Pandoc readers take a document and convert e.g. bold word into an abstract holder for a bold inline element. This is the semantic meaning without having any of the html-specific bits attached. Pandoc writers then take this abstract bucket and translate it to the specific output format. As an example, I have a custom writer that converts markdown into BBCode , so this post is written in markdown, and Pandoc uses a custom transformer pastes out BBCode 8)

If you can get your custom RTF translated to HTML, then Pandoc may help in getting the HTML reliably into an extended markdown output and back to HTML. But then again if the hardest bit is getting the RTF into HTML meaningfully in the first place then Pandoc can’t help (it can write RTF, but not read it). For example, say you have a named “code block” style, your parser needs to get your custom scrivener style (which you inject into the RTF right?) translated into the html block (textutil etc. doesn’t understand the styles). Pandoc can’t help you in this first part…

12

I’ve reworded the description of the option a bit, to better emphasise that it only supports a limited subset of the Markdown specification on conversion.

13

So, I don’t know if this helps in the original scenario, but I’ve been using custom styles to output escaped LaTeX code for quite a while now. I’ve transitioned to styling everything except for body text (including numbered / bulleted lists).

To make my standard paragraphs gain an extra empty line after them, I created a custom COMPILE style called “body”. That style is there just so I can tell it to add an extra newline after every paragraph. Then I go to the compile formats, and apply that style to the body text of those formats. So in Scrivener, I don’t have to have empty lines after ever paragraph; the compile settings add them to un-styled text automatically.

That means I can’t use Rich Text lists, but I’ve given up on them wherever I can afford to; they just don’t line up the way I like them, and I’m okay with using asterisks and manually numbering them. For lists, I just apply a “list” style to them so that they line up, and so they don’t get the compile style of “body” applied to them. In the compile settings, the list of styles, I just let the code treat that style as pure markup, but don’t add any prefixes or suffixes to that text; they come out as plain text that essentially looks like what I typed in Scrivener.

I do wish that there was a better way to deal with tables; I have, in the past, created styles and Replacements in compile to re-format neat-looking tab-delimited data into MMD tables, but that is a huge pain, and it’s much harder to generalize for all possible table configurations.

14

Thanks, Ioa!

15

If you have that impression, no wonder you’re confused! I’m at a loss as to how you got it though…

  • searches her own posts *

Perhaps you got that impression from this post:
[url]Scrivener should become subscription - #24 by Silverdragon]
If so, I’m sorry; that wasn’t the impression I wanted to give. What I wanted to say was:

  1. Having non-text documents in a Scrivener project isn’t important to me, as an individual. Most of my research clippings are in Evernote. Of the rest, I’ve got everything linked already except cover images and a composition mode backdrop that I almost never use (because I only use composition mode to check my answer to a forum post, or check the look of a Scrivener appearance theme I develop.) It’s not a call for “more iOS Scrivener-like”, just an acknowledgement this is a feature I personally don’t use much.
  2. In Mac Scrivener, there are often several ways to achieve the same result. If you choose to reduce duplication, that’s fine with me as well. Again, not “more iOS Scrivener-like”, just “if there’s any fat there, sure, cut it.”
  3. I’d like to see the Mac interface move more towards an obviously non-WYSIWYG paradigm. That doesn’t mean dumping rich text, nor does it mean “more like iOS Scrivener.” But as this thread proves, WYSIWYG is very low on my list of desired features—it’s something I put up with to have Scrivener. And I believe (perhaps wrongly) that less WYSIWYG appearance might lead to less confusion for new users. (This is based solely on my own experience of how difficult it was to break the WYSIWYG paradigm in my brain. When it dawned on me that I could use whatever font I wanted(!!!) to write with and Compile would make my output look professional nonetheless, it was as if I’d been let out of jail.)

So there you have it. My preferences aren’t on a spectrum between iOS Scrivener and Mac Scrivener. They’re off the main sequence. :smiley: Make of that what you will.

16

For the record;

No, I don’t want iOS Scrivener on Mac. I love Mac Scrivener. iOS Scrivener I can do without, and have for the last month and a half.

Yes, there are Mac Scrivener features I can live without if I have to. That’s what the Great Big Scrivener Survey is for. :smiley:

17

I’m with you on Rich Text lists: the few lists I use all look better now that I use MMD for them. As for tables, I tested them with the MMD->Rich Text workflow, but I thank the gods that I just don’t need tables for novels. Yet.

Regarding your style that exists solely to add a paragraph break to each paragraph: I just tried this on my test project. Sadly, the MMD->Rich Text compile option removes the paragraph breaks before your style is applied, so there’s only one paragraph to append a paragraph break to (this confirms my observation and Keith’s assertion that styles/section layouts are applied after the MMD conversion.) So no, it won’t help if using an MMD->Rich Text workflow.

18

Weird! I could have sworn you were some time ago suggestion that Scrivener should be rewritten with Catalyst or something and be much simpler.

19

Thank you all so much for this thread! I started using Markdown years ago when Ulysses first pulled me away from Scrivener, and the ease with which it allows me to focus on my prose and nothing else until the last possible moment is why I continue to do my primary drafting in Ulysses.

I prefer literally everything else about Scrivener though. My problem with rich text styling is that I suffer from incredible decision paralysis about most things, and obsessing over how something looks is the last thing I want to do in my writing process. Not something I want to avoid entirely, just something I want to do, you know, last.

While I haven’t had the chance yet to directly implement any of these solutions, I’m very grateful to learn this is a thorn in more sides than mine.

@silverdragon Thank you, specifically, for sharing your pain points and workarounds.

20

That was someone else. I’m not sure who any more, and I’m not willing to go back through and find out. I think I did reply in that thread, something to the effect that if L&L made such a decision, I wouldn’t automatically abandon Scrivener but would re-evaluate its place in my writing. That’s a far cry from saying I want iOS Scrivener on Mac.

Of course, that was before I’d seen any Catalyst-transferred apps. If anyone wants to know how bad an idea this is, look at Notability Mac v. Notability iOS…

EDIT: Of course I couldn’t resist searching this. The Catalyst advocate was JoRo.