Broken Captions for Binder–Linked & Embedded Images

Scrivener Scrivener for macOS
21

For the former part, is that not the desired result? In fact escaping these is precisely the sort of thing escaping them is meant to accomplish: an identical result post-Pandoc/MMD processing. Here we have the input, written by an RTF-based author with some citation tags highlighted for clarity. They are going to want these tags identical on output so that Word/etc. can scan and format them:


[size=80]Above: Pandoc to Word DOCX as seen in LibreOffice; Below: Original text in Scrivener[/size]

Now, if we didn’t escape them they might very well have been passed through cleanly because these snippets here don’t collide with any Markdown usage—but can we always guarantee that will be the case? It may be unlikely, but since the actual result is entirely what one would want here, what’s the problem with escaping them?

As to the latter part, my point above about not wanting visible internal caption codes in your KF8 or PDF still stand, applied to this as well. You wouldn’t want visible LaTeX, MathML or whatever codes there either—you’re probably using MathType, or some other mechanism for inserting equations as pre-typeset images into the input.

22

If this only to be scanned in i.e. Word, then yes, but if you use CiteProc (how pandoc or other markdown bibliographies work) or Bookends on the markdown, then you end up with a broken citation. Here is the output taken from Word. Note with escaping, CiteProc did not understand this was a citation block, left the in place and thought these were naked citations so ( ) the year instead of the citation itself (CiteProc allows such “naked” refs so you still get some conversion, just incorrectly):

(If I use Bookends ref manager I get the same type of incorrect output)

Pandoc and MMD both have mechanisms that handle maths for you (Pandoc in particular has excellent support, directly converting to Word’s native equation format for example), from simple sub/superscript equations, inline and block. If you want to write in rich text but want to use this standard TeX maths notation, escaping breaks them:

out.png
out.png1332×1200 72.3 KB

When trying to convert the escaped scrivener compile you get this warning:

> pandoc --bibliography="$HOME/.pandoc/Core.json" -t docx -o blank.docx blank.md [WARNING] Could not convert TeX math 'e\^\{ix\}=r(\\cos \\theta +i\\sin \\theta )', rendering as TeX: e\^\{ix\}=r(\\cos \\theta +i\\sin \\thet ^ unexpected "c" expecting "%", "\\label", "\\nonumber" or whitespace
Without escaping there are no errors.

The point is anything that will be processed at the markdown level will break. If what you want is only processed in Word, then it is OK, but it is also OK for 99% of cases if you don’t escape because it would be rare to get a collision. A setting in compile would solve it for either use case!

23

I’m aware of equation handling, but with this kind of rich text workflow, if you are going to be taking advantage of this capability wouldn’t you not want to put these into an Equation style, or maybe even a section type for them?

Never mind the utility of doing so, you aren’t going to want $$^{ix}=r blah blah$$ in your Kindle file, so maybe you want your MathType or PNG LaTeXiT equation instead. Throw both into alternating style fields and omit one or the other based on compile format.

The main point that I’m trying to make here is that I think with the sort of writer that desires what this checkbox represents, they are going to be using very little if any actual markup in the editor—and if those uses of markup are detrimental to the output of other formats, you would want some way to conditionally omit the markup, meaning you’ll want to use styles—and since you’re using styles you can use the raw markup feature.

I’ll freely admit not being that sort, and so I’m presuming a little here—but this checkbox was made for the many people we’ve had writing in over the years that were disappointed when they found out LaTeX required one to use markup. They had no interest in it, no curiosity about adopting it as a writing method, and many wouldn’t even want to use it in little bits. They were merely interested in Scrivener’s LaTeX capabilities because they saw it on the site and didn’t realise you had to change everything about how you write to use it.

Perhaps there will be some that want to use a little markup, perhaps there are some that prefer markup and want some aspect of what this checkbox provides—I think we’re okay for that though with the system that provides on-demand markup pass-through—especially since most of the cases I can think of where you might want to do that, semantically marking that text would be advantageous and congruent with the workflow anyway, one doesn’t even have to go out of the way. It’s no different a workflow than the eBook oriented author that wants to use a little raw HTML now and then and makes use of the very same checkbox for doing so (or historically, Preserve Formatting).

I think we should wait and see how the intended audience takes to it. To me it all boils down to the above: it’s a checkbox aimed at people also using the rest of the Compile For menu—and once you accept that fact then the concept of using raw markdown in your editor drops off rather steeply I would think. Maybe I’m wrong!

24

Well, I suspect we have come as far as we can on this topic :smiley: I respect the underlying reticence to add another setting — each additional setting has an implementation and complexity cost (I suspect toggling escape parsing is easy for Keith, but maintaining it and documenting it less so), and I can only hypothetically imagine an academic scrivener user who wants to use citations or TeX maths simply without fussing with styles; perhaps this user does not exist! :stuck_out_tongue: And if they do then the recommendation to that person is they should use semantic styles for things that will be broken by a rich text conversion. My instinct is nevertheless to offer them more flexibility, but I don’t have to maintain the additional compile feature… :wink:

25

If I may reopen this discussion, I would describe myself as …

… and, without going too philosophical here, I’m fairly certain I do exist :wink:

I really like nontroppo’s suggestion above, to have in the Compile Options another option, aside from “Convert Rich Text to Markdown” where one would be able to specify whether or not to escape markdown.
Even better, if that would be a textbox where I could define myself which characters I’d want to escape that would add amazing flexibility for those of us who are really into hybrid best-of-both-worlds approaches. Let’s say, I do want to escape asterisks but not brackets, curly braces, and backslash.

I’m a LaTeX user, but I hate to write in raw LaTeX, so converting rich text to markup is the way I’ve been doing it so far.
But there is occasional LaTeX markup in my text and I also \input{} other files directly in Scrivener (which I don’t need to see before the final output) – I do have styles defined for these code spans/blocks, yes.
And I also need citations/references which I enter in Pandoc format. I know I’m able to either convert [^ … ] back to [^ … ] during post-processing or define a character style that I apply to references that I then “Treat as raw markup”, as AmberV suggested. However, both of these options involve extra steps as nontroppo’s rejoinder points out.

At the moment (starting my first big project after updating to Scrivener 3), I’m probably going to define semantic styles for emphasis, strong, etc. and rebind the keyboard shortcuts. However, what if I get a document from someone else which I want to incorporate, copy and paste from a different program, or import a rtf/doc, etc.? Then I’d always have to make sure to manually convert italics and bold to my character styles. Here it’s be really great if I could just let Scrivener handle this upon compile.

Scrivener actually seems to follow this hybrid approach when exporting to Fountain format. Here strong, italics, and footnotes all get converted to Fountain markup, while other stuff like brackets or asterisks are not escaped.

So you certainly have my vote for that extra setting.

26

You might be okay with what you’re currently doing, but do note that in the 3.0.3 update we now support an alternate LaTeX workflow that does not make use of any intermediary visual markup. You’ll find the demonstration as a project template in the Non-Fiction category. One could use it very simply and type in most LaTeX raw, but as I understand you would want to abstract from that as much as possible, you would be interested in the various techniques it uses to generate LaTeX syntax from formatting. It isn’t comprehensive (I doubt anything could be!), but as a palette of techniques that can be adapted and refined to build more specialised workflows depending on what you need, it might be what you’re looking for.

At the least, since it is a pure LaTeX workflow, you can just type it in whenever you want, rather than having to escape it with MMD/Pandoc markings.

Speaking of which, I don’t quite follow your aversion to using styles for raw LaTeX here, seeing as how the alternative is not simply being able to type in:

\input{name_of_file}

But rather, without any styles adding the escape codes as prefix and suffix:

`\input{name_of_file}`{=latex}

Surely it is easier to hit a style shortcut (making this as simple as writing an italic word) than typing in all of that extra stuff?

Well, like I said above, I don’t really get why anyone would want to avoid semantic markup in a program that offers it, while working toward a system that thrives entirely upon the concept of semantic markup, using an intermediate system of markup that is itself fully semantic. Diversity is the grist of a healthy civilisation however, so while I may not fathom it, I respect it. :slight_smile:

You would do as you would in any stylesheet-based workflow in any word processor: select by formatting ranges and convert. Any attempt at doing this automatically is extremely complicated because you are asking software to at some level achieve an awareness of intent through variables as diverse as point size shifts and letterform variants.

In Scrivener we have a few tools for this. The Edit ▸ Select ▸ Select Similar Formatting command will be the most useful to you. A simple two-step approach of bulk selecting all italics and then using your modified Cmd-I shortcut to upgrade them, or clicking on “Emphasis” in the Styles panel, and then repeating for bold, and now you are presumably done and can move on with a more elegant and flexible document structure than a thing that merely has meaningless font settings strewn throughout. :slight_smile:

I wouldn’t myself so easily compare an extremely specific format like Fountain, that only has one purpose, with a general purpose system that has applications as diverse as web, ebook, scientific, technical, formal print, programmatic application and so forth and so on. The systems you are using are broad and powerful, and as such they require human clarity to be made sense of, to hone into a desired output. Fountain is one thing for one group of writers that never deviates from one standard approach.

27

In Scrivener 2 there was a Format > Convert > Bold and Italics to MultiMarkdown Syntax option. I found this post while looking for the Scrivener 3 version of it, and now I’m concerned that the sudden disappearance of my “hybrid” workflow means that this conversion is now impossible. I’ve tested it out already and it seems that all my other Markdown formatting is destroyed (by escaping brackets and some other characters) when using the Compile option to convert rich text to MultiMarkdown.

Since it’s normally possible to type a bracket or a tilde in a Markdown file without escaping it, I don’t understand the point of escaping them. Yes, accidents may happen, but how often do you really expect someone to type a sequence of Markdown formatting characters and not mean it? (That is, ones that wouldn’t have already been treated as the literal characters.) It seems like the case of intentional hybrid use of Markdown is much more likely.

28

If you’re writing in Markdown, the expectation is that you are also using asterisks for bold and italics. The old Convert feature of Scrivener 2 was never intended for a hybrid workflow, but to allow for a basic conversion for those wanting to convert rich text to Markdown. Scrivener 3 offers full rich text to Markdown conversion.

29

There may be some confusion of which sort of tools we’re referring to as well. The closest thing to what you are describing in Scrivener 2 is now the Edit ▸ Copy Special ▸ Copy as Markdown, menu command. Granted, that also assumes the source text is rich text and not Markdown, but that’s going to be closer to what you’re describing than any kind of compile option.

To be fair the old v2 command also assumed the source material was rich text as well, it was just a very basic and incomplete conversion. One would still have to go through and fix their paragraph spacing and all other forms of formatting other than italics and bold. Now one might only have to do a few things to fully convert over to Markdown.

To ensure the document outputs as it was typed in. We might be able to know, in a contextual sense, when it is “safe” to type in a raw punctuation mark that has special uses in Markdown, but unless an algorithm is fully aware of what Markdown is, that’s not going to be easy to do. Thus the safest approach is to escape anything that might accidentally become something meaningful.

What I don’t understand is, how is this any different than selecting “Compile for: HTML” and asking why Scrivener converts “<” into “<”? If you really want what you typed in to be treated as raw HTML, you mark that text with a special style so Scrivener knows to leave it alone.

Out of curiosity, if all you ever wanted was bold and italic fonts mixed in with raw Markdown otherwise, why not just use styles for them, going forward? You get the best of both worlds that way:

30

Since I don’t want to copy anything–I want to convert parts of the document itself to Markdown–that doesn’t sound like a replacement, either. And I think there was a way to get paragraph spacing automatically later just in the conversion to plain text, though I don’t recall the details.

Well, HTML is a strict format that is easily broken by a misplaced reserved character; Markdown is a loose format that is intended to be human readable and thus not full of unnecessarily escaped characters.

It looks like I can do that and even save it to a template, but it doesn’t necessarily help with other RTF-to-Markdown post-processing–for example, paragraph breaks, though I know there’s a different way to handle those.

My situation is not one where I (or, more importantly, less savvy users of my template) write a pristine RTF document that I want converted into the exactly analogous Markdown; it is one where I am more or less trying to write plain text, including lots of Markdown and other weird stuff, in an RTF editor that has no pure plain text mode. Because of that, I may be tempted by the Bold button, or some accidents may happen that will be hard to spot and track down later. The Scrivener 2 Convert bold/italics… menu item was one handy tool for that situation, so at first I was just looking for it in v.3.

Scrivener 3 seems to have a lot more power to convert accidental RTF to plain text automagically, but sadly a design choice about escaping things that most people just don’t type without meaning to has made a lot of that power inaccessible to me. It’s more frustrating than anything else.

31

Copying with a transformation filter is converting, if after copying you paste. For example a way of stripping out all annotations and comments from a portion of text once you are done with them is the “Copy without Comments and Footnotes” command, in that same menu.

Indeed, that is still there as well, in the Transformations compile option pane. We’ve made it better in fact: now it can also work off of formatting settings implemented by the compile settings, so you don’t have to have artificial spacing that mimics literal whitespace in the editor unless you want to.

The conversion I’m talking about above however corrects paragraph spacing no matter what your formatting looks like, and it does so in a single command, not via a dozen different moving pieces that all need to be set up together and then exported from the project. With the new copy command you can take a colleague’s RTF data and get a passable Markdown document out of it with two commands (copy and paste).

A fair point from our perpsective. But to reiterate, keep in mind Scrivener cannot know the difference between a “safe” special character and one that is meaningfully used; it would have to achieve a level of parsing awareness that would equal Pandoc or MultiMarkdown in order to do so. So lacking that, from its perspective Markdown is more strict than we as humans would perceive it to be. Blind escaping is the only really good solution.

This isn’t about making pretty Markdown files for the most part (although we did spend a fair amount of effort making sure it wasn’t horrible), this is about opening up technical formats to authors that don’t use Markdown. If you need to provide a publisher with a .tex file but you’ve never done that before, you’re not going to care if the intermediate source file used to generate the .tex file is, in a purely aesthetic/ideological sense, over-escaped. :slight_smile:

It’s a bit academic, since as you say there are probably better ways of doing that, but Styles could in fact do that. :slight_smile: Create a “Body” paragraph style in the Styles compile format pane, have it do nothing but add a carriage return after each paragraph. Then in the Section Layout that works with text, set it to override formatting for notes and text and then assign this “Body” style to all text that isn’t already styled.

Perhaps in some cases that might be preferable to the whitespace conversion feature, which is all or nothing.

Perhaps that is a problem that will wane in time for you (and if you’re writing pure plain text, maybe consider turning off the Format Bar—I mostly keep it off, as for me it’s 99% UI clutter, and it has a handy shortcut for those occasions where I need it to change highlight colours or something), but one solution to solving it would be the Edit ▸ Select ▸ Select Similar Formatting menu command. With a bulk selection made, conversion to the style based approach would be simple. As the sort of thing done a few times at key points in a project’s lifespan, that doesn’t seem to me a significant burden.

32

Wouldn’t I have to do copy and paste on a per-document basis? That’s not practical because there could be hundreds of documents in a draft.

The only reason there would be RTF data around in my plain text workflow is because the text was written fresh in Scrivener and things got RTFfy, as it were. I’m not sure I really follow what dozen moving pieces you’re thinking of; Format > Convert could be applied to the whole document and I don’t recall linebreaks being complicated, either. I did try out the new Transformations section.

If part of the appeal of using Scrivener is the ability to get Markdown out of it (not just a post-processable intermediate MultiMarkdown step than no one will ever look at, but a Markdown file you or other humans can work with later), then it does matter what the Markdown looks like. But the bigger problem is, as others in the thread have mentioned, that the escaping process makes it impossible to put Markdown into a document that might also contain some RTF that you want automatically converted. A simple option to take my own chances with possibly unescaped characters would solve that particular problem without the effort of recreating large swathes of the Markdown spec in the form of new styles.

I already write in plain text; my question arose from updating the documentation of my Scrivener 2 template which took into account different workflows. Generally speaking, if it takes more effort that making a menu selection or checking a checkbox, then I don’t think I’ll be documenting the approach. Initially I was hoping I could remap the actual bold and italic buttons to styles under my (template’s) control, but I got the impression somewhere in the forums that that is not an option, either. Trying to explain that users can’t use the bold button to make things bold (not to mention warning them not use the convert to markdown option to convert to markdown) is a job I don’t envy you.

I am excited about using some of the other new options in Scrivener 3, though; they may simplify my workflow considerably.

33

This is great, thanks very much. I played around with it a bit; it’s certainly well-designed for a pure LaTeX workflow. However, while LaTeX is the main format I’m exporting to, I’m going to stick with a pandoc workflow for my project for the following reasons:
(1) I really like to enter citations in the pandoc format as it’s so intuitive and easy.
(2) I have my LaTeX project fully set up and only want Scrivener to supply the main text of each chapter, i.e., everything between \begin{document} and \end{document} while also \input{}-ing some LaTeX files generated elsewhere.
(3) I do export parts of what I’m working on to Word, for comments from advisors and peers. All I need in those Word files is the text and citations/references and pandoc with citeproc does a fantastic job here.

You’re right and you’ve convinced me :slight_smile: and I have now created a couple of paragraph and character styles for the raw markup. That then also allows me to convert rich text to markdown for italics and bold. As far as the other discussions in this thread are concerned, I nonetheless would support the call for allowing more custom control over characters to be escaped and not escaped.

In any case thanks for your valuable help!

34

You must be misrembering. There as nothing like you describe in Scrivener 2, and the option in the Transformations pane is not new. It has been around for eight years. Additionally the conversion command you refer to only ever did one thing, asterisks around bold and italics. All of these other things you are talking about required a dozen moving pieces.

Now they do not. Scrivener 3 simplifies full conversion a great deal for those that need it.

That is possible, in fact what I was suggesting you do earlier, as an upgrade to your old conversion-based workflow. If one forgets not to use bold and italics while writing in a plain-text system, then it seems sensible to me to reroute those shortcuts to something useful. Use the standard instructions for custom shortcuts in macOS, and target those Format ▸ Style ▸ Emphasis and ▸ Strong commands. In projects that do not have those styles, the system will fail to rebind and allow the shortcuts to failsafe back to their defaults, Format ▸ Font ▸ Bold and ▸ Italic.

That is certainly the idea behind the aforementioned approach. Like I said above, the old method meant you had remember to run conversion commands to take unsemantic RTF markings and turn them into Markdown, and then proof the result. This is a clumsy way of working, and we felt it could be optimised and made more direct. Now you can cut to the chase and wire your bold/italic commands directly to semantic styles that correlate automatically to correct Markdown when you compile. In the net, that’s a simplification of how you write. Now your tool produces Markdown for you with no intermediate conversion layer.

I think overall you’ve acquired a misunderstanding of what is being suggested here: that one must engage in “recreating large swathes of the Markdown spec in the form of new styles”, which nobody is advocating for.

⠂─────── ⟢⟡⟣ ─────── ⠂

Sounds good to me—I’m in the Markdown camp to be clear. I wouldn’t want to write in pure LaTeX myself—and the main reason for me is that I need other things than just LaTeX. Markdown is a nexus format for me, not just a conversion layer to one output.

Yeah I think that’s the main problem here, and why we are concerned about turning this checkbox into something new: a tool for Markdown authors. The latter is so diverse and complicated in terms of just what that means. Right now it is very simple. If you do not know what Markdown really is and have no desire to learn, but you want Scrivener to create a DocBook XML file: then you can these checkbox to bypass the technical bits.

Otherwise you use Scrivener like you have since version 1 or 2, with maybe a few small adjustements to your settings and habits.

I’m not sure if one optional disengagement is going to magically make this work for everyone trying to find some middle-ground. It’s why I think the buffet approach that currently exists is a better option. You can make these paragraph styles you needed to bridge the gap, and now the software works the way you need it to. Someone else can make an italic style to bridge what they perceive as the gap. We would need a battery of checkboxes as long as what already exists in that tab, to address all of the variation of what “hybrid” means to everyone.

35

Well, again, I don’t know what dozen moving parts you’re talking about; I was not dealing with any of them besides a straight conversion of linebreaks to plain text (however Scrivener 2 was doing it for me) and the menu option that turned actual RTF bold into actual Markdown bold markup. So while I’m sure the changes have helps lots of people, my only experience of them is how they are interfering slightly with my simple workflow (by making the more primitive menu item for bold/italic to markdown conversion disappear and making the new bold/italic to markdown conversion method useless for my use case).

Yes, I got that part; the specific thing I was thinking of was remapping the buttons, which might remap the shortcuts automatically and would certainly be more idiot-proof. Not to say that users are idiots, but we’re not all necessarily interested in learning that much about Scrivener, at least not up front.

The issue there is that if I want to use the new compile step that does automatic conversion of RTF, especially bold/italic, to Markdown, then it will also break any Markdown I already have in my draft. Ergo, I would need to switch to an RTF-centric workflow to use it, and unMarkdownify all my existing Markdown using styles, document links, or whatever, in order to prevent the Great Escaper from escaping it. Depending on how much markup someone was actually using, that might be a practical approach, or it might involve “recreating large swathes of the Markdown spec in the form of new styles.”

However, I have other issues like inline freeform code in my draft that would never survive great-escaping, so an RTF-centric workflow isn’t a serious option for me. I’ll have to stick to something more plain-text oriented.

I’m not sure what a second optional disengagement would be. The proposed extra checkbox is pretty binary, just not escaping plain text characters (that I put into my draft intentionally, in full knowledge of the target plain text language and its foibles–or at least I’m willing to take my chances that they were intentional) while still escaping actual RTF markup. It’s not really about making every hybrid workflow work; it’s just about not unnecessarily breaking the ones that have turned out broken so far.

I agree that a hybrid workflow is not optimal, your suggestions are a better approach, and the problems mentioned in this thread seem pretty work-aroundable. But part of the appeal of Scrivener is that you don’t (usually) have to commit to learning it/doing it right all at once; you can improve your setup gradually. That’s harder when some intermediate (e.g., hybrid) step is unexpectedly missing or broken.

36

Yes, the whole point of the option is that it is for users of rich text. If you’re not using rich text, then you should be using Markdown (or MultiMarkdown throughout). Scrivener has never been designed for use with a hybrid of both, that is the whole point.

All the best,
Keith