I could use some help how to add a pandoc citation with its own character style in an inline footnote. (Update: … or how to use any character style within a footnote.)
I have defined a style for citations which are “treated as raw markup” upon compile. Now I have a longer footnote with a pandoc-style [@citation] in the middle of the footnote text. That citation has the citation character style assigned to it. This is how it looks in Scrivener. The green shading is the citation character style.
When I compile with markdown, it splits the footnote into two separate notes and puts the citation into the regular paragraph. It ends up looking like this:
When I expect it to be looking like this:
Any way how this can be prevented?
Some additional info about my setup: I “convert richtext to markdown” throughout, and have defined a number of different styles (such as the citation character style) for anything that I enter in raw markup.
Update: I just tested with other styles. Any character styles used in a footnote seem to split the footnote into smaller chunks.
First part of footnote text [@citation]. Second part of footnote text. ↩︎↩︎
Thanks for the report. It looks like there are a few bugs in there. I ran some tests on a variety of different style settings, and ran into issues with the behaviour you describe, as well as when using styles to express formatting, which ends up not being converted to MMD (for example, a style that converts text to bold in the compiler will insert ** in normal paragraph text, but nothing happens within an inline footnote or annotation).
I did not however get splitting with anything other than the raw markup setting. Were there any particular settings I should try? I tried both prefix/suffix and formatting.
My setup is a fairly complex one. Now I’m trying to reproduce the issue with a new project, with only that one style, and now the issue doesn’t show up. Dang! (or, Yay!) So let me dig a bit deeper and I hope I can isolate which weird combination of settings produces the problem.
I found it! (wasn’t that hard.)
The splitting happens when the checkbox “Convert richt text to MultiMarkdown” is checked.
Try create a blank project, create a style for “citation” and then in compile formats, under “styles” check “Treat as raw markup”. Create a footnote with that style in it.
It splits the footnote as described above when the “Convert richt text to MultiMarkdown” is checked.
When it is not, then it behaves as normal.
I hope this helps in fixing it.
PS: Any suggestion for a workaround for the time being?
By disabling, you mean unchecking “treat as raw markup” in the style compile setting? OK yeah, that might work. At least I can continue writing the document with assigning the style to citations and then enable this again once it’s fixed. And add another regex to fix it upon compile.
Do you have any predictions though, how long it might take to fix it?
Yeah, it’s either one of those ingredients, I would imagine that fixing a few brackets would be less impactful than fully switching over to Markdown as a way of writing.
If something critical comes up it may be sooner, but otherwise no later than shortly before Apple’s launch of 10.14, to make available any compatibility fixes necessary for that upgrade, as well as to introduce the dark mode version of Scrivener we’re working on. I realise that’s a few months out yet though.
Unfortunately, I need to come back to this issue as well once more.
So I’ve now unchecked “Treat as raw markup” for my style Citation and have added a replacement pattern
\[(.?@\w+.?)\] to be replaced with [$1]
which in all tests captures every possible Pandoc formatted citation [see @citekeya, 23; @citekeyb]
however, apparently before Pandoc even sees it, the newly inserted brackets seem to have been re-escaped by backslashes. Could it be that replacements are done before the “transform rich text to markup” is applied?
I also checked both locations of replacements (the one in the compile format settings, and the one directly in compile on the right hand side of the compile window) and the same experience.
However, adding a similar replacement in the ruby script we’re discussing in the other thread, does indeed work. So it looks like “transform rich text to markup” is done as the very last step before passing it on to post-processing … which makes it impossible to do any cleanup in Scivener itself to work around the bug above.
It seems I’m not getting around using postprocessing scripts for all of this
When I tested during the beta, IIRC “Transform Richtext…” was the last action performed in the compile chain, and therefore escaping was not undoable except with post-processing outside of Scrivener. I’m not sure what the best order of operation should (style replacements | regex replacements | RTF conversions), but again, if we could disable escaping this would make for more flexible workflows…
I am a bit confused by this thread though (perhaps my speed scan of the posts is at fault), I thought we could not use inline styles at all in Scrivener footnotes, so what have I missed? — EDIT: oh OK this is about inline footnotes…
Yeah, inline footnotes are the tool of choice if you need complex editor formatting such as styles. So this should work, and styles in general do work in footnotes for other compile methods. Once the underlying bug is fixed then this shouldn’t be a problem.
Indeed, my earlier comment about fixing the brackets as a workaround for the time being was not meant to imply that Replacements would work (though I was thinking more along the lines of search and replace in a text editor, post-processing would work as well). That is by design, as this checkbox will convert rich text to Markdown, and thus presumably Replacements are operating within the rich text context, producing text that is meant to be printed verbatim in KF8, PDF or a Markdown-based export, as the case may be. A replacement will only serve to generate code, if the entirety of the string replacement is done within a style that is marked as such (and then we’re back to the bug).
These bugs (footnotes being split by “Treat as Raw Markup” + “Convert rich text to MMD” and also inline footnotes and annotations not having styles options applied) should be fixed for 3.0.4.
In the meanwhile, this postprocessing script cleans up escaped brackets for all Pandoc citations that I could think of.
#!bin/sh
perl -pe '#
s/\\\[(.*?@\w+.*?)\\\]/\[$1\]/g;
s/(\@\w+.*?)\\\[(.+?)\\\]/$1\[$2\]/g;
' $1 |
pandoc .... whatever ...
Interestingly, it only works with perl, not with sed, for whatever reason (took me forever to figure out why it wouldn’t work, apparently sed has a problem with the 0 or more .*? expression ).
Neat, thanks for posting the solution for anyone else that might need it.
As for the sed issue, I just responded over in the other thread as well: but yes sed doesn’t do non-greedy syntax. You have to design patterns to work around that. More often than not I switch engines if I get to the point of needing them, as you did.
— EDIT: oh OK this is about inline footnotes…