ScrivQ | A template to control Quarto, export multiple files, manage bibliography and easily create cross-references

Scrivener Markdown & LaTeX
, , ,
5

Let me know how it goes. I am still organizing the Templates and Printout folders, so avoid them for now. Also, Ian (@nontroppo), I hope you didn’t get distracted by the other compile formats that appeared in the first version, as they were rubbish. In the last version, it is down to one.

Next, I would like to integrate it with Git/Github Pages. There is so much that could be done in this Scrivener/Quarto interface that I wish this could be the subject of an academic research grant (so that I could work on nothing else for a semester or so, but, alas, Scrivener is not Aristotle!).

3 Likes
6

It may make sense to version your template (as simple as add a version number somewhere, or tag releases on github). Yes I did get confused by all the different project formats (they still appear present in the latest template I just downloaded), and some of them treat all the YAML sections as-is which will fail to compile properly:

Screenshot 2023-06-16 at 09.13.03
Screenshot 2023-06-16 at 09.13.031166×462 34.2 KB

I haven’t investigated to see how you split your files which is why it all still seems a bit magic to me :face_with_peeking_eye: — is the file splitter part of the modified ruby script, and it only works when a Files section layout is applied, not on general Headings?

Yes, it is an interesting idea to keep bibliography in Scrivener, I hadn’t thought about the benefits project search may bring!

The main downside about using the Binder for YAML metadata is mostly the added complexity in the compiler, both the long list in the documents selector panel and the application of the correct section layouts, but the advantages lie in discoverability (so nice to see documentation etc.) and tweakability from within Scrivener. As you found, Scrivener does not handle editing code documents like YAML well, it keeps autocorrecting etc (it would be great if you could turn off autocorrect per document if needed…) — so I understand your route to use binder+custom metadata + compile powers; it is one of the nicest and coolest uses of the Binder I’ve seen!

edit: you can use ImageOptim to compress your project PNG files, they will compress by >70%. brew install imageoptim

1 Like
7

Apologies for that. I forgot to remove them. As I was (and still am) actively editing the compile settings, I keep it all in one place.

Yes! Here is where it starts; and here is where it ends.

All the main fixes happen before we split the files from the text. So, after your original script is done editing the text, we look for the markup associated with a new file. If it is not there, life goes on; if it is there, then we go ahead and check for footnotes. If there are footnotes, we turn them into inline footnotes.* Then, we scan the text to retrieve the content and path of each file.

<!-- begin_file: "<$custom:ID>" -->
---
title: Hello!
---

[]{#scriv202}

Lorem ipsum dolor sit amet, consectetur adipisicing elit...

<!-- end_file -->

After we are done creating the files, we remove them from the main text (that ends up in the new qmd file, which, I suppose, could be hardcoded as index.qmd).

* Note: Scrivener is capable of outputting inline footnotes in txt export, but not in markdown; the problem with txt is that tables are not exported as markdown tables

Agreed. It is not possible to fold the long list in the compile panel, so sometimes it causes Scrivener to slow down; also, the correct application of the section layouts can indeed be challenging (especially in an array of dictionaries).

(To be continued)

8

“Aristotle would have used Scrivener” looks like a great exergue for a modern thesis!

“Abstract: Here we advance the hypothesis that Aristotle’s complete work would have survived up to our times, if the great Stagirite had used Scrivener to keep his writings in order. A parallel between Aristotles logic and snowflake idea shaping method is also proposed.”

Paolo

2 Likes
9

So I’ve just been able to get some time to try this. I’ve downloaded the template from GitHub, imported it as a template into Scrivener, and then created a new test project from the Template chooser.

When I Compile, one Project Format is selectable, Quarto Publish PDF. Selecting that and choosing Compile resulted in an error. To get it to work, I had to install a few packages in R:

> install.packages("markdown")
> install.packages("tidyverse")
> install.packages("kableExtra")

and then in a terminal I did:

brew install font-code-new-roman-nerd-font

and also installed the .otf files separately as at first I thought the brew command hadn’t worked (I think it actually installs slightly different fonts, so I’m not sure which one worked).

I now get a different LaTeX failure, and the relevant part of the output log seems to be this:

LaTeX Warning: Reference `fig-castle' on page 5 undefined on input line 640.

[5]

LaTeX Warning: Reference `fig-scriv158' on page 6 undefined on input line 692.

<quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf, id=159, 387.4475pt x 242
.9075pt>
File: quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf Graphic file (type p
df)
<use quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf>
Package luatex.def Info: quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf
used on input line 708.
(luatex.def)             Requested size: 387.44655pt x 242.90689pt.
[6<./Elephant2.jpg>]
Package marginnote Info: right page because not two side mode on input line 722
.
Package marginnote Info: xpos seems to be \@mn@currxpos  on input line 722.

! LaTeX Error: Not in outer par mode.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...

l.733 \caption
            {This table uses Section Type \texttt{{[}Code\ R{]}} to insert the

Here is how much of LuaTeX's memory you used:
 46452 strings out of 478278
 100000,1977958 words of node,token memory allocated
 9285 words of node memory still in use:
   55 hlist, 12 vlist, 20 rule, 60 disc, 11 local_par, 3 dir, 2 math, 204 glue,
65 kern, 37 penalty, 4 margin_kern, 602 glyph, 359 attribute, 94 glue_spec, 123
attribute_list, 2 temp, 1 if_stack, 5 write, 1 save_pos, 1 user_defined, 4 pdf_d
est, 6 pdf_action, 11 pdf_colorstack, 1 pdf_setmatrix, 1 pdf_save, 1 pdf_restore
 nodes
   avail lists: 1:2,2:2845,3:816,4:184,5:550,6:39,7:6384,8:36,9:428,10:9,11:562,
12:1
 65002 multiletter control sequences out of 65536+600000
 170 fonts using 65343063 bytes
 108i,9n,123p,10610b,1126s stack positions out of 10000i,1000n,20000p,200000b,200000s
!  ==> Fatal error occurred, no output PDF file produced!

Any thoughts on this?

I figured out how to get an HTML output file, but that shows up with a very plain style:

Screenshot 2023-07-23 at 4.04.51 PM
Screenshot 2023-07-23 at 4.04.51 PM1822×1644 245 KB

Is there something I need to do to get the normal Quarto HTML styling?

11

This look amazing! Thanks so much for your efforts, and for sharing them here.

I have a Scrivener+Pandoc+Pandoc-crossref workflow that is working nicely for a book I’m writing. And I had given some thought to writing up the workflow after the book is finished, to share with others. But I suspect that this effort of yours is far superior to what I’ve worked up, and better documented than I could do, too! Also, I rely so heavily on pandoc-crossref and I worry that it’s just one guy and that it could stop working well with pandoc if he abandon’s the project. Am I right that while Quarto’s cross-referencing tools are based on pandoc-crossref, they are being maintained by the Quarto folks and are for that reason possibly less likely to become deprecated?

Thanks again!

P.S. I’m just now seeing @nontroppo’s related post from last year. Thanks to you both for these super useful templates and write-ups!

1 Like
12

@nontroppo and @bernardo_vasconcelos: Can these templates be used to output to LaTeX as well as PDF and docx?

13

Have you tried to use Quarto manually with a test document, is the LaTeX and HTML output as expected? The HTML is definitely not as it is supposed to be or as I see at least with my Quarto template.

Sometimes a simpler alternative is worth having. My pandoc template is pretty advanced (using metadata via pandocomatic etc), and this is overkill for some people who would nevertheless benefit from a markdown+pandoc workflow, so don’t feel discouraged…

Yes I think that is correct, and indeed pandoc-crossref is susceptible to Pandoc API changes (quite often the homebrew Pandoc formula cannot be updated as it must wait on pandoc-crossref to ensure compatibility; to be fair the pandoc-crossref author has always updated within a week or two). Now in theory we should be able to extract out the quarto-crossref system, but when I looked it is quite a complex bevy of Lua scripts…

At least there is a keep-tex: true option so the PDF route keeps the .tex file…

1 Like
14

Thanks for all those useful responses!

In trying @bernardo_vasconcelos’s template here, what do I need to have installed beyond Quarto and Pandoc? I have those, but when I tried to compile the project as it is out of the box, I got these outputs with no pdf:

Screenshot 2023-07-26 at 9.45.25 AM

Do I need R installed?

15

I think ScrivQ’s template does have some R documents in the default compile – you’ll see it in the Methods > stimulus Plotting parts. Just deselect these in the compiler, then no need to install R. There is also some mermaid / graphviz documents that IIRC need a working Chrome, so try removing those, or use quarto to install chromium (quarto tools).

You can open Quarto.log and read through to see what Quarto said; it seems at least a .qmd file was produced which suggests Scrivener and the Ruby script did their job at least…

2 Likes
16

Thanks!

Random question: I see here that one can specify certain pandoc arguments in the YAML material when using Pandocomatic. E.g. the .bib file here (with the end of the path clipped off in this screenshot):

IMG_5317
IMG_5317750×644 70.2 KB

Do you know if I can include my bib file’s path, and my csl file’s path, as an item in YAML when not using Pandocomatic? That is, when using just vanilla Pandoc as my post-processor? Right now I include those in my post-processing string in the compile pane, but I’d love to be able to set those in YAML instead.

17

Yes, this is possible in vanilla pandoc:

---
title: Test
author: Jane Doe
bibliography: /Users/ian/.local/share/pandoc/Core.json
csl: apa.csl
---

# Intro

Some Text here [@shipp2013]. More blah.

Compiled using pandoc

▶︎ pandoc --citeproc -t html test.md
<h1 id="intro">Intro</h1>
<p>Some Text here <span class="citation" data-cites="shipp2013">(Shipp
<em>et al.</em>, 2013)</span>. More blah.</p>
<div id="refs" class="references csl-bib-body hanging-indent"
data-line-spacing="2" role="list">
<div id="ref-shipp2013" class="csl-entry" role="listitem">
Shipp, S., Adams, R. A., &amp; Friston, K. (2013). Reflections on
agranular architecture: Predictive coding in the motor cortex.
<em>Trends in Neurosciences</em>, <em>36</em>(12), 706–716. <a
href="https://doi.org/10.1016/j.tins.2013.09.004">https://doi.org/10.1016/j.tins.2013.09.004</a>
</div>
</div>

You should also check out Pandoc’s defaults files: Pandoc - Pandoc User’s Guide – these are another mechanism for setting up multiple config options…

2 Likes
18

Wow, @bernardo_vasconcelos, this looks like it took an immense amount of work. Many thanks to you and to @nontroppo for being pioneers in getting Quarto to work with Scrivener.

Unfortunately, I cannot get your sample file to compile to PDF. I have looked within the metadata section, but I cannot find if there are any file paths that need to be adjusted. What are the steps needed for a “hello, world” example here?

Secondly, I do applaud you for the innovative use of the binder as a “checkbox” method of specifying the YAML frontmatter that Quarto looks for… but I do find this harder to read than a simple YAML document at the top. How would I go about making it work with a simple YAML document in the frontmatter?

Sorry for the troubleshooting questions! I love the idea of getting Quarto working with Scrivener, but it is far from easy!

~ Alexander

2 Likes
19

Another thing that I am struggling with: I need to activate a conda virtual environment (“conda activate quarto”) via the Scrivener compilation process, but I am unable to find a way to do that. It seems to be a common problem with conda in sub-shells, which is what Scrivener uses to run the script. Is there a way to make this happen via the Ruby script that you use in the compile settings?

20

To activate Conda you need to add-in the code that would be normally run by the user’s initialisation script (e.g. .zshrc). But Scrivener does not run post-processing as a user shell (annoyingly) and therefore loses any setup. The good news is Ruby can run commands for you, and the Ruby script does contain a list of file paths to add. So you’d need to add the path to the path list, then run the eval script

Regarding the problems you see, check the log files Quarto or Scrivener produces. You should also be able to replace the YAML magic with a simple single document, just select the simple doc as frontmatter – but you’d need to know the exact settings as I’m sure some part of scrivq needs specific values. You could use my older project template which uses simple YAML, though it doesn’t support mutiple bibliographies…

21

Thank you for your feedback. I was able to get it to work last night using a bit of bash shell scripting in the Scrivener Compile > Edit format… > Processing > “Post-process on command-line” script window:

source /path/to/miniconda3/etc/profile.d/conda.sh
source ~/.bash_profile
conda activate quarto_environment_name
mv $1 $2.qmd
quarto render $2.qmd --to=pdf
open $2.pdf

where $1 and $2 represent the arguments <$inputfile> and <$outputname>, respectively. The mv was necessary to rename the file because I couldn’t get Scrivener to compile via MultiMarkdown and use the .qmd extension instead of .md.

This solution is clearly not anywhere near as cool or robust as what you or @bernardo_vasconcelos have created. I have simply been trying to create the “leanest” solution – partly to keep it simple, but partly to also learn how everything works.

22

Yes, a simple shell script works wonders! If you wanted to utilise the Ruby script, you could still save it as a file and call it from your bash code.

But the most important thing the Ruby script does (at least my original one) is rewrite cross-referencing labels from a format that works well in Scrivener’s compiler and those that are required by Quarto’s crossref filter. So if you don’t need cross-referencing (remember Scrivener can do general cross-references too), then you may not need the ruby script at all. Bernardo’s script adds the ability to split one document into multiple files, which again you may or may not require…

23

Do you have a Scrivener project outfitted with this Ruby script and demonstrating your native Scrivener cross-referencing method that the script converts? I’d love to explore/play around with it.

24

Sorry for the late reply, @jpkell, @cavalierex, @lyndondrake and @ptram. I had been away from my home and workspace since June; and shortly after arriving a few days ago, I moved places, so I am still slowly catching up. The last version of ScrivQ has been ready since before I left, but I couldn’t finish documenting changes (which can be very obscure within a Scrivener project). I will be updating the GitHub project soon with it.

@lyndondrake, the error seems LaTeX-related, with regards to a caption and/or a margin note. You could try to locate it in the source file, comment it out, and try to compile the latex file again. If this is not something you could easily do, maybe you could just try the next version in which the error maybe will not show up. About the HTML styling, just add the YAML option as shown here: Quarto - HTML Theming

@jpkell and @cavalierex, lets us see if I can make this next version more friendly and easier to compile without R dependencies.

1 Like
25

No apology needed @bernardo_vasconcelos. I’m loving the template. What I might do is wait for the update on GitHub and try again with that.

On the point about HTML styling, the .qmd file produced has this:

  html:   
…
    theme:   
      light: [Pulse]  
      dark: [Cyborg]

but it doesn’t seem to be having much effect. Interestingly, this output happens even though the light option in the Scrivener metadata is set to default. I’ll re-test it when the new release is out on GitHub.

(Also I haven’t forgotten about the other conversation, of working on some filters and CSL styling via the Visual CSL Editor to get citations working the way I need them in pandoc-citeproc.)