ScrivQ | A template to control Quarto, export multiple files, manage bibliography and easily create cross-references

This template tries to integrate Quarto into the Scrivener writing environment. It includes new Section Types , Paragraph Styles , Character Styles , Custom Meta Fields , YAML Parameters for Quarto, new icons, and more.⁠ If you already have Quarto and R, it introduces zero new dependencies and you should be able to compile it immediately. All the auxiliary files – bibliographies, lua filters, project metadata – will automatically be created in the export folder each time it compiles (if already present, they will be overwritten).

The template’s first notable difference is the way the YAML Front matter is written and the size it is. Instead of using a single binder text item for all the options, as is common practice, we are using one binder item for each YAML parameter with the idea of having all the options properly set up and available to be included or excluded from compiling by simply ticking a box. The YAML structure is automatically (WIP) formed by using the correct Section Type from whence the sub-items will inherit their own type (there is a digit between parenthesis next to the Section Type name indicating the number of empty spaces used as a prefix). This strategy can be used to control a high number of variables, as we are doing here, and to control the behavior of Quarto websites.

Quarto Options

Apart from being able to export the content of the project into multiple files, it already features nearly every parameter there is for Quarto Books (PDF, DOCX, HTML) and Websites (HTML), which amounts to some 600 options. (Yes, it was an insane amount of work). Each parameter contains the description offered at the Quarto website with further links under Document Links.

Bibliography

This means that it can output any combination of these parameters in any file it creates, which makes it ideal for maintaining complex projects. Or you can ignore all of that and simply use it as it is. It already demonstrates how to keep a simple CSL-YAML bibliography in the project itself (which is not to say that you can’t simply paste and keep a BibTeX bibliography).

It includes all the advanced bibliography features from my previous template, including the Cite Tools integration; but now we have the option of keeping everything within the Scrivener project. You can check right there in Scrivener the author name that will come up if you apply the style Cite Author to a given reference. This is all optional, of course. I don’t keep my bibliography files inside the project, but it is very useful for creating self-containing examples.

Cross-references

The cross-reference mechanism received special attention. Any element that can be referenced will automatically receive a label, so all we need to do is create a link and apply the character style corresponding to the element (figure, equation, table, listing, etc). And did I mention that we have all of the main amsthm environments already set up (Conjecture, Corollary, Definition, Example, Exercice, Lemma, Proposition, Theorem)?

There are several other important features and elements that I will skip mentioning for lack of time. With this, I noticed better than ever that documenting stuff you come up with is so much more work than getting it done in the first place. I try to explore everything in the template itself. Explore, enjoy. I am pretty satisfied with it and I hope you will be, as well.

Sidenote: with 34MB, it is a bit on the heavy side. This is mainly due to the icons and the elevated number of parameters. I suggest keeping this version and making another slimmed-down template version for your daily needs. You won’t need all the options all of the time.

Download | Github Repository | PDF Sample

3 Likes

Wow Bernardo this is a wonderful template, congratulations!!! Some really nice ideas and execution (like using the Binder to build the YAML options). It failed to compile for me due to a missing CSL file (still hardcoded to your system). I didn’t dig into it in too much depth yet, but was curious how your “Files” binder item (which becomes back-matter) works to export files like the BIB into the refs folder?

Yes, I totally get the “documenting” being harder than “doing” point, it can take ages to clearly document a tool or workflow…

2 Likes

I am happy to know you liked this part, Ian. Be it for Quarto or Scrivomatic, metadata files can grow quite big sometimes, right? Please let me know if you have any suggestions or observations of any kind. I would be genuinely interested in hearing them.

Thank you for pointing it out. Fixed! I will post the updated version shortly.

It could also be in the Manuscript or in the Frontmatter, as this won’t affect anything other than the heading levels of some automatic sections within the sub-items.

I think I know what you mean here, and the answer is that it is just a trick of magic. At first, I thought about trying to recreate some sort of folder structure within Scrivener, but I quickly noticed that it would be impractical right now (due to time constraints), so the easy solution runs as follows: I add the full relative path to the ID field of the items that will be exported. One of the CiteTools extension Lua Filters is exported to _extensions/citetools/citation-backlinks.lua because that is the content of the ID field, and not because it lives somewhere specific in the project. So, for your bibliography file, all that matters is that it has a File Section Type and that the content of the ID field is refs/workflow.bib (the script runs mkdir before creating the files). I was unsure about keeping the bib files in Scrivener, but there are some clear advantages, I think. Scrivener’s blazing-fast search starts reaching the content of the project bibliographies as well.

This also means that, in the future, to create links between the content of two different files (or pages), this could be used to our advantage. Imagine this is our link [content](<$custom:ID>/scriv<$linkID>). We would then link the <$custom:ID> to the File and the scriv<$linkID> to the sub-element we are linking to (let us just assume that it is valid for the examples’ sake). That would print both the full project path and the sub-element’s address.

Scrivomatic is a work of art in this regard.


Something that occurred to me just now is that we could even further bind the content of options such as csl to our internal files. In the image below the csl item would receive the correct path to the chicago csl file wherever we decide to allocate the in the project or in the export folder (by changing the ID).

We could also leave auto as a reminder in outline mode, of course.

Fantastic work, Bernardo! I can’t wait to try it. Making a subset from the big master one is a smart idea. Thank you very much for your work!

Paolo

1 Like

Let me know how it goes. I am still organizing the Templates and Printout folders, so avoid them for now. Also, Ian (@nontroppo), I hope you didn’t get distracted by the other compile formats that appeared in the first version, as they were rubbish. In the last version, it is down to one.

Next, I would like to integrate it with Git/Github Pages. There is so much that could be done in this Scrivener/Quarto interface that I wish this could be the subject of an academic research grant (so that I could work on nothing else for a semester or so, but, alas, Scrivener is not Aristotle!).

3 Likes

It may make sense to version your template (as simple as add a version number somewhere, or tag releases on github). Yes I did get confused by all the different project formats (they still appear present in the latest template I just downloaded), and some of them treat all the YAML sections as-is which will fail to compile properly:

I haven’t investigated to see how you split your files which is why it all still seems a bit magic to me :face_with_peeking_eye: — is the file splitter part of the modified ruby script, and it only works when a Files section layout is applied, not on general Headings?

Yes, it is an interesting idea to keep bibliography in Scrivener, I hadn’t thought about the benefits project search may bring!

The main downside about using the Binder for YAML metadata is mostly the added complexity in the compiler, both the long list in the documents selector panel and the application of the correct section layouts, but the advantages lie in discoverability (so nice to see documentation etc.) and tweakability from within Scrivener. As you found, Scrivener does not handle editing code documents like YAML well, it keeps autocorrecting etc (it would be great if you could turn off autocorrect per document if needed…) — so I understand your route to use binder+custom metadata + compile powers; it is one of the nicest and coolest uses of the Binder I’ve seen!


edit: you can use ImageOptim to compress your project PNG files, they will compress by >70%. brew install imageoptim

1 Like

Apologies for that. I forgot to remove them. As I was (and still am) actively editing the compile settings, I keep it all in one place.

Yes! Here is where it starts; and here is where it ends.

All the main fixes happen before we split the files from the text. So, after your original script is done editing the text, we look for the markup associated with a new file. If it is not there, life goes on; if it is there, then we go ahead and check for footnotes. If there are footnotes, we turn them into inline footnotes.* Then, we scan the text to retrieve the content and path of each file.

<!-- begin_file: "<$custom:ID>" -->
---
title: Hello!
---

[]{#scriv202}

Lorem ipsum dolor sit amet, consectetur adipisicing elit...

<!-- end_file -->

After we are done creating the files, we remove them from the main text (that ends up in the new qmd file, which, I suppose, could be hardcoded as index.qmd).

* Note: Scrivener is capable of outputting inline footnotes in txt export, but not in markdown; the problem with txt is that tables are not exported as markdown tables

Agreed. It is not possible to fold the long list in the compile panel, so sometimes it causes Scrivener to slow down; also, the correct application of the section layouts can indeed be challenging (especially in an array of dictionaries).

(To be continued)

“Aristotle would have used Scrivener” looks like a great exergue for a modern thesis!

“Abstract: Here we advance the hypothesis that Aristotle’s complete work would have survived up to our times, if the great Stagirite had used Scrivener to keep his writings in order. A parallel between Aristotles logic and snowflake idea shaping method is also proposed.”

Paolo

2 Likes

So I’ve just been able to get some time to try this. I’ve downloaded the template from GitHub, imported it as a template into Scrivener, and then created a new test project from the Template chooser.

When I Compile, one Project Format is selectable, Quarto Publish PDF. Selecting that and choosing Compile resulted in an error. To get it to work, I had to install a few packages in R:

> install.packages("markdown")
> install.packages("tidyverse")
> install.packages("kableExtra")

and then in a terminal I did:

brew install font-code-new-roman-nerd-font

and also installed the .otf files separately as at first I thought the brew command hadn’t worked (I think it actually installs slightly different fonts, so I’m not sure which one worked).

I now get a different LaTeX failure, and the relevant part of the output log seems to be this:

LaTeX Warning: Reference `fig-castle' on page 5 undefined on input line 640.

[5]

LaTeX Warning: Reference `fig-scriv158' on page 6 undefined on input line 692.

<quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf, id=159, 387.4475pt x 242
.9075pt>
File: quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf Graphic file (type p
df)
<use quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf>
Package luatex.def Info: quarto-book-test_files/figure-pdf/fig-scriv158-1.pdf
used on input line 708.
(luatex.def)             Requested size: 387.44655pt x 242.90689pt.
[6<./Elephant2.jpg>]
Package marginnote Info: right page because not two side mode on input line 722
.
Package marginnote Info: xpos seems to be \@mn@currxpos  on input line 722.

! LaTeX Error: Not in outer par mode.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...

l.733 \caption
            {This table uses Section Type \texttt{{[}Code\ R{]}} to insert the

Here is how much of LuaTeX's memory you used:
 46452 strings out of 478278
 100000,1977958 words of node,token memory allocated
 9285 words of node memory still in use:
   55 hlist, 12 vlist, 20 rule, 60 disc, 11 local_par, 3 dir, 2 math, 204 glue,
65 kern, 37 penalty, 4 margin_kern, 602 glyph, 359 attribute, 94 glue_spec, 123
attribute_list, 2 temp, 1 if_stack, 5 write, 1 save_pos, 1 user_defined, 4 pdf_d
est, 6 pdf_action, 11 pdf_colorstack, 1 pdf_setmatrix, 1 pdf_save, 1 pdf_restore
 nodes
   avail lists: 1:2,2:2845,3:816,4:184,5:550,6:39,7:6384,8:36,9:428,10:9,11:562,
12:1
 65002 multiletter control sequences out of 65536+600000
 170 fonts using 65343063 bytes
 108i,9n,123p,10610b,1126s stack positions out of 10000i,1000n,20000p,200000b,200000s
!  ==> Fatal error occurred, no output PDF file produced!

Any thoughts on this?

I figured out how to get an HTML output file, but that shows up with a very plain style:

Is there something I need to do to get the normal Quarto HTML styling?

This look amazing! Thanks so much for your efforts, and for sharing them here.

I have a Scrivener+Pandoc+Pandoc-crossref workflow that is working nicely for a book I’m writing. And I had given some thought to writing up the workflow after the book is finished, to share with others. But I suspect that this effort of yours is far superior to what I’ve worked up, and better documented than I could do, too! Also, I rely so heavily on pandoc-crossref and I worry that it’s just one guy and that it could stop working well with pandoc if he abandon’s the project. Am I right that while Quarto’s cross-referencing tools are based on pandoc-crossref, they are being maintained by the Quarto folks and are for that reason possibly less likely to become deprecated?

Thanks again!

P.S. I’m just now seeing @nontroppo’s related post from last year. Thanks to you both for these super useful templates and write-ups!

1 Like

@nontroppo and @bernardo_vasconcelos: Can these templates be used to output to LaTeX as well as PDF and docx?

Have you tried to use Quarto manually with a test document, is the LaTeX and HTML output as expected? The HTML is definitely not as it is supposed to be or as I see at least with my Quarto template.

Sometimes a simpler alternative is worth having. My pandoc template is pretty advanced (using metadata via pandocomatic etc), and this is overkill for some people who would nevertheless benefit from a markdown+pandoc workflow, so don’t feel discouraged…

Yes I think that is correct, and indeed pandoc-crossref is susceptible to Pandoc API changes (quite often the homebrew Pandoc formula cannot be updated as it must wait on pandoc-crossref to ensure compatibility; to be fair the pandoc-crossref author has always updated within a week or two). Now in theory we should be able to extract out the quarto-crossref system, but when I looked it is quite a complex bevy of Lua scripts…

At least there is a keep-tex: true option so the PDF route keeps the .tex file…

1 Like

Thanks for all those useful responses!

In trying @bernardo_vasconcelos’s template here, what do I need to have installed beyond Quarto and Pandoc? I have those, but when I tried to compile the project as it is out of the box, I got these outputs with no pdf:

Screenshot 2023-07-26 at 9.45.25 AM

Do I need R installed?

I think ScrivQ’s template does have some R documents in the default compile – you’ll see it in the Methods > stimulus Plotting parts. Just deselect these in the compiler, then no need to install R. There is also some mermaid / graphviz documents that IIRC need a working Chrome, so try removing those, or use quarto to install chromium (quarto tools).

You can open Quarto.log and read through to see what Quarto said; it seems at least a .qmd file was produced which suggests Scrivener and the Ruby script did their job at least…

2 Likes

Thanks!

Random question: I see here that one can specify certain pandoc arguments in the YAML material when using Pandocomatic. E.g. the .bib file here (with the end of the path clipped off in this screenshot):

Do you know if I can include my bib file’s path, and my csl file’s path, as an item in YAML when not using Pandocomatic? That is, when using just vanilla Pandoc as my post-processor? Right now I include those in my post-processing string in the compile pane, but I’d love to be able to set those in YAML instead.

Yes, this is possible in vanilla pandoc:

---
title: Test
author: Jane Doe
bibliography: /Users/ian/.local/share/pandoc/Core.json
csl: apa.csl
---

# Intro

Some Text here [@shipp2013]. More blah.

Compiled using pandoc

▶︎ pandoc --citeproc -t html test.md
<h1 id="intro">Intro</h1>
<p>Some Text here <span class="citation" data-cites="shipp2013">(Shipp
<em>et al.</em>, 2013)</span>. More blah.</p>
<div id="refs" class="references csl-bib-body hanging-indent"
data-line-spacing="2" role="list">
<div id="ref-shipp2013" class="csl-entry" role="listitem">
Shipp, S., Adams, R. A., &amp; Friston, K. (2013). Reflections on
agranular architecture: Predictive coding in the motor cortex.
<em>Trends in Neurosciences</em>, <em>36</em>(12), 706–716. <a
href="https://doi.org/10.1016/j.tins.2013.09.004">https://doi.org/10.1016/j.tins.2013.09.004</a>
</div>
</div>

You should also check out Pandoc’s defaults files: Pandoc - Pandoc User’s Guide – these are another mechanism for setting up multiple config options…

2 Likes

Wow, @bernardo_vasconcelos, this looks like it took an immense amount of work. Many thanks to you and to @nontroppo for being pioneers in getting Quarto to work with Scrivener.

Unfortunately, I cannot get your sample file to compile to PDF. I have looked within the metadata section, but I cannot find if there are any file paths that need to be adjusted. What are the steps needed for a “hello, world” example here?

Secondly, I do applaud you for the innovative use of the binder as a “checkbox” method of specifying the YAML frontmatter that Quarto looks for… but I do find this harder to read than a simple YAML document at the top. How would I go about making it work with a simple YAML document in the frontmatter?

Sorry for the troubleshooting questions! I love the idea of getting Quarto working with Scrivener, but it is far from easy!

~ Alexander

2 Likes

Another thing that I am struggling with: I need to activate a conda virtual environment (“conda activate quarto”) via the Scrivener compilation process, but I am unable to find a way to do that. It seems to be a common problem with conda in sub-shells, which is what Scrivener uses to run the script. Is there a way to make this happen via the Ruby script that you use in the compile settings?

To activate Conda you need to add-in the code that would be normally run by the user’s initialisation script (e.g. .zshrc). But Scrivener does not run post-processing as a user shell (annoyingly) and therefore loses any setup. The good news is Ruby can run commands for you, and the Ruby script does contain a list of file paths to add. So you’d need to add the path to the path list, then run the eval script

Regarding the problems you see, check the log files Quarto or Scrivener produces. You should also be able to replace the YAML magic with a simple single document, just select the simple doc as frontmatter – but you’d need to know the exact settings as I’m sure some part of scrivq needs specific values. You could use my older project template which uses simple YAML, though it doesn’t support mutiple bibliographies…

Thank you for your feedback. I was able to get it to work last night using a bit of bash shell scripting in the Scrivener Compile > Edit format… > Processing > “Post-process on command-line” script window:

source /path/to/miniconda3/etc/profile.d/conda.sh
source ~/.bash_profile
conda activate quarto_environment_name
mv $1 $2.qmd
quarto render $2.qmd --to=pdf
open $2.pdf

where $1 and $2 represent the arguments <$inputfile> and <$outputname>, respectively. The mv was necessary to rename the file because I couldn’t get Scrivener to compile via MultiMarkdown and use the .qmd extension instead of .md.

This solution is clearly not anywhere near as cool or robust as what you or @bernardo_vasconcelos have created. I have simply been trying to create the “leanest” solution – partly to keep it simple, but partly to also learn how everything works.