Scrivener + Quarto: a technical/academic publishing workflow

As I previously demonstrated how to use Scrivener with pandoc, it was straightforward to add a new workflow for writing in Scrivener amd compiling using the new scientific and technical publishing system, Quarto. Quarto bundles Pandoc along with a set of extensions and templates to provide an all-in-one experience, including Tufte-like output, extensive cross-referencing, running code (R, Python, Julia, Dot, Mermaid etc.) to generate figures and complex layouts, generation of complete websites and more…

Because Scrivener supports Pandoc-flavoured markdown and automated post-processing, it already “supports” Quarto. However, for cross-referencing Quarto places labels at the end of the line, and this conflicts with Scrivener’s block style injection of markup. A small mod of a previous pandoc-crossref post-processing script solved this problem. This new script runs from the post-processing pane of the Scrivener compiler, and runs quarto automatically after tweaking the path and fixing the cross-references.


DOWNLOADS:

Scrivener + Quarto Project Template: save this file removing any .txt suffix if added by your browser; import the Scrivener + Quarto.scrivtemplate file using File → New Project… dialog, and then select the Non-Fiction → Scrivener + Quarto template to create your new project!

Alternative: project as a regular ZIP file.

Example PDF compiled directly from the above Scrivener project.

The bundled post-processing script can be checked here.

Quarto supports many journal templates and extensions which can be downloaded and installed. See this awesome list for a curated list.


Most Quarto features can be used via three routes, and the Scrivener + Quarto template demonstrates all three:

  1. Using Scrivener Styles — examples: Callouts, Live Code. The main caveat with Styles is Scrivener doesn’t allow paragraph [block] styles to be nested.
  2. Using Scrivener Section Layouts — example: multi-item images / tables using cross-referencing attributes. This method allows nested blocks. Scrivener document templates are provided.
  3. You can always write plain markdown in the editor if you prefer…

Quarto’s documentation is excellent. Some of the cool features that Quarto includes and work well with Scrivener:

  1. Panels with sub-figures and sub-tables.
  2. Based on Tufte, you can use margins for notes, figures, equations etc. Layout can be modified to span the page in many ways. You can rewrite layouts using Pandoc’s templating language.
  3. Comprehensive cross-referencing, using Pandoc’s @citation markup (based on pandoc-crossref). For sub-figures / tables, you can add labels, like Figure 3(c) and these can be customised.
  4. Academic author + affiliation metadata «similar to scrivomatic and pandoc scholar».
  5. Run Python, R, Julia code “live” to plot figures on each compile.
  6. Use graphing engines like Mermaid and Graphviz.
  7. HTML output has lots of cool tweaks, see https://quarto-dev.github.io/quarto-gallery/page-layout/tufte.html for example.
  8. Quarto allows installable extensions, so you can plug in new layout templates and code to extend its powers!
  9. As it is Pandoc-based, there are many formats supported and Pandoc’s custom filters customise many aspects of the final document.

Quarto installs [almost] everything required (you can use it to manage Pandoc, LaTeX, Graphviz, Mermaid and other tools all-in-one). For LaTeX it can install and update TinyTeX, which enables a small install footprint and automated addition of packages from CTAN as required (no fussing with tlmgr)…


Here are some screenshot of the HTML/PDF output from Scrivener, showing a few of the nice features:


Showing margin notes, cross-referencing of figure blocks, academic metadata, equations, citations etc. «HTLML example»



A margin note ala Tufte; these can include figures, equations etc. The figure also demonstrates the use of an auto-generated Mermaid graph made from editor text at compile time. «PDF example»



Multi-part tables; note the cross-referencing includes the sub-table labels, e.g. Table 2(B), and the custom styling of table rows. «HTML example»



Academic in-text citations can include hovered boxes containing the final bibliographic information. «HTML example»


Requirements

  1. Install Quarto (quarto bundles pandoc for you). Both brew «macOS/Linux» and scoop «Windows» can install quarto for you.
  2. You can then use Quarto to install LaTeX: quarto tools install tinytex — TinyTeX is small yet can auto-install required packages if they are missing as needed; OR you can use an existing TeX Live installation (I use BasicTeX on macOS).
  3. For Mermaid & Graphviz, Quarto can install a Chromium runtime if you don’t have Chrome installed: quarto tools install chromium.
  4. The script requires Ruby, which is already installed on macOS. You can use brew or scoop to install/update it otherwise.
9 Likes

I look forward to trying out this out! Thanks for investigating Scrivener-Quarto integration, @nontroppo, and for sharing your work.

It would be helpful perhaps for someone to illustrate how Quarto syntax for cross-referencing differs from the markdown specified in the Pandoc user guide. I may try to illustrate and summarize the important differences when I get a chance…

@Edmund — Pandoc doesn’t [yet] support figure/table/equation/theorem cross-references natively. Nevertheless Quarto utilises Pandoc’s existing image attributes extension and uses the #id tag (see HTML for the origin) to support cross referencing. There are two methods of linking / figure blocks in markdown; inline links:

![Caption](foo.jpg){#id .class width=30 height=20px}

And reference links, which is the format Scrivener’s compiler generates:

![Caption][ref]
…
[ref]: foo.jpg {#id .class width=30 height=20px}

Scrivener generates it when it converts an embedded-image + caption style into markdown, and it creates this reference at compile time, it doesn’t exist at all in the editor. So the question is, how can the writer in Scrivener add a “unique” #id in the editor for each paragraph block, which the compiler can use? My solution is simple: the Scrivener user writes the {#id} at the beginning of the caption block in the editor, this passes through to the markdown untouched:

Screenshot 2022-09-14 at 21.42.49_SMALL

My post-processing script takes the Scrivener-compiled markdown:

![{#fig-trunk2} Angry elephant with big trunk.][elephant]

[elephant]: elephant.jpg 

…and rewrites it to a format Quarto’s cross-referencing system can utilise:

![Angry elephant with big trunk.][elephant]

[elephant]: elephant.jpg {#fig-trunk2} 

The other tweak the script does is for maths cross-referencing. Quarto requires the #id after the maths block without any line break:

$$e^{ix}=r(\cos \theta +i\sin \theta)$$ {#eq-one}

BUT Scrivener’s paragraph style prefix/suffix would put $$ after {#eq-one}, breaking the equation and cross-ref. So my workaround is to write the label on the next line

and remove the newline between $$ and {#eq-one} from the markdown file, allowing Quarto to cross-ref properly.

2 Likes

Thanks for the explanation, @nontroppo.

There is a great list of Quarto resources in this awesome list:

Scrivener is now mentioned as a supported editor…

1 Like

Thank you very much, Ian (@nontroppo). The template is excellent. I always learn new things through your Scrivener templates, Ruby scripts, and Alfred workflows. Whatever you come up with, keep sharing. :slight_smile:

3 Likes

I second what @bernardo_vasconcelos says. Thanks @nontroppo , we are learning so much from you.

3 Likes

Ian @nontroppo, I noticed that the crossref label style is absent from the styles panel in the file compile settings. Is this something you experimented with and later dropped?

I am curious because I thought Quarto would allow referencing code blocks as well, or simply adding labels to any block and reference anywhere, but that doesn’t seem to be the case.

Yes, probably that style should have a different name, something like Equation label, as the style is only used to visually identify the label used for equations:

In this case, nothing is required to be changed for compile, as the text itself is complete: {#label} is valid markdown already.

You could add labels to code blocks I suppose, but this may require some custom styles, so for example in this desired markdown output[1]:

~~~ruby {#id}
x = y
~~~

The question is how we add {#id} in the scrivener editor and use this to move it to the correct place? One solution is to use custom metadata field and a placeholder in the compiler, another solution is to put the {#id} label AFTER the code block, then use a script to move it up to the correct place (like I do for equation labels). My current script doesn’t do this yet for code blocks though, though it would be easy to add. Or just write the raw markdown directly, but my general workflow always prefers Styles of Section Layouts over raw markup whenever possible. If you do need crossref labels for code blocks and want to use the script, I’m happy to add that or if you know enough ruby you can create a pull request on github.


[1]: this particular format may require the next version of pandoc, see Syntax revision for markdown fenced code attributes · Issue #8174 · jgm/pandoc · GitHub, but the previous method of ~~~{.ruby #id} would work with existing pandoc…

Checking the quarto docs, the correct format for cross-referencing code blocks is found here: Quarto - Cross References

Something like:

~~~{#lst-customers .sql lst-cap="Customers Query"}
SELECT * FROM Customers
~~~

So it needs an #id as well as a lst-cap= attribute. My probable solution would be a section type using the existing custom metadata (the template has ID, class and attributes fields already defined). The existing Section Layout Multipart block contains section prefix and suffix to add :::{<$custom:ID> <$custom:Class> <$custom:Attributes>}\n::: some a modified format to use code identifier rather than div identifier and it should work…

Suppose we are willing to give the code block (or any other entity, as this would be content agnostic) a (scrivener) text file of its own. In that case, we can always just give it a standard markdown label (I usually place it as a prefix to the section), such as []{#scrivauto:<$linkID>} and create links using [text](scrivauto:<$linkID>). When converting to HTML, it will just become a link to a part of the text that will have an invisible anchor; when converting to Latex, we could, for example, use a filter to change it to something like \autoref{$1} to turn it into the correct page number.

This is my current approach that I am slowly recreating in the Quarto environment.

I will play with and script, and if something good comes out of it, you’ll see a pull request :wink:

Finally, wouldn’t a valid markdown be []{#eq-one}? I am slightly confused because just {#eq-one} doesn’t seem to work for me in Pandoc.

Yes, if you make it a document then you can use a Section Type and Section Layout prefix / suffix and use a placeholder. Quarto requires fairly specific ID labels, so for figures #fig-ID, for tables #tbl-ID, for equations #eq-ID — as long as you follow this ID format then Quarto should do the right thing.

Now for cross-referencing, as long as you use document links on top of <$linkID> then you should be able reference anywhere. Although I don’t think this works outside of the text itself (i.e. we can’t use <$linkID> in custom metadata). To be honest I prefer to give my labels manual names, like #fig-statsplot, so in other documents I can use @fig-statsplot to reference it, without needing to use document links etc. and being able to use custom metadata. But there are so many ways to skin this fish anything you are comfortable with, use!

If you look at [Multipart Table] or [Multipart Figure] in the template you will see they use the custom metadata fields to assign #id .class and attributes — I personally like this way of working.

Regarding {#eq-one} being on its own line, this is specific to the cross-referencing filter in Quarto, not a pandoc thing. It is a workaround for using the Maths Block Style. Scrivener paragraph styles will add a prefix and suffix so for example, this text styled as Maths Block:

x = y

will get transformed by the compiler style into:

$$ x = y $$

Now quarto wants this:

$$ x = y $$ {#eq-one}

So my simple trick is to write this in the Scrivener editor:

x = y
{#eq-one}

Which compiles to:

$$ x = y $$
{#eq-one}

And my script searches for $$\n{#eq- and replaces the \n with a space, thus generating the correct markup for cross-referenced equations. You could also do something like:

x = y {#eq-one}

to generate:

$$ x = y {#eq-one} $$

and use a script to move {#eq-one} after the $$. But I quite like the way the label looks on its own line in the editor, so choose that route…

Here is an example for adding cross-referenced label, syntax kind (i.e. which syntax highlighting to use) and caption text to a code listing using custom metadata. This uses the custom-metadata + Section Layout prefix + <$custom:placeholder> method. This in the Scrivener editor:

Becomes this HTML after processing by Quarto:

I think this is an easy way to work, managing the document labels and values using the Inspector, and having the exact label available as plain text for the cross-referencing. Note for HTML outputs you can add other attributes like code-line-numbers="true" to add line numbers and other stuff, see Quarto - HTML Code Blocks for the details.

Thanks for the detailed answer, Ian. I got a little confused about the position of that label because I never used equations, but it is all clear now. I also like to use the Pandoc crossref filters whenever possible, and I also don’t mind using the markup in code blocks. With Quarto, we can hit compile to have Scrivener create a website out of the text – with awesome search, dark mode, reader mode, etc – and with a download button for the pdf, docx, and epub versions. All of them look as gorgeous as they can be. Pretty amazing.

I recently finished a similar job creating a system to do the same thing using DOCX files and ruby scripts (with mkdocs), but the system with Quarto is so much simpler. When I have some time, within a few weeks, I will try to do it from a Scrivener project, that is, export a Quarto project for a website from Scrivener, perhaps using this template GitHub - nmfs-opensci/QuartoReport: Example government report

1 Like

Yeah, the HTML output is really flexible and works great (small details like clipboard copy for code blocks etc.)

One issue for web sites / books — Quarto uses multiple QMD documents and a separate yaml file to identify the “chapters”/“pages” to be included in the index. From Scrivener this will require us to split the single-file markdown output on the headings and then create the index from them using a scriptwith the post-processing script. I had planned to modify my script at some point to add a commandline option --split, but it is very low priority in my work life. It isn’t complicated, but it is the main hurdle to getting multi-page Quarto websites working with Scrivener…

One nice tweak would be to integrate this with their publish mechanism for e.g. github pages: Scrivener compiles to markdown, post-processor splits the doc up, then commits and pushes it. Github will actually run Quarto to generate the static html and host it via their github actions automagically. This would allow you to use Scrivener to run a git-versioned writing workflow to a fully hosted site with minimum fuss. Write, compile, and let tech do the rest… :nerd_face:

2 Likes

Indeed! And I feel tempted to put in the time, but priority-wise, it is at the bottom right now. But, to get the ball rolling, maybe this could be useful: inline_fn | RubyGems.org | your community gem host :slightly_smiling_face: There is no need to use the gem, of course. It’s just one short script inline_fn/inline_fn.rb at Main · bcdavasconcelos/inline_fn · GitHub to convert the markdown footnotes to (pandoc|mmd) inline-style footnotes. After we sprinkle the footnotes back on the text, we can safely split it using some custom delimiter between sections.

To split the files, I used this little script in one of my projects. I think we could probably do better than it for a Quarto template :rofl: but it was there already. I hope the comments provide enough insight.

#!/usr/bin/env ruby
# frozen_string_literal: false

# Simple Ruby File Spliter Script
# ===============================
# 
# Split <file> using <delimiter> into <folder> adding <extension> to the new files.
# 
# -f / --file	    $FILE
# -d / --delimiter  $DELIM
# -t / --folder     $FOLDER
# -e / --extention  $EXTENSION
# 
# Examples:
# 
# Ex.1:	ruby split_file.rb -t '~/text_file.txt' -d '***' -t '~/Dropbox/my_folder' -e '.txt'
# Ex.2:	ruby split_file.rb --file '~/text_file.txt' --delimiter '***' --folder '~/Dropbox/my_folder' --extension '.txt'
# 
# Both will split the `~/text_file` using the `***` as delimiter, placing the newly create files into `~/Dropbox/my_folder` and giving them the `.txt` extension.
# 
# Report issues at...
# https://github.com/bcdavasconcelos

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

text = STDIN.read

###  Args  ###
if ARGV.include?('-h' || '--help')
  puts "\n\n\t\tSimple Ruby File Spliter Script\n\t\t===============================\n\nSplit <file> using <delimiter> into <folder> adding <extension> to the new files.\n\n\t-f/--file\t$FILE\t\te.g.\t-f '~/text_file.txt'\n\t-d/--delimiter\t$DELIMITER\te.g.\t-d '***'\n\t-t/--folder\t$FOLDER\t\te.g.\t-t '~/Dropbox/my_folder'\n\t-e/--extention\t$EXTENSION\te.g.\t-e '.txt'\n\nExamples:\n\nEx.1:\truby split_file.rb -t '~/text_file.txt' -d '***' -t '~/Dropbox/my_folder' -e '.txt'\nEx.2:\truby split_file.rb --file '~/text_file.txt' --delimiter '***' --folder '~/Dropbox/my_folder' --extension '.txt'\n\nBoth will split the `~/text_file` using the `***` as delimiter, placing the newly create files into `~/Dropbox/my_folder` and giving them the `.txt` extension.\n\nReport issues at...\nhttps://github.com/bcdavasconcelos\n\n"
  return
end

file_path = if ARGV.include?('-f')
               ARGV[ARGV.index('-f') + 1]
            elsif ARGV.include?('--file')
               ARGV[ARGV.index('--file') + 1]
            else
               ARGV[0]
            end

delim = if ARGV.include?('-d')
           ARGV[ARGV.index('-d') + 1]
        elsif ARGV.include? '--delimiter'
           ARGV[ARGV.index('--delimiter') + 1]
        else
           ARGV[2]
        end

dest = if ARGV.include?('-t')
          ARGV[ARGV.index('-t') + 1]
       elsif ARGV.include? '--folder'
          ARGV[ARGV.index('--folder') + 1]
       else
          ARGV[3]
       end

ext = if ARGV.include?('-e')
         ARGV[ARGV.index('-e') + 1]
      elsif ARGV.include? '--extention'
         ARGV[ARGV.index('--extention') + 1]
      else
         ARGV[4]
      end

### Paths ###
files.each do |file_path|
  fullpath = File.expand_path(file_path)
  input = File.open(fullpath, 'r').read
  base_name = File.basename(file_path, 'md')
  parts = input.split(delim)

  parts.each do |i|
    filename = i.split("\n")[0]
    file_content = i.split("\n").drop(1).join("\n")

    if file_content =~ /[a-zA-Z]+/ && !filename.nil?
      new_file = File.open("#{dest}/#{base_name}/#{filename}.#{ext.gsub(/\./, '')}", 'w')
      new_file.puts file_content
    end
    puts "File created at #{dest}/#{filename}.#{ext.gsub(/\./,'')}" if i =~ /[a-zA-Z]+/ && filename != nil
  end
end

It feels so good each time I hit ⌘⌥E + return and such a beautiful document pop-up. With Scrivomatic there was always something preventing me from compiling directly to PDF from Scrivener, even files that otherwise would compile just fine upon opening. I ended up with my own custom ruby script to take the exported markdown file from Scrivener and do the necessary magic with Pandoc. It was a lot of work, but I learned a lot too about Ruby. Now with Quarto, more often than not, it will work seamlessly. Amazing.

1 Like

Hello there, I am trying out the Scrivener + Quarto template but I run into a problem where the source file specification that gets sent to Quarto on the command line ends with the file name repeated as in:

filename.md/filename.md

Then warning error messages are produced in the log file generated upon compilation.

The log file starts with:

--> Input Filename: /Users/deo/Desktop/QS Document/QS-1.md/QS-1.md

The final lines of the log file are

--> Running Command: quarto render /Users/deo/Desktop/QS Document/QS-1.md/QS-12.qmd --to html --log-level=INFO --verbose
e[91mERROR: No valid input files passed to rendere[39m

There was some problem opening /Users/deo/Desktop/QS Document/QS-1.md/QS-12.html, check compiler log…

Strangely, if I open the Scrivenor-generated .qmd file in VS Code the same doubling of the filename occurs in the file’s path that appears in the editor, as the screen shot illustrates.

IMG - File path in VS Code for Scrivener doc

Note that at the end of the file specification, “qs-1.md” appears before “qs-12.qmd”, and this occurs even when I remove “qs-1.md” generated by Scrivener from the .qmd file’s path.

In contrast, when I create a new .qmd file from scratch the path appears correctly in VS Code, terminating in the filename.qmd but without the leading filename.md. In this case the file I created is My_Quarto_Scratch-1.qmd and it is contained in the folder Quarto_From_Scratch, as illustrated in the screenshot.

Anyway, if either @nontroppo or @bernardo_vasconcelos or someone else could help me solve this problem so that the template file can compile on my M1 MacBook Air, then I’d be greatly appreciative.

This is a standard Scrivener “feature” — it will create a folder and then compile into the folder, so if you select a Desktop/Test.md for save, then Scrivener will create a folder Test.md and inside that folder save its compile files. This is because Scrivener will export any linked pictures and other bits so “wraps” it all up in a folder. To stop that, you can append -mmd to a pre-existing folder name. See §21.5.1 of the user manual for the details. I have a folder called compile-mmd on my desktop and compile into that folder and so Test.md simply becomes Desktop/compile-mmd/Test.md and the Quarto script converts to Desktop/compile-mmd/Test2.qmdand runs that file throughquarto`.

BUT, that shouldn’t cause an error. I tried compiling in Scrivener to ~/Desktop/Test.md/Test.md and Quarto could run just fine:

--> Running Command: quarto render /Users/ian/Desktop/Test.md/Test2.qmd --to html --log-level=INFO --verbose
2 Likes

Thanks so much for your prompt response, @nontroppo ! I recall reading in your Scrivomatic instructions that “-mmd” should be appended to the folder name but I had forgotten to actually do it. I will make that change and see if that fixes the compilation problem…

1 Like

I am still unable to compile, @nontroppo . I wonder if it has to do with the fact I had duplicated, moved and changed the name of the Scrivener file using Finder, outside of the Scrivener UI? Full log file follows

--> Input Filename: /Users/deo/Desktop/Scrivener-Quarto/SQ_1-Compilation_Folder-mmd/Scrivener-Quarto_1.md
--> Modified path: /usr/local/bin:/Library/TeX/texbin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/Scrivener.app/Contents/Resources/MultiMarkdown/bin
--> CrossRef figure details: Label={#fig-elephant .column-body}  | We add the *cross-referencing label* to the **_start_** of the caption. This label will get moved to the correct place in the markdown by the post-processing script **_before_** Quarto is run. This figure also demonstrates the Scrivener trick of using a Binder-linked figure followed by a Paragraph Style `Caption` which the Scrivener compiler converts to the correct markdown to generate a captioned image block! | Elephant1
--> CrossRef figure details: Label={#fig-castle}  | Elephant castle. | Elephant2
--> CrossRef figure details: Label={#fig-trunk}  | Angry elephant with big trunk. | Elephant3
--> CrossRef figure details: Label={#fig-withattributes .myclass fig-align="right" width=3cm height=2cm}  |  This figure uses custom metadata values to identify the class, ID, width and height. The ««A​B»» tag at the start of the caption is replaced with the correct Scrivener placeholders by the compiler; see global replacements for the details… | Elephant3-1
--> CrossRef figure details: Label={#fig-elespan}  | This should span the whole page. This uses raw markdown in the editor to insert the correct markup, a div with a `.column-page` class, for Quarto's layout for extend-to-page-width. | Elephant1-1
--> CrossRef figure details: Label={#fig-elespan3   width= height=}  |  This should span the page to the right in HTML. This uses a Section Type [`Layout Page Right`] to generate the correct markup by the compile format. | Elephant1-2
--> CrossRef figure details: Label={#fig-castle2}  | Elephant. | Elephant2-1
--> CrossRef figure details: Label={#fig-trunk2}  | Angry elephant with big trunk. | Elephant3-2
--> CrossRef figure details: Label={#fig-alignright fig-align="right"}  | This should be right-aligned if there is space… | Elephant3-3
--> Modified File with fixed cross-references: /Users/deo/Desktop/Scrivener-Quarto/SQ_1-Compilation_Folder-mmd/Scrivener-Quarto_12.qmd
--> Parsing took: 0.002074s

--> Running Command: quarto render /Users/deo/Desktop/Scrivener-Quarto/SQ_1-Compilation_Folder-mmd/Scrivener-Quarto_12.qmd --to html --log-level=INFO --verbose
e[91mERROR: YAMLException: duplicated mapping key (Scrivener-Quarto_12.qmd, 15:1)
14: e[34m#e[91m
15: e[34mtitle: "Quarto-Scrivomatic-1"e[91m
   ~~
16: e[34msubtitle: "A Compiler Workflow…"e[91me[39m

There was some problem opening /Users/deo/Desktop/Scrivener-Quarto/SQ_1-Compilation_Folder-mmd/Scrivener-Quarto_12.html, check compiler log…