Scrivener + Quarto: a technical/academic publishing workflow

As I use homebrew to install and there was some problems with some other tools in homebrew that use pandoc but were abandoned, I only just upgraded myself.

  • Quarto bundle their own Pandoc, so that will probably stay at 2.x until they verify a safe upgrade. Installing V3.0 will not affect Quarto at all, so you can play with v3.0 and keep using Quarto.
  • Direct from Scrivener, if you use Pandoc via MMD post-processing you should be fine, as you are responsible for all the command-line options. If you use a defaults file then you will need to tweak your files if you want to access to some of the new features.
  • Direct from Scrivener + Scrivener’s compile settings: as @AmberV mentioned there shouldn’t be a breaking change, though as these formats are closed (in the sense we can’t see what the pandoc command is), there isn’t much we could do if there was, so if you are cautious stick to V2.x until someone else tries.
  • Direct from Scrivener + Scrivomatic, as I use Pandocomatic then this update is being tracked here: Investigate what pandoc version 3.0 means for pandocomatic · Issue #109 · htdebeer/pandocomatic · GitHub — I still need to test so the best bet for scrivomatic users is stick to V2.x until I’ve tested.

Note: pandoc’s executable is a single file so it is easy to install several versions and use symlinks to call them. In general as all template and data files (i.e. the LaTeX template, docx reference etc.) are embedded in the executable, it is only your own user data files you need to worry about.

2 Likes

Thanks, that is something I meant to point out as well. First thing I did was copy pandoc to pandoc2, before installing v3, which simply just put another pandoc there. If I run into a bug or workflow break and need a quick workaround, I can just call on the old binary. The only other thing the installer adds to the system are some man files. If I wanted to play it cautious I would have done it the other way around so that Scrivener would keep using v2 for a spell—but as the one that gets to stress test everything, I want the opposite of cautious. :slight_smile:

2 Likes

Since the last update, Quarto no longer renders directly from Scrivener. It hangs. Then I have to kill pandoc to unfreeze Scrivener. But it renders fine from the command line. Anyone else experiencing this?

I use the preview version (V1.3.x), and don’t see a hang. How are you calling quarto, and do you use logging (i.e. redirect stderr and stdout to a log file as in my demo template, pass --log-level=INFO --verbose to the command etc.)?

If I try to knit R code I see this error: ERROR: Internal error: temporarily disabled until deno 1.28.* gets puppeteer support (though it doesn’t hang), but I don’t normally use live R code so this doesn’t affect me, part of the price to pay with a beta version…

Thanks for getting back to me, Ian @nontroppo. I managed to pinpoint the cause of the issue. It wasn’t the update, but the the multibib filter. If I disable it, it works without problems. I have no idea as to why it does this, but it is definitively the cause.

I managed to fix it! To celebrate it, and the conclusion of my doctorate here is my augmented version of the excellent Scrivener + Quarto template.

Scrivener project file download

https://www.dropbox.com/s/tulnkeepgavck8r/Quarto-Augmented.scriv.zip?dl=1

Sample output

The changes to the original project are documented on page 22.

Changes to the original Scrivener project

Depth override

The section type Attributes .. Header & Text can be used to override the depth of the current binder item; that is to say, change the number of hashes before the title. Place the desired number of hashes in the Attributes metadata field with no space at the end (e.g. ###).

Automatic anchors

All of the Section Layouts received a new prefix: []{#scriv<$linkID>}. This allows us to create links to any binder item whatsoever. Particularly useful to link to sections that do not have a Header element.

Multiple bibliographies

There is a new Section Layout and a new Section Type to provide support for the Multibib/Multiple-bibliographies lua filter. In this way, we can create complex bibliographies with ease, spliting primary and secondary sources. Each bibliography section will be named after the binder item whence it originated, as it happens with any chapter of the text which makes it very easy to managem them.

Only the multiple-bibliographies filter is compatible with the template. The Multibib filter uses a single variable to store all of the file paths, so this creates problems for the template. To use: choose the Multibib section type, fill the Path custom meta-data field with the path to the bib file, and you’re set.

Indexes & glossaries

The multiple-bibliographies filter can be extended in interesting ways using the citation-backlinks filter that links back from the ref-entry in the bibliography to each inline citation in the text. This means that we get to make indexes using Citeproc that work both for LaTeX and HTML (see the output sample linked below).

Generating bibliography files

In order to create these separate bibliographies using Bookends, we could i. create a library for each bibliography section and sync the references to bibtex; ii. use an Applescript to export different Bookends groups as separate bibliography files.

Here is a script that will 1. ask for the prefered format, 2. the file extention and then 3. list all front library groups allowing the selection of multiple items. They will be exported to the folder especified in the BasePath variable.

Applescript for Bookends

set RevealAfterFinish to true
set UseGroupPaths to true -- {"Group 1", "Group 2"} OR {"Path/to/Group 1", "Path/to/Group 2"}
set BasePath to "/Users/bcdav/Dropbox/dev/pandoc/refs/"
set FlattenBool to {"Yes", "No"}
set MarkupList to {"Plain", "BibTeX", "HTML", "RTF"}
set FileExtensionOptions to {".bib", ".bex", ".json", ".md", ".qmd", ".ris", ".txt", ".xml", ".yml"}
set FormatList to {"BibTeX", "BibTeX4Zotero", "RIS", "Endnote XML", "Pandoc.fmt", "AAPG Bulletin.fmt", "ABNT Markdown.fmt", "ABNT MMD.fmt", "ABNT_MD.fmt", "ABNT.fmt", "Antiquity.fmt", "APA 5th Edition.fmt", "APA 6th Edition.fmt", "APA 7th Edition.fmt", "Berliner Blätter.fmt", "BibTeX 2.fmt", "BibTeX.fmt", "BibTeX2.fmt", "BibTeX4Export.fmt", "BibTeX4Export(2).fmt", "BibTeX4Pandoc.fmt", "Chicago 15th A.fmt", "Chicago 15th B.fmt", "Chicago 16th A.fmt", "Chicago 16th B.fmt", "Chicago 17th A.fmt", "Chicago 17th B.fmt", "Discourse Studies.fmt", "DOI.fmt", "DPG (Deutsche Gesellschaft für Psychologie).fmt", "DT3-URL.fmt", "MLA 6th Edition.fmt", "MLA 7th Edition.fmt", "MLA 8th Edition.fmt", "Nature.fmt", "Oxford Author-Date.fmt", "Oxford Notes.fmt", "Pandoc.fmt", "Printout.fmt", "Ref with Abstract.fmt", "Rename.fmt", "RIS 2.fmt", "RIS Export.fmt", "RIS.fmt", "RISwithNote.fmt", "ShortRef.fmt", "Turabian (7th ed).fmt", "Turabian Author-Date.fmt", "Turabian SBTS author-Date.fmt"}

tell application "Bookends"
	
	set theLibraries to library windows
	repeat with theLibrary in theLibraries
		if UseGroupPaths is true then
			set GroupList to my split(return, («event ToySRGPN» given «class PATH»:"true")) -- 1
			set GroupList to my simple_sort(GroupList) -- 1
		else
			set GroupList to name of every group item of theLibrary -- 2
		end if
		set GroupPaths to my get_opt("Bookends Groups", "Select the desidered groups to be exported as  individual bibliography files", GroupList, true)
		if GroupPaths is false then error "Please select some items."
		
		set theFormat to item 1 of my get_opt("The Format", "Select the desidered Bookends format", FormatList, false)
		set theMarkup to item 1 of my get_opt("The Markup", "", MarkupList, false)
		set theExtension to item 1 of my get_opt("The Extension", "The file extension the bibliography files should use", FileExtensionOptions, false)
		set Flatten to item 1 of my get_opt("Flatten", "Should the folder structure be flattened? 

If you select YES, then all of the files will be exported to BasePath.

If you select NO, then it will recreate the folder structure using /Path/to/Group.", FlattenBool, false)
		
		repeat with GroupPath in GroupPaths
			if UseGroupPaths is true then
				set theGroup to the last item of my split("/", GroupPath) -- 1
			else
				set theGroup to GroupPath
			end if
			set theResult to ""
			set theRefs to publication items of group item id theGroup of theLibrary
			
			repeat with theRef in theRefs
				if theMarkup is "BibTeX" then
					set theRef to format theRef using theFormat as BibTeX
				else if theMarkup is "HTML" then
					set theRef to format theRef using theFormat as HTML
				else if theMarkup is "RTF" then
					set theRef to format theRef using theFormat as RTF
				else
					set theRef to format theRef using theFormat
				end if
				-- Endnote XML | RIS | plain text
				-- Deprecated: set theRef to «event ToySGUID» theID given «class RRTF»:"false", string:"Bibtex"
				set theResult to theResult & theRef & linefeed
			end repeat
			
			if Flatten is "Yes" then set GroupPath to name of group item id theGroup of theLibrary
			set GroupPath to item 1 of (my split(".", GroupPath))
			
			--			set GroupPath to my gsub(my gsub(GroupPath, ".bib", ""), "Bib ", "")
			set ExportPath to BasePath & "" & GroupPath & theExtension
			--				return ExportPath
			my exportSource(theResult, ExportPath)
			--		return theGroup
			
		end repeat
		if RevealAfterFinish is true then
			tell me to do shell script "open " & quoted form of ExportPath & " -R"
		end if
	end repeat
end tell


on gsub(theString, old, new)
	set {TID, text item delimiters} to {text item delimiters, old}
	set theStringItems to text items of theString
	set text item delimiters to new
	set theString to theStringItems as text
	set text item delimiters to TID
	return theString
end gsub

on join(delim, theContent)
	set oldDelims to AppleScript's text item delimiters
	set AppleScript's text item delimiters to delim
	set theResult to theContent as string
	set AppleScript's text item delimiters to oldDelims
	return theResult
end join

on split(delim, theString)
	try
		set oldDelims to AppleScript's text item delimiters
		set AppleScript's text item delimiters to delim
		set delimitedList to every text item of theString
		set AppleScript's text item delimiters to oldDelims
	on error
		set AppleScript's text item delimiters to oldDelims
	end try
	return delimitedList
end split

on rb(rb_script)
	return do shell script "ruby -r ERB  <<EOF
" & rb_script & "
EOF"
end rb

on URLEncode(findThis)
	set findThis to paragraphs of findThis
	return my rb("puts ERB::Util.url_encode %Q{" & findThis & "}")
end URLEncode

on exportSource(theText, ExportedFile)
	set the clipboard to theText
	set theFolder to my join("/", (items 1 thru -2 of (my split("/", ExportedFile)))) as string
	tell me to do shell script "mkdir -p " & quoted form of theFolder
	set theScript to "touch " & quoted form of ExportedFile & " && LANG=pt_BR.UTF-8 pbpaste > " & quoted form of ExportedFile
	tell me to do shell script theScript
end exportSource

on simple_sort(my_list)
	set the index_list to {}
	set the sorted_list to {}
	repeat (the number of items in my_list) times
		set the low_item to ""
		repeat with i from 1 to (number of items in my_list)
			if i is not in the index_list then
				set this_item to item i of my_list as text
				if the low_item is "" then
					set the low_item to this_item
					set the low_item_index to i
				else if this_item comes before the low_item then
					set the low_item to this_item
					set the low_item_index to i
				end if
			end if
		end repeat
		set the end of sorted_list to the low_item
		set the end of the index_list to the low_item_index
	end repeat
	return the sorted_list
end simple_sort
on get_opt(theTitle, thePromt, theList, MultipleSelection)
	if MultipleSelection is not false and (count of theList) > 1 then
		set theOption to choose from list theList with title theTitle with prompt thePromt with multiple selections allowed
	else
		set theOption to choose from list theList with title theTitle with prompt thePromt
	end if
	if theOption is false then error "Aborted."
	return theOption as list
end get_opt

I hope these additions prove useful to someone at some point.

EDIT: Updated the Applescript to include new options (Markup) and fix some bugs.

4 Likes

Wow, congrats on finishing the doctorate; and a great use of Quarto and addition of multiple bibliographies. This is a prime example of the extensibility and flexibility of Pandoc workflows… :+1:

1 Like

Thanks, Ian @nontroppo. For the humanities, the ability to split up the bibliography is pretty much the difference between being able to use Citeproc (with bib JSON :cupid:) and getting stuck with BibLaTeX’s nightmarish compile times. The citation-backlinks, moreover, replace the best BibLaTeX feature and create the possibility of having indexes for both LaTeX (which I didn’t think possible without getting one’s hands dirty with tex) and HTML’s output.

The end result is very satisfying and I love how the pop-up in the HTML export makes it possible to jump around between different occurrences of the same reference, author, or concept.

1 Like

So I’ve tried this, aiming to use it for the next stage (getting my thesis ready for publication as a monograph), and got this error from the sample project:

JSON parse error: Error in $: Incompatible API versions: encoded with [1,22,2,1] but attempted to decode with [1,23].
Error running filter /Users/lyndon/.local/share/pandoc/filters/multiple-bibliographies.lua:
PandocFilterError "pandoc" "Filter returned error status 64"
stack traceback:
	.../.local/share/pandoc/filters/multiple-bibliographies.lua:50: in upvalue 'run_citeproc'
	.../.local/share/pandoc/filters/multiple-bibliographies.lua:82: in function <.../.local/share/pandoc/filters/multiple-bibliographies.lua:68>

Is this because my local pandoc is version 3?

Could you try using this Quarto extension I prepared? Just install it to the export folder and replace the filters in the YAML with citetools.

https://bcdavasconcelos.github.io/citetools/

I’ve installed it, but I still get the same error. I think the filter calls pandoc, which relies on the (Homebrew-installed) pandoc rather than the one embedded in quarto?

Could you share a dummy copy of the Quarto project/folder in which the error happens?

You can easily add a V2.x of pandoc (download the self-contained binary and just place it in a folder that is earlier in the path than homebrew), and see if that helps. Quarto bundles its own Pandoc, as AFAIK is still at V2.x even in the nightlies? Is the filter this one: lua-filters/multiple-bibliographies.lua at master · pandoc/lua-filters · GitHub — can you run it alone with a simple test document in V2.x vs. V3.x?

yep it’s just the Scrivener project exported (and with the change of filter you mentioned)
Quarto 2.zip (589.0 KB)

It now works fine. Is there any chance you can post a copy of the bibliography file you’re using, so that I can generate comparable output?

1 Like

Yes that worked. I think the issue is that the marshalling is done by the Quarto-included version (=2.x) and the filter then calls the system path one (=3.x in my case), and rightly checks the version number on demarshalling. Is there no way for filters to ensure that they access the Quarto-included version of pandoc, not the system one? I can’t think of a reason to prefer the system pandoc within a filter that Quarto has called.

1 Like

This is perhaps a question worth raising via the Quarto github issues page. I do not know what internal mechanisms quarto uses to try to isolate its pandoc, but it certainly doesn’t seem to be working properly given your example!

The bibliography files are in the research folder of the Scrivener project. Look for Authors and Glossary.

By the way @lyndondrake, the current preview version of Quarto does now bundle Pandoc V3.1.1:

▶︎ quarto --version
1.3.319

▶︎ quarto check

[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.1: OK
      Dart Sass version 1.55.0: OK

Cool, I’ve pulled those into a bib file. There are quite a few citations missing, and the cross-references don’t seem to work:

[WARNING] Citeproc: citation barrett2015 not found
[WARNING] Citeproc: citation copenhaver2014 not found
[WARNING] Citeproc: citation crivellato2007 not found
[WARNING] Citeproc: citation eq-one not found
[WARNING] Citeproc: citation eq-two not found
[WARNING] Citeproc: citation fig-airquality not found
[WARNING] Citeproc: citation fig-castle not found
[WARNING] Citeproc: citation fig-elephant not found
[WARNING] Citeproc: citation fig-elephants not found
[WARNING] Citeproc: citation fig-elephants2 not found
[WARNING] Citeproc: citation fig-elespan not found
[WARNING] Citeproc: citation fig-marginalia not found
[WARNING] Citeproc: citation fig-mermaid not found
[WARNING] Citeproc: citation fig-statemachine not found
[WARNING] Citeproc: citation fig-withattributes not found
[WARNING] Citeproc: citation hoffman2014 not found
[WARNING] Citeproc: citation siegel2015 not found
[WARNING] Citeproc: citation simmons2013 not found
[WARNING] Citeproc: citation tbl-first2 not found
[WARNING] Citeproc: citation tbl-panel2 not found
[WARNING] Citeproc: citation tbl-second not found
[WARNING] Citeproc: citation tbl-second2 not found
[WARNING] Citeproc: citation tbl-test not found
[WARNING] Citeproc: citation barrett2015 not found
[WARNING] Citeproc: citation copenhaver2014 not found
[WARNING] Citeproc: citation crivellato2007 not found
[WARNING] Citeproc: citation eq-one not found
[WARNING] Citeproc: citation eq-two not found
[WARNING] Citeproc: citation fig-airquality not found
[WARNING] Citeproc: citation fig-castle not found
[WARNING] Citeproc: citation fig-elephant not found
[WARNING] Citeproc: citation fig-elephants not found
[WARNING] Citeproc: citation fig-elephants2 not found
[WARNING] Citeproc: citation fig-elespan not found
[WARNING] Citeproc: citation fig-marginalia not found
[WARNING] Citeproc: citation fig-mermaid not found
[WARNING] Citeproc: citation fig-statemachine not found
[WARNING] Citeproc: citation fig-withattributes not found
[WARNING] Citeproc: citation hoffman2014 not found
[WARNING] Citeproc: citation siegel2015 not found
[WARNING] Citeproc: citation simmons2013 not found
[WARNING] Citeproc: citation tbl-first2 not found
[WARNING] Citeproc: citation tbl-panel2 not found
[WARNING] Citeproc: citation tbl-second not found
[WARNING] Citeproc: citation tbl-second2 not found
[WARNING] Citeproc: citation tbl-test not found
[WARNING] Citeproc: citation barrett2015 not found
[WARNING] Citeproc: citation copenhaver2014 not found
[WARNING] Citeproc: citation crivellato2007 not found
[WARNING] Citeproc: citation eq-one not found
[WARNING] Citeproc: citation eq-two not found
[WARNING] Citeproc: citation fig-airquality not found
[WARNING] Citeproc: citation fig-castle not found
[WARNING] Citeproc: citation fig-elephant not found
[WARNING] Citeproc: citation fig-elephants not found
[WARNING] Citeproc: citation fig-elephants2 not found
[WARNING] Citeproc: citation fig-elespan not found
[WARNING] Citeproc: citation fig-marginalia not found
[WARNING] Citeproc: citation fig-mermaid not found
[WARNING] Citeproc: citation fig-statemachine not found
[WARNING] Citeproc: citation fig-withattributes not found
[WARNING] Citeproc: citation hoffman2014 not found
[WARNING] Citeproc: citation siegel2015 not found
[WARNING] Citeproc: citation simmons2013 not found
[WARNING] Citeproc: citation tbl-first2 not found
[WARNING] Citeproc: citation tbl-panel2 not found
[WARNING] Citeproc: citation tbl-second not found
[WARNING] Citeproc: citation tbl-second2 not found
[WARNING] Citeproc: citation tbl-test not found
[WARNING] Citeproc: citation barrett2015 not found
[WARNING] Citeproc: citation copenhaver2014 not found
[WARNING] Citeproc: citation crivellato2007 not found
[WARNING] Citeproc: citation hoffman2014 not found
[WARNING] Citeproc: citation siegel2015 not found
[WARNING] Citeproc: citation simmons2013 not found

Cool! Do you know a way to get homebrew to pull the preview version? I imagine it won’t be long now before the full release as I can see the 1.3 milestone is marked 100% in Github Issues anyway.