I’ve spent many hours looking at forums, youTube videos, web searches, and etc. trying to find a well documented way to approach writing a biography with citations and a bibliography that allows me to focus on writing biographic content without worrying about the format until the end.
I’ve started down the path using my Mac with Scrivener 3 as my primary writing application with the General Non-Fiction(Latex) template. I chose to use latex/biblatex within Scrivener to create my citations and bibliography. I use Zotero with the Better BibTex extension as my bibliography manager.
My process for doing citations includes
copying the Zotero citation identifier into my biography in Scrivener using the \cite{} latex command.
Exporting the bibliography from Zotero into a .bib file for compilation with the biography’s .tex file.
Compiling in Scrivener for “Plain text (.txt) with the LaTex(Memoir) project format and its default values for the other compilation settings
This isn’t working for me. The citations are undefined and the bibliography doesn’t get created in the document.
I learned from some online searches that the process for creating a LaTeX document that uses biblatex for citations and the bibliography involves a four-stop process in order to get all of the references linked up correctly: pdflatex <biography.tex>, biber <biography.bib>, pdflatex <biography.tex>, pdflatex <biography.tex>.
I confirmed this four-step process using TexShop and pure LaTeX files with the .bib file exported from Zotero. The final .pdf correctly inserted the citations and included a bibliography of the referenced citations.
My problem now is to figure out how to get Scrivener to mirror this four-step compilation process or find another simpler way to do this. I’m such a newbie in this arena and the terminology is often unfamiliar that the discussions of possible options, processes, and workflows, i.e. MMD, Pandoc, EndNote, BookEnds, and etc. leave me totally confused. In my head I’m thinking … “this isn’t rocket science and not the first time a biography with research references has been created outside of MS Word. Why is this so difficult?”
I would be very grateful if someone can point me to a process/set of applications that can help me solve my biography project issues. I want to use Scrivener but am open to other paths besides the one I referenced above.
Unless you have prior experience with LaTeX that leads you to opt for it as your target output, or unless your editor/publisher is demanding it (which I suspect is not the case), then I strongly recommend that you target markdown output which you’ll then process with pandoc to produce whatever output format you want—docx, LaTeX, or pdf. Scrivener can be set up for this, and indeed @nontroppo is an expert at helping people get up and running with a setup like this.
If you tie yourself to a LaTeX output from the beginning, you will curse yourself later if you ever have to produce a docx version of your work. Trust me.
There are many of us that use reference manager software with Scrivener, and no, it’s not rocket science. But there are a specific set of steps that need to be done in the right order. Here is what I would do, trying to simplify
Don’t depend on only a LaTeX workflow: I’ve used LaTeX for years, have modified LaTeX for specific outputs. While it excels at detailed and complex layout tasks, it is difficult to tweak and hard to understand when it goes wrong. There is a ton of trial and error, and endless trips to Google for every detail. The LaTeX template in Scrivener is great, but it guides you to LaTeX-only with a PDF as the final product.
Do use a compile path that uses Markdown as the output. Markdown is far more flexible, it can export to many more outputs so doesn’t lock you into one path. You can use LaTeX as an output, but are not locked into that path, and can compile the same project to LibreOffice/Word/EPUB etc.
Scrivener supports two tools that work to convert markdown to several outputs: MMD and Pandoc. They both support LaTeX / ODT / HTML. BUT Pandoc supports many other outputs MMD doesn’t, and is much more flexible[1].
Pandoc supports bibliographies directly from Scrivener. This is a massive plus. Pandoc has a citeproc engine that takes temporary citations and turns them into formatted bibliographies without any extra work.
As far as Scrivener goes, that means you can use a general non-fiction template. For writing, you set Better BibTeX to output Pandoc citations (see below). For compiling you use the MMD option, and if you want a final product without any more fussing, use post-processing in Scrivener to run Pandoc for you.
I can quickly craft a simple Scrivener template to do that later today if it would help. I have multiple academic Scrivener templates already crafted (search for scrivomatic / Quarto / Tufte Book / Typst), and you can see some of the outputs here all with citations and bibliographies direct from Scrivener: scrivomatic/sample-output at master · iandol/scrivomatic · GitHub
The second much less important point: while Zotero is free, it is harder to integrate with Scrivener and the workflow is a little bit more complex. You’ll find that most Scrivener users that depend on bibliographies that are active on the forum use Bookends, whose feature set and simplicity of use are offset by the fact that you have to purchase it.
I outline workflows for several different reference managers here:
As Bookends is macOS / iOS only, I have set up several students and collegues up with either JabRef or Zotero workflows and both are viable. To use Zotero+BBT with Scrivener+Pandoc:
This creates citations like [@Jones2022]. You can “optimise” the workflow by using a UI to select citations, the wiki outlines several options (the best ones use Alfred as a floating UI).
[1]: Pandoc is ~30MB download, MMD is ~1.5MB which is why Scrivener bundles MMD.
It uses Scrivener styles for writing (no markdown is in the editor, this will be created by the compiler). There are 4 compiler formats: DOCX | EPUB | TypstPDF | TeXPDF
You need to install pandoc at a bare minimum. I recommend https://brew.sh homebrew for this. brew can also install typst (a 10MB engine that can create PDFs) and I assume you already have TeX installed. There is a test.bib and apa.csl file that needs to be in the folder you compile to: one trick is to use a folder name that has -mmd at the end of the name so Scrivener will leave any files in the folder alone.
This contains the markdown and the sample outputs, just to give you a sense of how the Scrivener text ends up after the compiler does its thing:
And here is an example of how easy it can be to input citations using Alfred and this great workflow (it uses a bib file as generated by Bookends / JabRef / Zotero+BBT):
Well, the easiest way is to use File ⇨ Export ⇨ Files…:
Although the UI is a bit misleading, it does export formats like .bib or .csl untouched. Note I also create a folder called SimpleCitations-mmd – this folder can now be used as a “permanent” Scrivener desitnation, files inside it will not be deleted so test.bib and apa.csl remain in place. You could move the .bib and .csl file to another place then you must edit the Metadata.
This screencast shows the compilation into that folder, you’ll note the test.bib and apa.csl files remain in place and also how amazing fast Pandoc+Typst is to render a formatted final PDF!!!
ScrivQ is great, but pretty complex, so it is better to work up slowly to this. Also Quarto has some benefits but also some downsides, so it depends on your requirements ultimately. Quarto is basically Pandoc + Opinionated templates and filters — it is more structured, but also less freeform. The good thing is most of this is interchageable, Quarto filters from ScrivQ are actually pandoc filters etc…
Features: multiple bibliographies, reference backlinks, and the ability to pull in arbitrary content from a reference into the text (cite text). For more advanced bibliographies these are useful additions, and reinforce the strong advantages of Scrivener + Pandoc
You all probably don’t have the faintest idea how much this thread has helped me!
A big thanks to @mtgardn for posting the question and an even bigger one to @nontroppo for the superb and very pedagogical explanation, not to mention the example files. This is the first time any of the “Markdown and Pandoc stuff” made sense to me! I’m on the “quest” to have this all working because I feel the RTF/ODF Scan plugin in Zotero is “too fragile” in the sense that it’s no longer actively worked on and only receives little maintenance and bug squashing. I even downloaded Bookends and will be looking carefully into this solution.
@nontroppo I followed your instructions on GitHub; overall, I think I did everything right.
I still have a couple of questions, so I’d like to give some background on what I’m trying to achieve.
I had one big project for everything: i) notetaking in some disciplines I did, ii) the thesis itself and iii) articles and working papers. I felt I was loosing control of it all, so splited everything, now I have two scriveners projects, one for my thesis and another for articles.
So my questions.
Can I just set the Resources folders in each of these two Scriv Projects, use Zotero+BBT, to sum up citations, and compile it all? I’ll have two folders on my iCloud, one Thesis-mmd and the other Articles-mmd, with all of the necessary resources. That’s basically it, right?
Suppose I need the Chicago Style instead of the APA one. Can I have both inside the Resource folder?
Besides that, can you point me to any material that will help me better understand the Scrivomatic/Pandocomatic stuff so that I can eventually tweak the “recipes”? Tweak in the sense to make it simpler. I’ll make sure to read your GitHub project page a couple of times more; for people like me, it does seem like “rocket science.”
Hi Felipe, glad some of this has been helpful to you. Just to be clear, you could get the SimpleCitations project linked on this thread to work, and you have also tried my more detailed sample scrivomatic workflow too?
Regarding project management, I also split into big projects (grants), papers and then some research projects. It is not common I need to combine the writing outputs of these[1], and I keep my resources for pandoc in a central location.
What exactly are you keeping in your Scrivener Resources folder? I usually keep research docs and PDFs in my live projects, but in my example projects (like SimpleCitations or scrivomatic) I do keep the accessory files like CSL or BIB. For teaching others this is helpful but my daily workflow my CSL / BIB / templates are all stored separately. In my case I use the Pandoc Data Directory, normally located at ~/.local/share/pandoc[2]. In this I have my main reference database, Core.bib (which in my case is synced automatically by Bookends), a CSL folder with my different CSL styles, and any other file I need. I don’t keep these files in the Compile-mmd compiling folder, mostly because I want the same files accessible to multiple projects. It sounds as if it would be better for you to use the same way to keep the compiling resources like CSL etc.?
Yes, you can keep them all together (as I do), anywhere on disk. Keeping them in the Pandoc Data Dirrectory is easiest, as Pandoc looks there when searching for CSL files. But you could also copy them into each compile folder if you want, that also would work…
The pandocomatic docs are helpful: Heer de Beer.org—Pandocomatic – and for a good example of a working recipes file, my working pandocomatic.yaml shows you almost everything you may want to do. Remember you don’t have to use pandocomatic, there is a simple “recipe manager” built-in to pandoc: Pandoc Defaults files. I prefer pandocomatic as it is more powerful, but both work fine.
I hope I understood your questions. There are several ways to achieve the same goals, which I think can be worrying if you are not too familiar with this way of working. My advice is start as simple as possible, and work forwards when you want to improve the workflow to suit your needs.
I’ve run into a snag using the Pandoc + Citations TeXPDF format. I haven’t found a way to insert a page break. Unlike the ‘Modern’ format that is available from Scrivener where a Chapter Title begins with a ‘page break’ followed by the title, there aren’t any formats that include the ‘page break’ and there’s not option to include in the separators drop down box. How can I add the ‘page break’ to my project formats?
Another issue I see is that the Table of Content’s markdown symbol “<$p>” doesn’t get replaced and all of the entries are just concatenated one right after the other with not formatting, i.e. a carriage return/line feed. How do I get the ToC to be be formatted correctly.
There are a bunch of solutions to this. The simplest IMO is to edit the compile format and use the Section Layout for the Sections to add a page break as a prefix:
Now you’ll see a \pagebreak TeX command before each section. If you want breaks only for level 1 sections then you’ll need to create another section type without the pagebreak command. Section Types are a powerful tool for structuring your work, including at the compilation stage.
You should be able to use the Separators panel to insert the same \pagebreak{} command; I tend to keep everything in the Section Layout as a personal preference.
If you want page breaks in DOCX, EPUB and Typst there is a nice pandoc filter that allows all these formats to understand the \pagebreak{} command:
For the TOC, you should rely on LaTeX, not use Scrivener’s TOC. To activate the native TOC
you add toc: true to the metadata:
I noticed a small problem with the figure numbering. My sample project uses Scrivener’s cross-referencing, but LaTeX and Typst both number figure captions, so you end up with: Figure 1: Figure 1 in the captions. One solution would be to remove the Scrivener cross-refs, but at least for LaTeX and Typst it is easy to disable the numbers in captions. To do so edit your metadata to add some header includes, these get added to the respective default templates to modify the output. I also added a mainfont and sansfont to demonstrate adding some more control, there are a few more of these variable to tweak outputs:
# Standard Scrivener + Pandoc Project YAML metadata
# IMPORTANT: YAML configuration files use SPACES for indentation to
# define structure, remember to straighten any smart quotes, and don't
# capitalise metadata keys like "author" or "title", they must be
# lowercase! Use "Show Invisibles" in Scrivener to see spaces.
title: "A Sample Project With Citations"
author: Jane Doe
lang: en
date: <$fulldate>
comments: "To err is human; … err?"
bibliography: test.bib
csl: apa.csl
link-citations: true
link-bibliography: true
notes-after-punctuation: false
toc: true
mainfont: 'Alegreya'
sansfont: 'Alegreya Sans'
fontsize: 13pt
header-includes: |
```{=typst}
#set figure(numbering: none)
```
```{=latex}
\usepackage[format=hang,labelfont=bf,labelsep=endash,font={sf,small}]{caption}
\captionsetup[figure]{labelformat=empty}
```
This gets captions back to normal… You can add more tweaks using this mechanism as you customise the output…
