Workflow no longer works after Mac update, help needed to export Scrivener to Word with live citations

[Sicahjes]

Short version: is there a simple guide on how to get a document with citekeys out of scrivener and into word, preserving live citations?

Details:
I was using a workflow where I would write in Scrivener and then eventually have a word document with live citations (necessary for my supervisor etc.). Basically:

1. Have a scrivener doc, and write.
2. Use an Apple script to insert citations from Zotero into Scrivener (which still works),
3. Compile to multimarkdown.
4. Open the multimarkdown using DocDown, which would convert the file (using Pandoc) to a Word doc with live citations based on my preferred template.

It wasn’t perfect, I needed to adjust some formatting, but it got me there. Since the last Mac update (15.3) this no longer works. The citekeys are just copied as plain text and Zotero no longer recognises them. When I try to refresh I get this error: You must insert a citation before performing this operation.

I’ve spent the last two days trying to fix this, to no avail. I’ve even gone the full Pandoc route, installing it using home-brew and getting it to convert the compiled file to .docx but the citations are still plain text.

Below is the most recent command I tried.

pandoc "path to scrivener compile file" --citeproc --bibliography="path to zotero bib file" -o "path to output"

When it is converting from a multimarkdown compile, it converts the citations but they’re not live (or formatted as needed). When converting from an rtf, the citations are plain text and not reformatted at all.

I’ve seen the Scrivener templates with some setup for pandoc on here, but I’m now so overwhelmed and lost (especially as I can’t figure out the cause of the problem) that I can’t figure out how to use them or where to begin.

I’m now 2-days behind on an incredibly tight deadline. I’m really hoping I can avoid having to manually enter the citations into Word (although in hindsight, it would have been faster).

Can anyone point me to a simple guide?

Note: I’m using zotero 7, my citations are formated in Scrivener as [@authorshortitleYYYY]

[nontroppo]

@AmberV – perhaps you can move the recent posts from on Citation managers - #62 by Sicahjes to here as I missed this post thread and replied over there; but it would be better if they were here…

[Sicahjes]

All of a sudden, Docdown is no longer recognising citations. I have a chapter written in Zotero without a way to render them. I’ve now installed Pandoc and I have a reference file etc. Could someone please recommend a simple guide to follow to get my work out of Scrivener with citations… I think I’m just going to have to go back to Word. The fear of losing everything is real!

[nontroppo]

Don’t panic, you haven’t lost anything. What were your compile options from Scrivener for docdown? I still don’t know what it is but IIRC you said it was based on Pandoc anyway so it should be easy to move to Pandoc directly.

For Pandoc:

• Compile for Multimarkdown:
  
• For simplicity, you can add title and author as metadata:
  
  
  &
  
• On the left side you can select a format, choose Basic Pandoc
  then right click and duplicate this format.
• Give a name, you can now customise this format as your needs require
  
  
• There are many compile settings to tweak your workflow, but lets ignore them for the moment.
• Click Compile, select a folder and use the .md file suffix. Lets assume the folder is called compile-mmd on your desktop, ou’ve now got a ~/Desktop/compile-mmd/file.md compiled from your project, let us now convert it with Pandoc.

---

• Open terminal
• First you copy your BibTeX file generated by Zotero and your CSL style to the same folder you compiled to, I will assume they are in your Documents folder (you can also do this with Finder, but lets use shell commands here):

➪ cp ~/Documents/biblio.bib ~/Desktop/compile-mmd
➪ cp ~/Documents/apa.csl ~/Desktop/compile-mmd

• Change directory to your compile folder

➪ cd ~/Desktop/compile-mmd

• lets check we have our files:

➪ ls 

biblio.bib     apa.csl       file.md

• Looking good. So now we run pandoc, we pass in the options to build what we need, in this case our bibliography and csl file and want to make a docx:

➪ pandoc --verbose --citeproc --output file.docx --csl apa.csl --bibliography biblio.bib file.md

[INFO] Running filter citeproc
[INFO] Loaded apa.csl from apa.csl
[INFO] Loaded test.bib from test.bib
[INFO] Completed filter citeproc in 55 ms
[INFO] Loaded xkcd.png from xkcd.png

• And open our new shiny docx:

➪ open file.docx

Scrivener can run pandoc directly for us, you can check the SimpleCitations project I shared to see it in action, in this case the DOCX with citations is automatically generated without any further manual work. You can use styles to get things like strong, emphasis, code, maths, quotations into the docx. You can also use a reference.docx which allows you to set up the Word or Libreoffice document with the fonts and styles you want.

[Sicahjes]

Oh, I’ve been in panic mode for 2-days now!

Thank you (again!). I’ll give this a go now… after making myself a strong cup of tea! I have downloaded your SimpleCitations project, but my rather simple brain couldn’t wrap its mind around it. It looks like there’s a bib file in there, but I am often updating my library. I like the idea of being able to do everything from Scriv, though. Is there a wiki/step-by-step for that process?

As an FYI, below is the most recent, I was just focused on getting the citations rendered live, not formatting:

pandoc "path to scrivener compile file" --citeproc --bibliography="path to zotero bib file" -o "path to output"

When it is converting from a multimarkdown compile, it converts the citations but they’re not live (or formatted as needed). When converting from an rtf, the citations are plain text and not reformatted at all.

[Sicahjes]

I just followed these steps, and it compiled the document using the bib file, but they’re not live. When I click refresh in Zotero, it returns the following error: You must insert a citation before performing this operation.

[nontroppo]

The citations are formatted and final. If you want to change the formatting, use a different CSL file. This workflow is solid and reliable, it assumes you write in Scrivener and do not need to fiddle with the output.

There is a Pandoc filter that generates live citations from the BBT developer:

Markdown/Pandoc :: Better BibTeX for Zotero

You need to add some metadata (you can use the Compiler UI, I prefer using a front-matter doc, or add it manually), and call the filter from the command line, or Scrivener’s post-processor. In theory this could all be automated. But as I’ve mentioned previously, I dislike the way Zotero / Endnote / Mendeley stuff citations into hidden fields, so I never use this filter myself…

[Sicahjes]

I don’t exclusively work in Scrivener, mainly due to the revision process. I need the ability to export live citations to Word. I did come across the info you linked, but it assumes a little knowledge that I don’t have.

I’m going to try deleting and reinstalling docdown, and then if that doesn’t work see if I can make the filter work.

Thanks for your time and assistance

[nontroppo]

@Sicahjes

but I am often updating my library. I like the idea of being able to do everything from Scriv, though. Is there a wiki/step-by-step for that process?

Right, I do not keep my BIB file in my Scrivener project (I add it to SimpleCitations just for demonstration). My own workflow is that I store my BIB file in Dropbox (Bookends automatically syncs its database to that BIB file), then I use a symlink to my local Pandoc data folder (~/.local/share/pandoc), and reference the symlink.

[Sicahjes]

Cool. It’s something to keep in mind for when I don’t need editable citations in Word.

[nontroppo]

The BIB file is always needed, whether the citations are live or not. I just tested this filter, and it works fine for me:

> pandoc --verbose --citeproc --csl apa.csl --bibliography test.bib --reference-doc custom.docx -L zotero.lua -o out.docx test.md 
[INFO] Running filter citeproc
[INFO] Loaded apa.csl from apa.csl
[INFO] Loaded test.bib from test.bib
[INFO] Completed filter citeproc in 64 ms
[INFO] Running filter /Users/ian/.local/share/pandoc/filters/zotero.lua
zotero-live-citations cd1ef3f
[INFO] Fetching https://retorque.re/zotero-better-bibtex/exporting/zotero.lua.revision...
new version "199d652" available at https://retorque.re/zotero-better-bibtex/exporting
[INFO] Fetching http://127.0.0.1:23119/better-bibtex/json-rpc?%7B%22jsonrpc%22%3A%222%2E0%22%2C%22params%22%3A%7B%22style%22%3A%22apa%22%2C%22asCSL%22%3Atrue%2C%22citekeys%22%3A%5B%22barrett2015%22%2C%22siegel2015%22%2C%22hoffman2014%22%2C%22copenhaver2014%22%2C%22simmons2013%22%2C%22crivellato2007%22%5D%7D%2C%22method%22%3A%22item%2Epandoc%5Ffilter%22%7D...
[INFO] Completed filter /Users/ian/.local/share/pandoc/filters/zotero.lua in 87 ms
[INFO] Loaded xkcd.png from xkcd.png
[INFO] Loaded custom.docx from custom.docx

To remove the “Do Zotero Refresh”, a simple refresh did not work, I needed to change the bibliography style.

out 2.docx.zip (69.1 KB)

[Sicahjes]

Thanks, I just got ChatGPT to give me the steps to set this up, and am in the process of doing it now:

1. Download the Lua filter (zotero.lua) from the official repository and save it in the same folder as your Markdown file.
2. Add metadata to your Markdown file (optional but recommended). At the top of the file, include:

---
zotero:
  client: zotero  # or jurism if using it
  csl-style: apa  # Change to your preferred style
  sorted: true  # Sorts clustered citations by author
---

3. Run the Pandoc command with the Lua filter:

pandoc -s --lua-filter=zotero.lua --bibliography=path_to_your_bib_file.bib -o output.docx input.md

Replace path_to_your_bib_file.bib with your .bib file’s location and input.md with your Markdown file. This ensures that citations are live and updated with Zotero’s data.

With the Lua filter in place, Pandoc will process the citations correctly, pulling them live from your Zotero library when converting to Word.

[nontroppo]

For anyone that wants it, I added a new compile format to deonstrate the “live” citations in the SimpleCitations example project:

SimpleCitations.scriv.zip (352.0 KB)

[Sicahjes]

The above didn’t work for me. It lists all of my cite keys with ‘not in zotero’ (but it works when not live citations).

I’m now looking at your file and my brain is positively fried at this point.

I cannot tell you how much I regret moving to Scrivener…

[Sicahjes]

I’m at the stage where I need to update the post processing, but I’m afraid I don’t know what ‘environments’ I’m meant to be adding. Betterbibtext is a plugin in Zotero, am I meant to be pointing to the Zotero app?

If I leave it blank, I get the error: This file could not be created. There was a problem generating the file using ‘pandoc’.

Edit: I’ve added /opt/homebrew/bin to environment, but I still get the error.

Edit 2: I missed the bit about the extra downloads, as I had already downloaded homebrew, pandoc etc. when trying the other methods. Installing now in the hope this fixes it.

[nontroppo]

@Sicahjes

I’m at the stage where I need to update the post processing, but I’m afraid I don’t know what ‘environments’ I’m meant to be adding. Betterbibtext is a plugin in Zotero, am I meant to be pointing to the Zotero app?

I would stick to running pandoc manually for the moment:

Do things step by step. In this workflow YOU are the post-processor. Scrivener generates markdown and you take it from there. You do not need to enter anything in the post-processor, no environment etc. needed. It only takes a few seconds to run the pandoc command manually…

When you are comfortable running pandoc manually and you can generate a working DOCX, and you have some time, then you can play to get Scrivener’s post-processor working. The environment field are the additional paths that the command can access. For your usecase, you should be able to leave it blank if you specify the path as /opt/homebrew/bin/pandoc? But again, make sure you are only doing this once you are comfortable with the manual path. I use the environment field when I am using typst or latex, in these cases pandoc needs to find the additional tools. But for DOCX it shouldn’t be needed.

[nontroppo]

@Sicahjes

If I leave it blank, I get the error: This file could not be created. There was a problem generating the file using ‘pandoc’.

Leaving it blank works for me as long as you enter the full path where pandoc is installed.

[Sicahjes]

So close, yet so far.

I unchecked post-process on command line and was able to export the compile. I used the earlier instructions on running the pandoc and got a docx file that converted the cite keys to readable citations.

pandoc --verbose --citeproc --output output-file.docx --csl chicago17-note.csl --bibliography Zotero-Pandoc-Library.bib chapter.md

However, they’re not live. When I click on ‘refresh’ in the Zotero ribbon, I get the below error.

Thank you so much for your help, but there’s something going on that’s preventing the citations from being recognised in Word. For this to work, I need to really understand pandoc, and I don’t. I need to go home and go to bed now. Tomorrow I’ll manually update the footnotes (all 106 of them) and move away from Scrivener (at least until after I submit my doctorate). I just can’t risk it.

[nontroppo]

@Sicahjes

pandoc --verbose --citeproc --output output-file.docx --csl chicago17-note.csl --bibliography Zotero-Pandoc-Library.bib chapter.md

You of course should use what works best for you. But FYI you are not using the zotero.lua filter[1] so you cannot expect the citations to be live…

---

[1] when you use the zotero filter you don’t need to pass --citeproc as the filter is taking this over, e.g. pandoc --verbose --output output-file.docx --csl chicago17-note.csl --bibliography Zotero-Pandoc-Library.bib -L zotero.lua chapter.md

[Sicahjes]

Thank you for the additional information. Unfortunately, it still did not work. I’m also noticing that sometimes the terminal will show a successful run, and other times I get a list of errors saying xyz citation could not be found (the latter when the workflow is design for live citations, I think).

An issue with the OS update meant that some of my apps needed to be reinstalled, including Zotero and Scrivener. After the testing* and the various options I’ve tried thus far, I’m guessing the issue might be a setting in one of those programs… but I think I’ve illustrated that I don’t really know what I’m doing, so who knows.

I need to try and catch-up today, but I may look into it later. Close to/after midnight sicahjes is in panic mode and wants to pack it all in (including the thesis), but light of the day sicahjes is open to fixing it, because a stable Scrivener > Word workflow would be fantastic.

As always, thank you for your generosity. I appreciate your support and time.

* When I first encountered issues with the DocDown workflow, I tried exporting a different chapter I’d previously exported successfully, but this time it failed (I have my chapters in different Scrivener projects).