Scriv4, Pandoc, Latex and citations

Scrivener Markdown & LaTeX
1

Hi,
I am just getting started with Scrivener 3. I am an academic (technical) writer, so I really need to use citations extensively. I also use Latex a fair bit, certainly enough to get tired of its editing, compiling converting to pdf cycle.
From what I have read, the best way to use citations and compile a bibliography with Scrivener is by using it with Pandora. But I have a few questions about this that I wonder if someone on this list would be kind enough to answer:

  1. My understanding is that Pandoc provides a way for me to produce many different document formats, including docx, latex, and others. But Scrivener seems to want to use MMD for latex, pdf, etc, and use Pandoc only for docx, xml and epub. Why not allow the use Pandoc for latex and other formats?

  2. Does anyone have a simple example that shows how to insert citations and compile a bibliography? (I currently use Papers 3 but may switch to Bookends).

  3. I have seen references to “Scrivomatic” (orthostat.wordpress.com/2014/06 … previewer/). These look like what I want to do - anyone have any experience with them, positive or negative?

But basically, any advice for getting started with citations/bibliography would be much appreciated.

Thanks.

2

Theses threads might be helpful.

There is a discussion on the use of Papers 3 with Scrivener 3 here: https://forum.literatureandlatte.com/t/papers-3-citations-working-in-scrivener-3/38301/1

A more general discussion on Scrivener in an academic workflow: https://forum.literatureandlatte.com/t/academic-non-fiction-workflow-dealing-with-citations/32755/1

3

(1) You can use Pandoc using the MultiMarkDown format post-processing Panel in Scrivener; this output format is unfortunately named (it was debated during beta testing, but ended up staying the same despite allowing both MMD and Pandoc conversion)…

(2) I have info for Bookends, but there are Papers3 users (@lunk comes to mind) who maybe can help you better. You need to be able to use a [@BiBTeXKey] style temporary citation copied from Papers into Scrivener. Then you simply pass a --bibliography=mybib.bib file to Pandoc and a CSL style and it does the rest. I can make a test project to show the Scrivener side if you want.

(3) I wrote the scrivomatic workflow, but you do not need to use my workflow, I’m happy to answer questions to guide you in your own way to run. I do think if you are outputting to more than one format, and perhaps several different templates within a format (I use two main LaTeX output styles for example, same for HTML, sometimes want to run reference conversion, sometimes not for DOCX), then using pandocomatic really makes everything much easier once you’ve set up your settings. You change one line in your Scrivener metadata, and change the whole compile chain without further fussing…

4

Thank you Kinsey and nontroppo (Ian?),
I will check those out. Regarding (2) above, would it be possible to please provide me with the small test project? I think once I see how the information is passed from Scrivener to Pandoc, I will be able to move forward from there.
Many thanks,

  • Phil
5

I’ve followed your advice by exporting to a text file and running Pandoc from the terminal with the --bibliography option. That works great. If I can figure out how to pass the Pandoc arguments from Scrivener I think I will be in business. It must have something to do with the compile options or settings?

6

Yeah, setting up custom command-line parameters is documented in the user manual under §24.22, starting on page 645. The Processing pane it refers to is something you would modify about a compile format itself, as documented at the top of that chapter, pg. 579.

7

Hi again,
Thanks, I tried using defining a new format after duplicating the MMD one. My compile format pane looks like the attached. However, when I compile it says “cannot produce pdf output from pdf”. (same result if I add <$inputfile> to the Arguments). I had thought the input file for the MMD compile format would be an md file?

8

Hi again,

I found that if I covert to docx instead of pdf it works. I.e., “–bibliography my bib.bib -t docx -o <$outputname>.docx” does the citation and makes the bibliography entry. I’m a little puzzled about why it won’t do a pdf, but this is enough to get my started. Thanks!

  • Phil
9

You can find a full Pandocomatic workflow project attached, it bundles all needed files into the Binder: a BiBTeX and CSL file for the bibliography, custom Word/CSS/LaTeX templates and the pandocomatic.yaml config file needed to specify the compile options. Read Resources Instructions in the Binder for details how to set this up (basically Export all resource files to the folder you will compile to, move pandocomatic.yaml to your pandoc data directory, change path for processing tool). The workflow includes examples of captioned figures, and compilation of custom inline styles like superscript and strong-emphasis…

Workflow.scriv.zip (322 KB)

10

Wow, that’s fantastic, nontroppo! it will take me a little while to digest, but looks like just what I was looking for. Thanks heaps for making it available. Cheers,

  • Phil
11

You may also want to look at TeXDown:

http://github.com/mnott/texdown

Here I’ve done a video about my larger workflow:

https://youtu.be/86rnBMz6XnU

12

Hi. I wonder if you can help me (particularly nontroppo) with a problem: I see to get different results when running scrivomatic from the command line and from Scrivener. [I imagine this has to do with environment variables, but I’m not sure which].

General info: my metadata instructs Pandocomatic to convert to LaTeX. Notice, I am running Scrivomatic with the “–build” option. So…

From the command line, I run:

/Users/xxx/bin/scrivomatic.rb workflow_mmd -v -b 2>&1 >>scrivomatic.log

It works perfectly.
But the very same code only produces the .tex when run from within scrivener, like so:

It seems that Scrivomatic is reading the “-b” properly. I see the following in the log:

#<struct Scrivomatic::OPT input="'workflow_mmd'", output=nil, to=nil, command="pandocomatic", envpath="/Users/zvb1/.rvm/wrappers/default:/Library/TeX/texbin:/Users/zvb1/bin:/usr/local/bin", build=true, verbose=true, dry_run=false, open_log=false>

But, alas, it doesn’t try to run latexmk after the pandoc conversion. This is the full log for the failed run is attached. Any ideas? Many thanks.

.
scrivomatic.txt (3.09 KB)

Untitled.jpeg
Untitled.jpeg756×476 74.8 KB

13

I’ve determined that Scrivomatic can’t find the .tex file (although it’s there). Not sure why.

14

same owner? read permissions? Does it get launched via globbing and need execute permissions?

What’s the log from a successful command line run look like?

15

zsteir: yes you are right, when Scrivener runs a command, it runs under a very basic environment which doesn’t contain any paths or other settings, and scrivomatic tries to find various tools and adds the paths, sets to the environment. I’ve updated scrivomatic to V1.0.13 to provide more extensive logging during the call to latexmk, and possibly fixed a problem where the default environment that scrivener uses may cause a text encoding issue.

github.com/iandol/scrivomatic/b … crivomatic

See if it fixes the problem. I’ve tested it with a complex document and it works fine here. By the way, I currently force the use of xelatex with latexmk, but you use pdflatex, not sure if this will cause problems. I could leave this open, or provide a command line option if it would help.

You can also use pandocomatic to output to PDF directly and then Pandoc calls latexmk rather than use scrivomatic to do it.

16

Thanks, nontroppo. Indeed, that did the trick! [I figured out the errant “$” in the gsub, but couldn’t get any further].

My issue was that I need to run Biber, and outputting directly to pdf (so pandoc calls latexmk) didn’t seem to do that. But no matter, this will work just fine. Thank you.

17

I’ve updated scrivomatic to V1.0.14 which checks if pandocomatic specifies the pdf-engine setting, then uses that instead of the default xelatex. Regarding biber, not quite sure why --biblatex option didn’t work for you, I’ve never used LaTeX for bibliographies so don’t really know what the process is for using direct Pandoc⇨PDF with Biblatex. Anyway, if Scrivomatic⇨Latexmk does work then great!

18

I am new concerning the handling of Scrivener. I would like to write my thesis with Scrivener. The advantages are obvious, but again and again I come across problems. I’ve worked a lot with latex and feel a bit more familiar with it, but I’ve got trouble getting started with Scrivener - scrivomatic and everything that follows.

The best way to describe the problem a bit. I downloaded “scrivomatic” and installed it according to the instructions. I have worked through the “getting started file” shown here in the forum. It lives. I can tell from exactly this file a .md file and it will also create a latex and a Word file. The latex file compiled. So far so good.

However, I would like to start my own Scrivener project and adapt it so that I can transfer it with the template (Eisvogel) in a latex document.

Now my problems start. How do I get the styles from the tutorial in my project? Does Scrivener subsequently create sections, subsections, … by the folder structure?

Is it enough to have .bib-file, scrivomatic, authorSimplifyMetadata, preventall, … in the target folder? Or must further precautions be taken there?

If this problem is solved and I can convert the first part of my thesis with scrivener I would like to use papers3 to incorporate the quotes. Are there traps there?

Thanks in advance

19

Hi fpramme, the getting started file is just a way to test this workflow without needing to install anything. Normally, the various filters and templates will live in subfolders of the Pandoc data directory (which is ~/.pandoc/ by default).

Editor styles can be imported into a new project from another project using Format > Style > Import Styles… — Compiler styles are part of the compile format, which can be either specific to a project (Project Formats)or shared across projects (My Formats). You can convert between these by editing the format and changing Save to….

Yes. The exact mapping between the Scrivener folder level and the LaTeX level can be controlled by changing the base-header-level metadata.

As I said above, normally these files are not stored in the target folder, this is just to make the getting started project work simply. I normally store scrivomatic in ~/bin/, all my templates in ~/.pandoc/templates etc. but the point is you can store them anywhere, as long as you edit the paths so Pandoc can find them. I would recommend to try to use the Pandoc data directory as I do though as it simplifies the paths significantly (Pandoc assumes files will be in its data directory by default). You can see my pandoc data directory structure as an example here:

github.com/iandol/dotpandoc

I store links (symlinks) to my .bib file in ~/.pandoc also, but these are actually stored in my Dropbox so they sync across all my machines. You can place your bib file wherever you want , again as long as you tell Pandoc the proper path.

Papers 3 can generate Pandoc style temporary citations so it should work fine (you need to configure this in preferences somewhere). I don’t use papers so don’t know the specific details, but as long as you can add citekeys and build a .bib you should be fine.

By the way, my dotpandoc already includes Eisvogel and you can use it by specifying it in the Scrivener metadata like so:

pandocomatic_:
  use-template: [docx-refs, eisvogel]

It can be fully configured from scrivener metadata or editing the pandocomatic.yaml directly:

github.com/iandol/dotpandoc/blo … .yaml#L179

20

Thank you very much for your help.

I started with a new and clean scrivener file. Than I insert a File at the top level called “Pandoc metadata”. After this I just copy the text out of your Tutorial-File.

Pandoc is installed via homebrew and the scrivomatic file is stored in /User/xxx/bin/scrivomatic and I selected this Path inside Scrivener.

I just insert some text in scrivener and now I am trying to compile this into an .m file. So I pressed the compile button, select scrivomatic-Format, use “Pandoc metadata” als front-metter and hit the compile button. Than I select a new folder called “output” and compile the project to an .m file. This works, and there is a scirvomatic.log and an html file now inside this folder. But no latex sources or an docx file.

I actually the pandomatic templates called docx, eisvogel and latex-with-toc. But there is just the html file inside my output folder.

[code]
=== ------------------------------------------------------ ===
=== Scrivomatic V1.0.18 Report @ 2018-05-08 10:06:15 +0200 ===
=== ------------------------------------------------------ ===
Running under Ruby 2.3.3
Working directory: /Users/florianpramme/Documents/Dissertation/thesis/Output
===------ Input Options: ------===
#<struct Scrivomatic::OPT input=“doctoral_thesis.m”, output=nil, to=nil, yaml=nil, command=“pandocomatic”, envpath="/Users/florianpramme/bin:/usr/local/bin:/Library/TeX/texbin", build=false, verbose=true, dry_run=false, open_log=false>
===------ Final ENV PATH: ------===
/Users/florianpramme/bin:/usr/local/bin:/Library/TeX/texbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
===------ TOOL PATHS: ------===
—pandoc: /usr/local/bin/pandoc | V: 2.2
—pandocomatic: /usr/local/bin/pandocomatic | V: Pandocomatic version 0.2.2.1
—ruby: /usr/bin/ruby | V: ruby 2.3.3p222 (2016-11-21 revision 56859) [universal.x86_64-darwin17]
—python: /usr/bin/python
—xelatex: /Library/TeX/texbin/xelatex
—latexmk: /Library/TeX/texbin/latexmk

… running pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /usr/local/bin/pandocomatic --debug doctoral_thesis.m
::: An unexpected error has occurred. You can report this bug via https://github.com/htdebeer/pandocomatic/issues/new.
::: /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:379:in parse': (<unknown>): could not find expected ':' while scanning a simple key at line 6 column 1 (Psych::SyntaxError) ::: from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:379:in parse_stream’
::: from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:327:in parse' ::: from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:254:in load’
::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandoc_metadata.rb:69:in load_file' ::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/command/convert_file_multiple_command.rb:50:in initialize’
::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandocomatic.rb:100:in new' ::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandocomatic.rb:100:in run’
::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/bin/pandocomatic:3:in <top (required)>' ::: from /usr/local/bin/pandocomatic:22:in load’
::: from /usr/local/bin/pandocomatic:22:in `’
:: exit status: pid 23908 exit 1

!!!—scrivomatic::runCommand() RETURN non-zero value: /usr/local/bin/pandocomatic --debug doctoral_thesis.m !!!

=== ------------------------------------------------------ ===
=== Scrivomatic V1.0.18 Report @ 2018-05-08 10:22:26 +0200 ===
=== ------------------------------------------------------ ===
Running under Ruby 2.3.3
Working directory: /Users/florianpramme/Documents/Dissertation/thesis/Output
===------ Input Options: ------===
#<struct Scrivomatic::OPT input=“doctoral_thesis.m”, output=nil, to=nil, yaml=nil, command=“pandocomatic”, envpath="/Users/florianpramme/bin:/usr/local/bin:/Library/TeX/texbin", build=false, verbose=true, dry_run=false, open_log=false>
===------ Final ENV PATH: ------===
/Users/florianpramme/bin:/usr/local/bin:/Library/TeX/texbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
===------ TOOL PATHS: ------===
—pandoc: /usr/local/bin/pandoc | V: 2.2
—pandocomatic: /usr/local/bin/pandocomatic | V: Pandocomatic version 0.2.2.1
—ruby: /usr/bin/ruby | V: ruby 2.3.3p222 (2016-11-21 revision 56859) [universal.x86_64-darwin17]
—python: /usr/bin/python
—xelatex: /Library/TeX/texbin/xelatex
—latexmk: /Library/TeX/texbin/latexmk

… running pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /usr/local/bin/pandocomatic --debug doctoral_thesis.m
::: An unexpected error has occurred. You can report this bug via https://github.com/htdebeer/pandocomatic/issues/new.
::: /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:379:in parse': (<unknown>): could not find expected ':' while scanning a simple key at line 6 column 1 (Psych::SyntaxError) ::: from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:379:in parse_stream’
::: from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:327:in parse' ::: from /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/2.3.0/psych.rb:254:in load’
::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandoc_metadata.rb:69:in load_file' ::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/command/convert_file_multiple_command.rb:50:in initialize’
::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandocomatic.rb:100:in new' ::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/lib/pandocomatic/pandocomatic.rb:100:in run’
::: from /Library/Ruby/Gems/2.3.0/gems/pandocomatic-0.2.2.1/bin/pandocomatic:3:in <top (required)>' ::: from /usr/local/bin/pandocomatic:22:in load’
::: from /usr/local/bin/pandocomatic:22:in `’
:: exit status: pid 24818 exit 1

!!!—scrivomatic::runCommand() RETURN non-zero value: /usr/local/bin/pandocomatic --debug doctoral_thesis.m !!![/code]