Hi, hello.
I have no idea where to ask this! I am using iandol’s Scrivomatic setup on macOS (and trying to wrangle it into my own despite a general lack of knowledge of pandoc & co).
Issue 1:
When I compile my Scrivener document into Word .docx (and Markdown as part of the process), the tables disappear. The text isnide the cells is still there, but it is separated with ‘xxx’ for the rows.
After looking everywhere in the documentation, I don’t see anything on whether one should do anything specific in Scrivener to compile tables properly in Markdown.
What am I missing?
Issue 2:
When compiling into LaTex, the log shows the following error:
After searching the web, I found a stackexchange post with a similar issue, but the fix in it didn’t work.
Again, any idea as to what I could be missing?
Issue 3:
I am totally at a loss as to where to start to create a table of content or append a bibliography using Scrivomatic. I am sure there are resources out there, and I would be happy to read them, but I might need a helpful hint to point me in the right direction or as to where to start.
I know iandol’s resource page hints that there is something to be done with embedding links inside Scrivener to refer to figure or elements, but I wasn’t able to quite understand how that works.
Thank you in advance for any and all help! I hope this wasn’t too confused, I’m afraid being confused myself doesn’t help with defining these issues properly.
On question one: to confirm, have you ticked the option, in the General options tab on the right side of the main compile overview window, to Convert tables and lists to MultiMarkdown? Without that, the result will be much like you describe I think, with the text linearised. (Pandoc can read MultiMarkdown lists and tables, so don’t mind the specificity of that option.)
As to the rest, I am less familiar with the specifics of Scrivomatic configuration, so hopefully someone can answer those questions.
The error seems to suggest you have Pandoc configured for references (though you say below you need a bibliography…)? \citeproc is a LaTeX command used in the newer Pandoc LaTeX templates:
This LaTeX error suggests you have triggered Pandoc for refs but you are using a custom older template that may not have this command? If you are using an old version of my custom LaTeX template you should upgrade the template (or downgrade pandoc). I keep all my Pandoc templates here (I keep the templates up-to-date with the latest Pandoc):
In general as Pandoc is updated to fix bugs and add features, templates will also change. For anyone that does use my templates (which work with my academic metadata structure), using git to keep up-to-date with the templates is the easiest thing (you can automate running git pull every week or so and git allows you to roll back and forwards in time as required)…
My custom templates mostly deal with academic metadata, so if you don’t need that then you can revert back to the standard templates by removing the template key from the pandocomatic recipe.
TOC
For a TOC you use Pandoc’s --toc toggle, which with metadata means you add toc: true or table-of-contents: true to pandocomatic.yaml or the document metadata. Here is an example for my typst recipe:
This can be added to your Scrivener metadata directly too I think to override anything in the pandocomatic recipe.
Bibliography
For a Bibliography, you enter references using the [@citekey] pandoc format in your text. Then when you compile, your pandoc settings / pandocomatic recipe needs the following ingredients:
Tell pandoc to use citeproc
Tell it where your bibliography file is
Tell it where your CSL style file is
So for example in my pandocomatic recipe I define a recipe for references:
My bibliography file Core.json is automatically created by Bookends, and I keep it in my pandoc data dir so no need for a path here but you should add the path. The other settings are just tweaks and not necessary. By specifying bibliography and citeproc in the recipe then pandoc knows to make the bibliography for you. It normally goes at the end of the document, but pandoc has a trick to place it anywhere which the Scrivomatic template uses, let me know if you need help on that…
This is for academia, and I am using Zotero and BetterBibtex, with synced JSON/BIBTEX exports in my pandoc folder.
I also have the zotero.lua filter set up to ensure live citations for docx documents. I did note that templates seem to work perfectly fine from the .yaml configuration file, but the Scrivener frontmatter override does not seem to be working as intended, when, for example, I tried to set a specific .csl configuration or just now with the toc command. Similarly, I couldn’t get the wordcount line to work.
Just for clarity (and in case my mistake was indentation), here’s what I had:
However, I tested the table-of-contents option in my pandocomatic.yaml file and it works perfectly. Brilliant and thank you so much!
For my purposes, getting it to work with Word is already great and a major win, but LaTex handles tables much more smoothly from what I’ve seen — it’s just a tad more complex to use for the novice.
I initially did clone your repo as-is but since I wanted to tweak the Word template, I ended up moving it to a subdirectory. It is recent from a few days ago, though, so I’m unsure what went wrong if this is working for you. I should note that LaTex is invoked because I am attempting to use the “pdf-refs” template!
Edit: I figured out what was going wrong! I didn’t realise that the conversion to .docx uses the pandocomatic.yaml that is inside the output folder while the conversion to .latex and .pdf uses the one in the pandoc folder, and it was set up to a .csl format that only covers footnotes
Issues 2 and 3 are solved!
Quick question on this topic — I see that this sets up in-document links from the footnotes to the bibliography, do you know if there is a way to make these two-way links (ie. clicking on the footnote sends to the bibliography entry, clicking on that entry sends back to the footnote)?
Thank you again, for your kind answers and for this really wonderful tool.
Right this is an indentation error, note you need two spaces BEFORE pandoc key then 4 spaces before table-of-contents as the pandoc key is supposed to “belong” to the pandocomatic key. Both of these would have worked:
here pandoc is the child of pandocomatic so it will be read. Many settings can go in the main metadata, but for some settings it is a bit of trial and error…
You could also give Typst a go, pandoc exports to Typst (I have a recipe in my dotpandoc) and it usually has less errors and is easier to tweak than LaTeX.
Do note that this filter bypasses Pandoc’s citeproc (I think Zotero’s citeproc-js engine does the processing), but possibly the metadata is the same (I did recently help someone on another post to get zotero.lua working, in theory you don’t need the bibliography or csl key for this filter but i did notice different behaviours with and without these keys).
git has a cool feature called branches, so you can fork a repo then make a local branch, that means you can make changes to some files and still pull the other files. I often keep the master or main branch of someone elses tool, then make my own feature branch then pull or merge into the custom branch, it means i can always keep their master up to date.
Right, this would definitely complicate things. For my workflow I do try to ensure everything is referenced only to my pandoc data dir.
There is this filter that can give mutliple refs back to each location in the text:
After trying this out, Scrivener compiler hangs. I am probably doing something wrong:
pandoc:
table-of-contents: true
The first one does nothing.
I have seen you note this in your workflow, but I have no idea how you get that to work — every time I compile a Scrivener document, it spews out a bunch of files including all the images in the project. How do you keep track when you have several projects?
Regarding zotero.lua, I think it does bypass citeproc but the resulting flexibility in changing out styles on the fly is a really good trade-off for me.
Thank you for the tip on the filter, I will have a look. Is citeproc also what I would need to look into if I want to have not a single bibliography at the end, but a references section after each chapter?
Hm, i tested both in the scrivomatic test project when I replied and both work to enable the TOC. But I did use toc: true rather than table-of-contents: true, and double checking the manual, the in-document metadata suggests to use toc: Pandoc - Pandoc User’s Guide — --table-of-contents is a command line and table-of-contents: is used in the defaults files, so hopefully this explains the discrepancy why it works for me but not for you…
For some sample projects I recommended a simple manual export of some pandoc[omatic] requirements into the compile folder as that simplifies trying the workflow out, but for longer term use: delete the compile folder you previously used, create a brand new empty folder (remember the name should end in -mmd), do NOT put pandocomatic.yaml or any lua filters or CSL files in that compile folder. Pandoc has its own data directory [PDD], by default this is ~/.local/share/pandoc — inside this folder keep ALL your compiler files: ~/.local/share/pandoc/csl for CSL, ~/.local/share/pandoc/filters for filters, ~/.local/share/pandoc/templates for compile templates etc. pandocomatic expects pandocomatic.yaml to be in the root of this folder, e.g. ~/.local/share/pandoc/pandocomatic.yaml.
For bibliography bib/json: I use a symlink ~/.local/share/pandoc/Core.json so I use the PDD to access my database, though in fact this is symlinked from my dropbox so I don’t keep this 18mb file itself in git but do sync it among my machines.
Where you keep files it ultimately up to you but with everything visible in PDD as real or symlinks, there is “one ring to rule them all”…
Right, I didn’t mean to discourage you, just to mention the workflow and metadata will have some subtle differences (i.e. say you change your CSL in your pandocomatic metadata, it may show in LaTeX but not DOCX even though you use the same values etc).
See this filter:
In general, pandoc filters can do a lot of the heavy lifting. @bernardo_vasconcelos has some nice documentation for this and other bibliography tweaks on his citetools docs: Cite Tools - Multiple Bibliographies — he works with quarto so the instructions are not immediately applicable but can be modified to work with vanilla pandoc…
My conversions to .docx relying on pandocomatic.yaml being present in the -mmd directory.
This was, in fact, due to using the Scrivener format of your workflow document as a template for Scrivomatic, where you have these arguments in the post-processing:
Now that I have fixed this, I properly point to the root pandoc folder for every compile.
Continue issues with the frontmatter metadata overrides.
Given my previous findings, I looked into the post-processing again and found that, indeed, the Script I originally had was different from the one listed here.
#!/usr/bin/env ruby
# encoding: utf-8
# scrivomatic is a wrapper script that adds tools to the path (as Scrivener
# does not use the user's path), and enables some other tweaks to optimise
# the workflow when scrivener calls pandocomatic.
# `scrivomatic --help` for details…
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
require 'open3' # ruby standard library class to handle stderr and stdout
require 'optparse' # ruby standard option parser
require 'fileutils' # ruby standard library to deal with files
require 'shellwords' # escapes strings to run in the shell
##
# main scrivomatic class
class Scrivomatic
##
# Sets up the path and performs other functions to improve running
# pandocomatic from scrivener...
attr_accessor :options
attr_reader :version, :cmd, :runLog
VER = '1.0.36'.freeze
OPT = Struct.new(:input, :output, :to, :yaml, :command, :envpath, :build, :cleanup, :verbose, :dry_run, :open_log, :data_dir)
DEFENVPATH = ENV['HOME'] + '/bin'
#-------------------------------class constructor
def initialize
@options = OPT.new(nil, nil, nil, nil, 'pandocomatic', DEFENVPATH, false, false, false, false, false, nil)
@version = VER
@fileext = ''
@cmd = ''
@toolPath = ''
@runLog = ''
@latexEngine = 'xelatex'
end
#-------------------------------run all options
def run
preBuild # removes old tex file if we are building
makePath # build the path
printInfo # print some information
buildCommand # build the pandoc[omatic] command
runCommand # run the pandoc[omatic] command
postBuild # run latexmk if we are building
end
#--------------------------------if we will run latexmk, remove old tex file before running pandoc
def preBuild
return unless options.build && !(options.dry_run)
texPath = options.input.gsub(/#{@fileext}$/, '.tex')
texFile = texPath.gsub(/\\+/, '')
if File.exist?(texFile)
puts "\n===------ Delete OLD TeX File: #{texFile}: ------===" if @options[:verbose] == true
system("latexmk -quiet -c #{texFile} >/dev/null 2>&1")
File.delete(texFile)
end
end
#-------------------------------make env path
def makePath
home = ENV['HOME'] + '/'
pathtest = [home+'/.rbenv/shims',
'/usr/local/opt/ruby/bin', '/usr/local/lib/ruby/gems/2.7.0/bin',
home+'Library/TinyTeX/bin/universal-darwin', '/Library/TeX/texbin',
'/opt/homebrew/bin', '/usr/local/bin',
home+'/anaconda/bin', home+'/anaconda3/bin',
home+'/miniconda/bin', home+'/miniconda3/bin', home+'/micromamba/bin',
home+'/.cabal/bin', home+'/.local/bin',
rvm_check]
pathtest.each { |p| @options[:envpath] = @options[:envpath] + ':' + p if File.directory?(p) }
ENV['LANG'] = 'en_GB.UTF-8' if ENV['LANG'].nil? # Just in case we have no LANG, which breaks UTF8 encoding
@options[:envpath].gsub!(%r{(//)}, '/') # remove double slash
@options[:envpath].gsub!(/(::)/, ':') # remove double colon
@options[:envpath].gsub!(/:$/, '') # remove final colon
ENV['PATH'] = @options[:envpath] + ':' + ENV['PATH']
end # end makePath()
#-------------------------------print initial report
def printInfo
return if @options[:verbose] == false
puts "\n=== ------------------------------------------------------ ==="
puts '=== Scrivomatic V' + @version + ' Report @ ' + Time.now.to_s + ' ==='
puts '=== ------------------------------------------------------ ==='
puts ' Running shell: ' + `printf $SHELL`
puts ' Working directory: ' + `pwd`
puts " Initiating with Ruby #{RUBY_VERSION}"
puts '===------ Input Options: ------==='
puts @options
puts '===------ Final ENV PATH: ------==='
puts ENV['PATH']
puts '===------ TOOL PATHS: ------==='
puts `echo "---pandoc: $(which pandoc) | V: $(pandoc -v | sed -nE '1 s/^pandoc // gp')"`
puts `echo "---ruby: $(which ruby) | V: $(ruby -v)"`
puts '!--ruby version incompatible with new pandocomatic, see https://github.com/iandol/scrivomatic/blob/master/Installing-Ruby.md' if `ruby -v` =~ /ruby 2\.3\.\d/
puts `echo "---pandocomatic: $(which pandocomatic) | V: $(pandocomatic -v | sed -En '1s/^Pandocomatic version /''/p')"`
puts '---paru library: V: ' + `ruby -e 'require "paru"; puts Paru::VERSION.join(".")'`
puts `echo "---rbenv versions:"; [[ -x $(which rbenv) ]] && rbenv versions`
%w[rbenv rvm gem panzer python xelatex latexmk].each do |c|
location = `which #{c}`.chomp
puts "---#{c}: #{location}" unless location.empty?
end
puts "\n … running #{@options[:command]}, please wait …\n"
end
#-------------------------------build the command line
def buildCommand
@cmd += ' --data-dir="' + @options[:data_dir] + '"' unless @options[:data_dir].nil?
@cmd += ' --output ' + @options[:output] + ' ' unless @options[:output].nil?
@cmd += ' --to "' + @options[:to] + '"' unless @options[:to].nil?
@cmd += ' -c ' + @options[:yaml] if !@options[:yaml].nil? && @options[:command] == 'pandocomatic'
@cmd += ' --debug' if @options[:verbose] == true && @options[:command] == 'pandocomatic'
@cmd += ' ---debug "panzerlogs"' if @options[:verbose] == true && @options[:command] == 'panzer'
@toolPath = `which #{@options[:command]}`.chomp
@cmd = @toolPath + @cmd + ' ' + @options[:input] + ' ' unless @options[:input].nil?
end
#-------------------------------run the command
def runCommand
`open scrivomatic.log` if @options[:open_log] && File.exist?('scrivomatic.log')
puts '===------ COMMAND OUTPUT: ------===' if @options[:verbose] == true
if File.exist?(@toolPath) && !options.dry_run
puts ":: Running: #{@cmd}\n" if @options[:verbose] == true
Open3.popen2e(@cmd) do |_stdin, oe, thread|
while (line = oe.gets)
puts '::: ' + line.chomp if @options[:verbose] == true
check_engine = line.match(/pdf-engine=(\w+)/)
@latexEngine = check_engine[1] unless check_engine.nil?
end
exit_status = thread.value
puts ':: exit status: ' + exit_status.to_s if @options[:verbose] == true
unless exit_status.success?
puts "\n!!!---scrivomatic::runCommand() RETURN non-zero value: #{cmd}!!!"
end
end
elsif !options.dry_run
puts "Tool doesn't exist!!!" if @options[:verbose] == true
puts "\n!!!---scrivomatic::runCommand() Couldn't find #{@toolPath} to run, please supply a proper path!"
elsif @options[:verbose] == true
puts 'Dry run, nothing actually executed...'
end
end
#-------------------------------parse inputs
def parseInputs(_arg)
optparse = OptionParser.new do |opts|
opts.banner = 'Scrivomatic V' + @version + "\n"
opts.banner += "=======================\n"
opts.banner += "Scrivomatic is a wrapper around pandocomatic or panzer, that sets up the environment path, enforces UTF8 encoding and other settings so they can be run from any other process that may not do this (e.g. Scrivener).\n\n"
opts.banner += 'Usage: scrivomatic [additional options] FILE'
opts.on('-i', '--input FILE', 'Input file') do |v|
v.gsub!(/(\A'|'\Z)/, '')
@options[:input] = v.shellescape
@fileext = Regexp.escape(File.extname(@options[:input]))
end
opts.on('-o', '--output [file]', 'Output file. Optional for pandocomatic.') do |v|
@options[:output] = v.shellescape
end
opts.on('-t', '--to [format]', 'Pandoc Format. Optional for pandocomatic.') do |v|
@options[:to] = v
end
opts.on('-y', '--yaml [file]', 'Specify which YAML file for pandocomatic.') do |v|
@options[:yaml] = v.strip.shellescape
end
opts.on('-c', '--command [command]', 'Tool to use: [pandocomatic] | panzer') do |v|
@options[:command] = v
end
opts.on('-p', '--path [dirpath]', 'Additional Path to Search for Commands.') do |v|
@options[:envpath] = v.strip.shellescape + ':' + @options[:envpath]
end
opts.on('-b', '--build', 'For LaTeX output, run latexmk') do |v|
@options[:build] = v
end
opts.on('-B', '--buildclean', 'For LaTeX output, run latexmk and cleanup') do |v|
@options[:build] = v
@options[:cleanup] = v
end
opts.on('-d', '--dry-run', 'Dry run.') do |v|
@options[:dry_run] = v
end
opts.on('-z', '--data-dir [file]', 'Pandoc data dir.') do |v|
@options[:data_dir] = v.strip.shellescape
end
opts.on('-v', '--[no-]verbose', 'Verbose output.') do |v|
@options[:verbose] = v
end
opts.on('-l', '--[no-]log', 'View log in Console.app.') do |v|
@options[:open_log] = v
end
opts.on('-h', '--help', 'Prints this help!') do
puts optparse
exit(0)
end
end # end OptionParser
optparse.parse!
# make sure we have an input file
return unless @options[:input].nil?
# otherwise check if we got passed the file
if ARGV.nil? || ARGV[0].nil?
puts optparse
abort "\n\n!!!---scrivomatic::parseInputs requires valid input file: --input"
else
v = ARGV[0].gsub(/(\A'|'\Z)/, '') # scrivener sometimes passes the file wrapped in '
@options[:input] = v.shellescape # we assume it was passed without -i flag
@fileext = Regexp.escape(File.extname(@options[:input]))
end
end # end parseInputs
#-------------------------------check for RVM
def rvm_check
rvm_home = ENV['HOME'] + '/.rvm'
return '' unless File.directory?(rvm_home)
rvm_home + '/wrappers/default'
end
#------------------------------check if we want to run latexmk
def postBuild
return unless options.build && !(options.dry_run)
texPath = options.input.gsub(/#{@fileext}$/, '.tex')
texFile = texPath.gsub(/\\+/, '')
pdfFile = texPath.gsub(/\.tex/, '.pdf')
if File.exist?(texFile)
if File.exist?(pdfFile)
puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose] == true
File.delete(pdfFile)
end
puts "\n===------ RUN LATEXMK on #{texPath}: ------===" if @options[:verbose] == true
@latexEngine = 'pdf' if @latexEngine =~ /pdflatex/
xcmd = "latexmk -logfilewarnings -interaction=nonstopmode -f -pv -time -#{@latexEngine} -f #{texPath}"
puts ":: directory: #{Dir.pwd}" if @options[:verbose] == true
puts ":: command: #{xcmd}" if @options[:verbose] == true
begin
Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
while (line = oe.gets)
if line.chomp.to_s =~ /^(Latexmk:|Run|LaTeX|This is|===|Accumulated|Missing|! )/
puts '::: ' + line.chomp if @options[:verbose] == true
end
end
exit_status = wait_thr.value
puts ':: exit status: ' + exit_status.to_s if @options[:verbose] == true
if exit_status.success? && @options[:cleanup] == true
logPath = File.basename(options.input, '.*') + '.log'
FileUtils.cp(logPath, 'latexlog_' + logPath) if File.file?(logPath)
`latexmk -C -quiet`
puts ":: Clean-up: used latexmk -c, but kept the latex build log as #{'latexlog_' + logPath}" if @options[:verbose] == true
elsif !exit_status.success?
puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
end
end
rescue StandardError => e
puts e
end
else
puts "!!!---Scrivomatic postBuild: could not find #{texPath}"
end
end
end #--------------- end Scrivomatic class
scriv = Scrivomatic.new
scriv.parseInputs(ARGV)
scriv.run
Having updated my script as well, I found myself troubled by the resulting log…
After looking into it, --enable pandoc-verbose is indeed a proper, documented command in pandocomatic. I updated my ruby install and my gems, to no avail.
This is rather puzzling — and I am no closer to understanding why I can use “toc: true” before “pandocomatic” in the frontmatter, but changing the csl does not work at all.
I suspect the command does not overwrite but add to existing instructions.
On a happier note, I managed to fiddle with the rest of the filters you recommended and they are fantastic! Thanks again.
Right in Workflow.scriv I set this sample up to run from the compile directory to make it easier to setup for a new user. I’ve added some more explicit instructions on what to do if you want to convert to using Pandoc data folder in the latest Workflow.scriv.zip – would this text have been helpful for you:
Right this and --log option are new. What is the full output now of your scrivomatic.log, it may be a wrong ruby is being called and thus still an old pandocomatic, scrivomatic does log which versions are being used, like:
I’m not sure in your case, but there are some subtle differences in metadata types. some are booleans, some allow single values and some combine values. Also some relate to settings, and others to variables used in templates. What exactly do you want, to specify your CSL in your frontmatter?
Pandoc filters are indeed great for finessing output. With a bit of time, it is well worth learning some Lua as with a bit of Lua + a GPT you can do quite a bit…
You must edit the Scrivener format used for compiling and change the post-processing arguments in the ‘Processing’ sub-menu: by deleting the -y "pandocomomatic.yaml pointer, pandocomatic will automatically go back to looking for this file in its default directory (instead of the compile target folder).
It just feels a little bit clearer to people who are not as tech-literate.
Got it! It’s running Ruby 2.6.10 — the macOS default and not my brew install. Now to fix this…
After spending hours trying to fix this, I am somewhat at a loss.
My terminal correctly points:
❯ which ruby
/opt/homebrew/opt/ruby/bin/ruby
❯ ruby -v
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24]
❯ which pandoc
/opt/homebrew/bin/pandoc
❯ pandoc -v
pandoc 3.6.3
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: /Users/me/.local/share/pandoc
❯ which pandocomatic
/opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic
❯ pandocomatic -v
Pandocomatic version 2.0.1
I have tried updating #!/usr/bin/env ruby to #!/opt/homebrew/opt/ruby/bin/ruby at the top of the script, and it is now correctly initating ruby (in fact I can also straight up make the script run using only the “ruby” shell command since my PATH is correctly set), but inside the script makePath needs to be updated — and probably other methods.
I don’t have the skills to do this, even with some friendly AI help.
Scrivener does not use the user’s environment (what you see when you open terminal), and so by default any post-processing tool cannot use the extra paths and tools available in the terminal. Only system paths are available. This annoying limitation was why I wrote scrivomatic in the first place, to add back any possible path that may contain a tool useful to compilation. So I support a bunch of extra paths for package manager tools like brew, rbenv, pixi, TeX etc. should all be supported. But non-standard paths are not added and so…
On Apple Silicon homebrew normally installs all binaries to /opt/homebrew/bin (and on Intel to /usr/local/bin). scrivomatic cannot find your ruby because it is not in either of these. Thus it will use an old pandocomatic etc. I previously recommended using rbenv to install ruby, but I am currently using Getting Started - Pixi by prefix.dev for my packages (i.e. i havent tested homebrew’s ruby since before there were Apple Silicon macs) so I tried removing my pixi package and reinstalling ruby using homebrew, and you are right. This is probably changed behaviour, and the reason is the following (brew tells you this on installing):
ruby is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.
If you need to have ruby first in your PATH, run:
echo 'export PATH="/opt/homebrew/opt/ruby/bin:$PATH"' >> ~/.zshrc
Note: it is ok for the system ruby to run the scrivomatic script to begin with (the top of the log can show an old ruby), as scrivomatic then will add the new ruby path and pandocomatic will run with the new ruby…
I actually used this comprehensive guide for my install — stuck to homebrew because I use Ruby only for Scrivomatic and my coding is usually in Python. Side note: this website is great and has a ton of tips for installing environments.
Wonderful! I’ll test it out.
After a whole day of hitting my head against a wall, I finally had what looked like a working solution too:
#!/opt/homebrew/opt/ruby/bin/ruby
# encoding: utf-8
# scrivomatic is a wrapper script that adds tools to the path (as Scrivener
# does not use the user's path), and enables some other tweaks to optimise
# the workflow when Scrivener calls pandocomatic.
# `scrivomatic --help` for details…
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
require 'open3' # ruby standard library class to handle stderr and stdout
require 'optparse' # ruby standard option parser
require 'fileutils' # ruby standard library to deal with files
require 'shellwords' # escapes strings to run in the shell
#require 'debug/open_nonstop' # debugger, use binding.break to stop
##
# main scrivomatic class
class Scrivomatic
##
# Sets up the path and performs other functions to improve running
# pandocomatic from Scrivener...
attr_accessor :options
attr_reader :version, :cmd, :runLog
VER = '1.0.41'.freeze
OPT = Struct.new(:input, :output, :to, :yaml, :command, :envpath, :build, :cleanup, :verbose, :dry_run, :open_log, :data_dir)
DEFENVPATH = ENV['HOME'] + '/bin'
#-------------------------------class constructor
def initialize
# Default command is 'pandocomatic', but force the Homebrew version if present.
default_command = 'pandocomatic'
homebrew_pandocomatic = "/opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic"
if File.exist?(homebrew_pandocomatic)
default_command = homebrew_pandocomatic
end
@options = OPT.new(nil, nil, nil, nil, default_command, DEFENVPATH, false, false, false, false, false, nil)
@version = VER
@fileext = ''
@cmd = ''
@toolPath = ''
@runLog = ''
@latexEngine = 'xelatex'
end
#-------------------------------run all options
def run
preBuild # removes old tex file if we are building
makePath # build the path
printInfo # print some information
buildCommand # build the pandoc[omatic] command
runCommand # run the pandoc[omatic] command
postBuildTeX # run latexmk if we are building
postBuildTypst # run typst if we are building
end
#--------------------------------checks if a file is less than 3 minutes old
def isRecent(infile)
File.exist?(infile) && (Time.now - File.mtime(infile)) <= 60
end
#--------------------------------if we will run latexmk, remove old tex file before running pandoc
def preBuild
return unless options.build && !(options.dry_run)
texPath = options.input.gsub(/#{@fileext}$/, '.tex')
texFile = texPath.gsub(/\\+/, '')
if File.exist?(texFile)
puts "\n===------ Delete OLD TeX File: #{texFile}: ------===" if @options[:verbose]
system("latexmk -quiet -c #{texFile} >/dev/null 2>&1")
File.delete(texFile)
end
end
#-------------------------------make env path
def makePath
home = ENV['HOME'] + '/'
# Updated order: ensure Homebrew gem bin comes before /usr/local/bin
pathtest = [
home + 'bin',
'/opt/homebrew/lib/ruby/gems/3.4.0/bin', # Homebrew Ruby gems bin (for pandocomatic v2.0.1)
'/opt/homebrew/bin',
'/opt/homebrew/opt/ruby/bin', # Homebrew Ruby bin
'/usr/local/bin',
home + '.pixi/bin',
home + '.rbenv/shims',
home + '.pyenv/shims',
home + 'Library/TinyTeX/bin/universal-darwin',
'/Library/TeX/texbin',
home + 'anaconda/bin',
home + 'anaconda3/bin',
home + 'miniconda/bin',
home + 'miniconda3/bin',
home + 'micromamba/bin',
home + '.cabal/bin',
home + '.local/bin'
]
pathtest.each { |p| @options[:envpath] = @options[:envpath] + ':' + p if File.directory?(p) }
ENV['LANG'] = 'en_GB.UTF-8' if ENV['LANG'].nil? # Ensure LANG is set
@options[:envpath].gsub!(%r{(//)}, '/') # remove double slashes
@options[:envpath].gsub!(/(::)/, ':') # remove double colons
@options[:envpath].gsub!(/:$/, '') # remove trailing colon
ENV['PATH'] = @options[:envpath] + ':' + ENV['PATH']
end
#-------------------------------print initial report
def printInfo
return if @options[:verbose] == false
puts "\n=== ------------------------------------------------------ ==="
puts "=== Scrivomatic V#{@version} Report @ #{Time.now} ==="
puts "=== ------------------------------------------------------ ==="
puts " Running shell: " + `printf $SHELL`
puts " Working directory: " + `pwd`
puts " Initiating with Ruby #{RUBY_VERSION}"
puts "===------ Input Options: ------==="
puts @options
puts "===------ Final ENV PATH: ------==="
puts ENV['PATH']
puts "===------ TOOL PATHS: ------==="
puts `echo "---pandoc: $(which pandoc) | V: $(pandoc -v | sed -nE '1 s/^pandoc // gp')"`
puts `echo "---ruby: $(which ruby) | V: $(ruby -v)"`
puts `echo "---pandocomatic: $(which pandocomatic) | V: $(pandocomatic -v | sed -En '1s/^Pandocomatic version /''/p')"`
puts "---paru library: V: " + `ruby -e 'require "paru"; puts Paru::VERSION.join(".")'`
puts `echo "---rbenv versions:"; [[ -x $(which rbenv) ]] && rbenv versions`
%w[pixi rbenv rvm gem python xelatex latexmk].each do |c|
location = `which #{c}`.chomp
puts "---#{c}: #{location}" unless location.empty?
end
puts "\n … running #{@options[:command]}, please wait …\n"
end
#-------------------------------build the command line
def buildCommand
@cmd += ' --data-dir="' + @options[:data_dir] + '"' unless @options[:data_dir].nil?
@cmd += ' --output ' + @options[:output] + ' ' unless @options[:output].nil?
@cmd += ' --to "' + @options[:to] + '"' unless @options[:to].nil?
@cmd += ' -c ' + @options[:yaml] if !@options[:yaml].nil? && @options[:command] == 'pandocomatic'
@cmd += ' --enable pandoc-verbose --log pandocomatic.log --log-level debug' if @options[:verbose] == true && @options[:command] == 'pandocomatic'
@toolPath = `which #{@options[:command]}`.chomp
@cmd = @toolPath + @cmd + ' ' + @options[:input] + ' ' unless @options[:input].nil?
end
#-------------------------------run the command
def runCommand
puts "===------ COMMAND OUTPUT: ------===" if @options[:verbose]
if File.exist?(@toolPath) && !options.dry_run
puts ":: Running: #{@cmd}\n" if @options[:verbose]
Open3.popen2e(@cmd) do |_stdin, oe, thread|
while (line = oe.gets)
puts "::: " + line.chomp if @options[:verbose]
check_engine = line.match(/pdf-engine=(\w+)/)
@latexEngine = check_engine[1] unless check_engine.nil?
end
exit_status = thread.value
puts ":: pandocomatic exit status: " + exit_status.to_s if @options[:verbose]
unless exit_status.success?
puts "\n!!!---scrivomatic::runCommand() RETURN non-zero value: #{cmd}!!!"
end
end
`open scrivomatic.log` if @options[:open_log] && isRecent('scrivomatic.log')
`open pandocomatic.log` if @options[:open_log] && isRecent('pandocomatic.log')
elsif !options.dry_run
puts "Tool doesn't exist!!!" if @options[:verbose]
puts "\n!!!---scrivomatic::runCommand() Couldn't find #{@toolPath} to run, please supply a proper path!"
elsif @options[:verbose]
puts "Dry run, nothing actually executed..."
end
end
#-------------------------------parse inputs
def parseInputs(_arg)
optparse = OptionParser.new do |opts|
opts.banner = "Scrivomatic V#{@version}\n"
opts.banner += "=======================\n"
opts.banner += "Scrivomatic is a wrapper to set up the shell environment, enforces UTF8 encoding and other settings.\n\n"
opts.banner += "Usage: scrivomatic [additional options] FILE"
opts.on('-i', '--input FILE', 'Input file') do |v|
v.gsub!(/(\A'|'\Z)/, '')
@options[:input] = v.shellescape
@fileext = Regexp.escape(File.extname(@options[:input]))
end
opts.on('-o', '--output [file]', 'Output file. Optional for pandocomatic.') do |v|
@options[:output] = v.shellescape
end
opts.on('-t', '--to [format]', 'Pandoc Format. Optional for pandocomatic.') do |v|
@options[:to] = v
end
opts.on('-y', '--yaml [file]', 'Specify which YAML file for pandocomatic.') do |v|
@options[:yaml] = v.strip.shellescape
end
opts.on('-p', '--path [dirpath]', 'Additional Path to Search for Commands.') do |v|
@options[:envpath] = v.strip.shellescape + ':' + @options[:envpath]
end
opts.on('-b', '--build', 'For Typst or LaTeX, run Typst and latexmk') do |v|
@options[:build] = v
end
opts.on('-B', '--buildclean', 'For LaTeX or Typst, run Typst and latexmk with cleanup') do |v|
@options[:build] = v
@options[:cleanup] = v
end
opts.on('-d', '--dry-run', 'Dry run.') do |v|
@options[:dry_run] = v
end
opts.on('-z', '--data-dir [file]', 'Pandoc data dir.') do |v|
@options[:data_dir] = v.strip.shellescape
end
opts.on('-v', '--[no-]verbose', 'Verbose output.') do |v|
@options[:verbose] = v
end
opts.on('-l', '--[no-]log', 'View log in Console.app.') do |v|
@options[:open_log] = v
end
opts.on('-h', '--help', 'Prints this help!') do
puts optparse
exit(0)
end
end
optparse.parse!
# Ensure an input file is provided.
return unless @options[:input].nil?
if ARGV.nil? || ARGV[0].nil?
puts optparse
abort "\n\n!!!---scrivomatic::parseInputs requires valid input file: --input"
else
v = ARGV[0].gsub(/(\A'|'\Z)/, '')
@options[:input] = v.shellescape
@fileext = Regexp.escape(File.extname(@options[:input]))
end
end
#------------------------------check if we want to run latexmk
def postBuildTeX
return unless options.build && !(options.dry_run)
texPath = options.input.gsub(/#{@fileext}$/, '.tex')
texFile = texPath.gsub(/\\+/, '')
pdfFile = texPath.gsub(/\.tex/, '.pdf')
if isRecent(texFile)
if File.exist?(pdfFile)
puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose]
File.delete(pdfFile)
end
puts "\n===------ RUN LATEXMK on #{texPath}: ------===" if @options[:verbose]
@latexEngine = 'pdf' if @latexEngine =~ /pdflatex/
xcmd = "latexmk -logfilewarnings -interaction=nonstopmode -f -pv -time -#{@latexEngine} -f #{texPath}"
puts ":: directory: #{Dir.pwd}" if @options[:verbose]
puts ":: command: #{xcmd}" if @options[:verbose]
begin
Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
while (line = oe.gets)
puts "::: " + line.chomp if @options[:verbose]
end
exit_status = wait_thr.value
puts ":: exit status: " + exit_status.to_s if @options[:verbose]
if exit_status.success? && @options[:cleanup]
logPath = File.basename(options.input, '.*') + '.log'
FileUtils.cp(logPath, 'latexlog_' + logPath) if File.file?(logPath)
`latexmk -C -quiet`
puts ":: Clean-up: used latexmk -c, but kept the latex build log as #{'latexlog_' + logPath}" if @options[:verbose]
elsif !exit_status.success?
puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
end
end
rescue StandardError => e
puts e
end
else
puts "===Scrivomatic postBuildTeX: did not find #{texPath}"
end
end
# This method performs post-build actions for Typst files.
def postBuildTypst
return unless options.build && !options.dry_run
typPath = options.input.gsub(/#{@fileext}$/, '.typst').gsub(/\\+/, '')
pdfFile = typPath.gsub(/\.typst/, '.pdf')
unless isRecent(typPath)
puts "===Scrivomatic postBuildTypst: did not find #{typPath}" if @options[:verbose]
return
end
if File.exist?(pdfFile)
puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose]
File.delete(pdfFile)
end
puts "\n===------ RUN TYPST on #{typPath}: ------===" if @options[:verbose]
xcmd = "typst compile --pdf-standard a-2b #{typPath}"
puts ":: directory: #{Dir.pwd}" if @options[:verbose]
puts ":: command: #{xcmd}" if @options[:verbose]
begin
Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
while (line = oe.gets)
puts "::: " + line.chomp if @options[:verbose]
end
exit_status = wait_thr.value
puts ":: exit status: " + exit_status.to_s if @options[:verbose]
if exit_status.success?
puts ":: Success" if @options[:verbose]
else
puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
end
end
rescue StandardError => e
puts "Error during Typst compilation: #{e.message}"
end
end
end # end Scrivomatic class
#binding.break
scriv = Scrivomatic.new
scriv.parseInputs(ARGV)
scriv.run
And while it did great at loading the correct versions…:
There was still somehow an issue with the “–enable” argument and verbose option:
… running /opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic file-entire.md
::: WARNING: Ignoring the pandoc option "--verbose" because it might interfere with the working of pandocomatic. If you want to use "--verbose" anyway, use pandocomatic's feature toggle "--enable pandoc-verbose".
::: [WARNING] YAML warning (line 1 column 1): Duplicate key: .author
::: [WARNING] YAML warning (line 1 column 1): Duplicate key: .title
::: WARNING: Ignoring the pandoc option "--verbose" because it might interfere with the working of pandocomatic. If you want to use "--verbose" anyway, use pandocomatic's feature toggle "--enable pandoc-verbose".
Frustrated, I decided to wait for your update.
Edit:
After testing the new script, it is causing Scrivener to hang.
I had to change back the shell command to /usr/bin/env ruby for it to work (otherwise I got a ruby not found error) and for now Scrivener/pandocomatic is stuck during the compile process.
Logs show the correct tool paths were loaded but something must have happened while pandocomatic was running.
This is just a warning and I think harmless, though interestingly I don’t see it (and I always keep verbose ON). You can remove the -v verbose flag to scrivomatic and also any verbose metadata in your document or pandocomatic metadata, and that warning should disappear.
The debugging should follow these steps:
Did Scrivener create the .md file in the compile folder?
Use pandoc on the command line and compile the md file directly, does it compile (use a simple format not LaTeX PDF as it has its own complex compilation path etc.)?
Use pandocomatic on the command line, does it work?
Use scrivomatic on the command line, does it work?
If it doesn’t, make a simple md document, does that work?
On the command line if a tool hangs, ctrl+c should unblock it and the error it gives may help in determining the problem.
The latest version of pandocomatic enables the generation of its own log file that can help to get context for a problem (this is enabled by scrivomatic’s -v verbose option, so you should have a pandocomatic.log file available already, when running directly add --log pandocomatic.log --log-level debug).
