Scrivomatic - Compiling issues with Scrivomatic/Pandoc (disappearing tables, unknown LaTex)

Hi, hello.
I have no idea where to ask this! I am using iandol’s Scrivomatic setup on macOS (and trying to wrangle it into my own despite a general lack of knowledge of pandoc & co).

Issue 1:
When I compile my Scrivener document into Word .docx (and Markdown as part of the process), the tables disappear. The text isnide the cells is still there, but it is separated with ‘xxx’ for the rows.
After looking everywhere in the documentation, I don’t see anything on whether one should do anything specific in Scrivener to compile tables properly in Markdown.
What am I missing?

Issue 2:
When compiling into LaTex, the log shows the following error:

::: Error producing PDF.
::: ! Undefined control sequence.
::: <recently read> \citeproc

After searching the web, I found a stackexchange post with a similar issue, but the fix in it didn’t work.
Again, any idea as to what I could be missing?

Issue 3:
I am totally at a loss as to where to start to create a table of content or append a bibliography using Scrivomatic. I am sure there are resources out there, and I would be happy to read them, but I might need a helpful hint to point me in the right direction or as to where to start.
I know iandol’s resource page hints that there is something to be done with embedding links inside Scrivener to refer to figure or elements, but I wasn’t able to quite understand how that works.

Thank you in advance for any and all help! I hope this wasn’t too confused, I’m afraid being confused myself doesn’t help with defining these issues properly. :slight_smile:

On question one: to confirm, have you ticked the option, in the General options tab on the right side of the main compile overview window, to Convert tables and lists to MultiMarkdown? Without that, the result will be much like you describe I think, with the text linearised. (Pandoc can read MultiMarkdown lists and tables, so don’t mind the specificity of that option.)

As to the rest, I am less familiar with the specifics of Scrivomatic configuration, so hopefully someone can answer those questions.

2 Likes

After a bit more looking around in menus, I found the option and it is now ticked. Thank you, it works! Issue 1 is solved. :tada:

Glad 1. got sorted thanks to @AmberV

The error seems to suggest you have Pandoc configured for references (though you say below you need a bibliography…)? \citeproc is a LaTeX command used in the newer Pandoc LaTeX templates:

This LaTeX error suggests you have triggered Pandoc for refs but you are using a custom older template that may not have this command? If you are using an old version of my custom LaTeX template you should upgrade the template (or downgrade pandoc). I keep all my Pandoc templates here (I keep the templates up-to-date with the latest Pandoc):

In general as Pandoc is updated to fix bugs and add features, templates will also change. For anyone that does use my templates (which work with my academic metadata structure), using git to keep up-to-date with the templates is the easiest thing (you can automate running git pull every week or so and git allows you to roll back and forwards in time as required)…

My custom templates mostly deal with academic metadata, so if you don’t need that then you can revert back to the standard templates by removing the template key from the pandocomatic recipe.

TOC

For a TOC you use Pandoc’s --toc toggle, which with metadata means you add toc: true or table-of-contents: true to pandocomatic.yaml or the document metadata. Here is an example for my typst recipe:

This can be added to your Scrivener metadata directly too I think to override anything in the pandocomatic recipe.

Bibliography

For a Bibliography, you enter references using the [@citekey] pandoc format in your text. Then when you compile, your pandoc settings / pandocomatic recipe needs the following ingredients:

  1. Tell pandoc to use citeproc
  2. Tell it where your bibliography file is
  3. Tell it where your CSL style file is

So for example in my pandocomatic recipe I define a recipe for references:

My bibliography file Core.json is automatically created by Bookends, and I keep it in my pandoc data dir so no need for a path here but you should add the path. The other settings are just tweaks and not necessary. By specifying bibliography and citeproc in the recipe then pandoc knows to make the bibliography for you. It normally goes at the end of the document, but pandoc has a trick to place it anywhere which the Scrivomatic template uses, let me know if you need help on that…

4 Likes

Thanks for your detailed answer!

This is for academia, and I am using Zotero and BetterBibtex, with synced JSON/BIBTEX exports in my pandoc folder.

I also have the zotero.lua filter set up to ensure live citations for docx documents. I did note that templates seem to work perfectly fine from the .yaml configuration file, but the Scrivener frontmatter override does not seem to be working as intended, when, for example, I tried to set a specific .csl configuration or just now with the toc command. Similarly, I couldn’t get the wordcount line to work.
Just for clarity (and in case my mistake was indentation), here’s what I had:

pandocomatic_:
  use-template:
    - docx-refs
    - pdf-refs
pandoc:
    table-of-contents: true

However, I tested the table-of-contents option in my pandocomatic.yaml file and it works perfectly. Brilliant and thank you so much!

For my purposes, getting it to work with Word is already great and a major win, but LaTex handles tables much more smoothly from what I’ve seen — it’s just a tad more complex to use for the novice.
I initially did clone your repo as-is but since I wanted to tweak the Word template, I ended up moving it to a subdirectory. It is recent from a few days ago, though, so I’m unsure what went wrong if this is working for you. I should note that LaTex is invoked because I am attempting to use the “pdf-refs” template!

Edit:
I figured out what was going wrong! I didn’t realise that the conversion to .docx uses the pandocomatic.yaml that is inside the output folder while the conversion to .latex and .pdf uses the one in the pandoc folder, and it was set up to a .csl format that only covers footnotes :sweat_smile:

Issues 2 and 3 are solved!

Quick question on this topic — I see that this sets up in-document links from the footnotes to the bibliography, do you know if there is a way to make these two-way links (ie. clicking on the footnote sends to the bibliography entry, clicking on that entry sends back to the footnote)?

Thank you again, for your kind answers and for this really wonderful tool.

1 Like

Right this is an indentation error, note you need two spaces BEFORE pandoc key then 4 spaces before table-of-contents as the pandoc key is supposed to “belong” to the pandocomatic key. Both of these would have worked:

table-of-contents: true
pandocomatic_:
  use-template:
    - docx-refs
    - pdf-refs

here we pass TOC to pandoc directly to pandoc, it is not sent to pandocomatic, or:

pandocomatic_:
  use-template:
    - docx-refs
    - pdf-refs
  pandoc:
    table-of-contents: true

here pandoc is the child of pandocomatic so it will be read. Many settings can go in the main metadata, but for some settings it is a bit of trial and error…

You could also give Typst a go, pandoc exports to Typst (I have a recipe in my dotpandoc) and it usually has less errors and is easier to tweak than LaTeX.

Do note that this filter bypasses Pandoc’s citeproc (I think Zotero’s citeproc-js engine does the processing), but possibly the metadata is the same (I did recently help someone on another post to get zotero.lua working, in theory you don’t need the bibliography or csl key for this filter but i did notice different behaviours with and without these keys).

git has a cool feature called branches, so you can fork a repo then make a local branch, that means you can make changes to some files and still pull the other files. I often keep the master or main branch of someone elses tool, then make my own feature branch then pull or merge into the custom branch, it means i can always keep their master up to date.

Right, this would definitely complicate things. For my workflow I do try to ensure everything is referenced only to my pandoc data dir.

There is this filter that can give mutliple refs back to each location in the text:

After trying this out, Scrivener compiler hangs. I am probably doing something wrong:

  pandoc:
    table-of-contents: true

The first one does nothing.

I have seen you note this in your workflow, but I have no idea how you get that to work — every time I compile a Scrivener document, it spews out a bunch of files including all the images in the project. How do you keep track when you have several projects?

Regarding zotero.lua, I think it does bypass citeproc but the resulting flexibility in changing out styles on the fly is a really good trade-off for me.

Thank you for the tip on the filter, I will have a look. Is citeproc also what I would need to look into if I want to have not a single bibliography at the end, but a references section after each chapter?

1 Like

Hm, i tested both in the scrivomatic test project when I replied and both work to enable the TOC. But I did use toc: true rather than table-of-contents: true, and double checking the manual, the in-document metadata suggests to use toc: Pandoc - Pandoc User’s Guide--table-of-contents is a command line and table-of-contents: is used in the defaults files, so hopefully this explains the discrepancy why it works for me but not for you…

For some sample projects I recommended a simple manual export of some pandoc[omatic] requirements into the compile folder as that simplifies trying the workflow out, but for longer term use: delete the compile folder you previously used, create a brand new empty folder (remember the name should end in -mmd), do NOT put pandocomatic.yaml or any lua filters or CSL files in that compile folder. Pandoc has its own data directory [PDD], by default this is ~/.local/share/pandoc — inside this folder keep ALL your compiler files: ~/.local/share/pandoc/csl for CSL, ~/.local/share/pandoc/filters for filters, ~/.local/share/pandoc/templates for compile templates etc. pandocomatic expects pandocomatic.yaml to be in the root of this folder, e.g. ~/.local/share/pandoc/pandocomatic.yaml.

For bibliography bib/json: I use a symlink ~/.local/share/pandoc/Core.json so I use the PDD to access my database, though in fact this is symlinked from my dropbox so I don’t keep this 18mb file itself in git but do sync it among my machines.

Now, we have single place for each file, so for example if we update apa.csl any Scrivener project will benefit from this as everything is centralised in one place. In my case, the folder is also a git repo, which in my case is stored at GitHub - iandol/dotpandoc: Pandoc Data directory, including customised filters and templates for producing multiple outputs for academic content. so I can also sync changes from one machine to all my other machines easily. By using this workflow you do not worry about different versions and pandocomatic.yaml or apa.csl etc.

Where you keep files it ultimately up to you but with everything visible in PDD as real or symlinks, there is “one ring to rule them all”…

Right, I didn’t mean to discourage you, just to mention the workflow and metadata will have some subtle differences (i.e. say you change your CSL in your pandocomatic metadata, it may show in LaTeX but not DOCX even though you use the same values etc).

See this filter:

In general, pandoc filters can do a lot of the heavy lifting. @bernardo_vasconcelos has some nice documentation for this and other bibliography tweaks on his citetools docs: Cite Tools - Multiple Bibliographies — he works with quarto so the instructions are not immediately applicable but can be modified to work with vanilla pandoc…

2 Likes

I continue to investigate the following issues:

  1. My conversions to .docx relying on pandocomatic.yaml being present in the -mmd directory.

This was, in fact, due to using the Scrivener format of your workflow document as a template for Scrivomatic, where you have these arguments in the post-processing:

 "<$inputfile>" -l -v -y "pandocomatic.yaml" >>scrivomatic.log 2>&1

Instead of these:

-i "<$inputfile>" -l -v  2>&1 >>scrivomatic.log 

Now that I have fixed this, I properly point to the root pandoc folder for every compile.

  1. Continue issues with the frontmatter metadata overrides.

Given my previous findings, I looked into the post-processing again and found that, indeed, the Script I originally had was different from the one listed here.

#!/usr/bin/env ruby
# encoding: utf-8

# scrivomatic is a wrapper script that adds tools to the path (as Scrivener
# does not use the user's path), and enables some other tweaks to optimise
# the workflow when scrivener calls pandocomatic.
# `scrivomatic --help` for details…

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

require 'open3' # ruby standard library class to handle stderr and stdout
require 'optparse' # ruby standard option parser
require 'fileutils' # ruby standard library to deal with files
require 'shellwords' # escapes strings to run in the shell

##
# main scrivomatic class
class Scrivomatic
	##
	# Sets up the path and performs other functions to improve running
	# pandocomatic from scrivener...
	attr_accessor :options
	attr_reader :version, :cmd, :runLog
	VER = '1.0.36'.freeze
	OPT = Struct.new(:input, :output, :to, :yaml, :command, :envpath, :build, :cleanup, :verbose, :dry_run, :open_log, :data_dir)
	DEFENVPATH = ENV['HOME'] + '/bin'

	#-------------------------------class constructor
	def initialize
		@options = OPT.new(nil, nil, nil, nil, 'pandocomatic', DEFENVPATH, false, false, false, false, false, nil)
		@version = VER
		@fileext = ''
		@cmd = ''
		@toolPath = ''
		@runLog = ''
		@latexEngine = 'xelatex'
	end

	#-------------------------------run all options
	def run
		preBuild # removes old tex file if we are building
		makePath # build the path
		printInfo # print some information
		buildCommand # build the pandoc[omatic] command
		runCommand # run the pandoc[omatic] command
		postBuild # run latexmk if we are building
	end

	#--------------------------------if we will run latexmk, remove old tex file before running pandoc
	def preBuild
		return unless options.build && !(options.dry_run)
		texPath = options.input.gsub(/#{@fileext}$/, '.tex')
		texFile = texPath.gsub(/\\+/, '')
		if File.exist?(texFile)
			puts "\n===------ Delete OLD TeX File: #{texFile}: ------===" if @options[:verbose] == true
			system("latexmk -quiet -c #{texFile} >/dev/null 2>&1")
			File.delete(texFile)
		end
	end

	#-------------------------------make env path
	def makePath
		home = ENV['HOME'] + '/'
		pathtest = [home+'/.rbenv/shims', 
			'/usr/local/opt/ruby/bin', '/usr/local/lib/ruby/gems/2.7.0/bin', 
			home+'Library/TinyTeX/bin/universal-darwin', '/Library/TeX/texbin', 
			'/opt/homebrew/bin', '/usr/local/bin', 
			home+'/anaconda/bin', home+'/anaconda3/bin',
			home+'/miniconda/bin', home+'/miniconda3/bin', home+'/micromamba/bin',
			home+'/.cabal/bin', home+'/.local/bin', 
			rvm_check]
		pathtest.each { |p| @options[:envpath] = @options[:envpath] + ':' + p if File.directory?(p) }
		ENV['LANG'] = 'en_GB.UTF-8' if ENV['LANG'].nil? # Just in case we have no LANG, which breaks UTF8 encoding
		@options[:envpath].gsub!(%r{(//)}, '/') # remove double slash
		@options[:envpath].gsub!(/(::)/, ':') # remove double colon
		@options[:envpath].gsub!(/:$/, '') # remove final colon
		ENV['PATH'] = @options[:envpath] + ':' + ENV['PATH']
	end # end makePath()

	#-------------------------------print initial report
	def printInfo
		return if @options[:verbose] == false
		puts "\n=== ------------------------------------------------------ ==="
		puts '=== Scrivomatic V' + @version + ' Report @ ' + Time.now.to_s + ' ==='
		puts '=== ------------------------------------------------------ ==='
		puts ' Running shell: ' + `printf $SHELL`
		puts ' Working directory: ' + `pwd`
		puts " Initiating with Ruby #{RUBY_VERSION}"
		puts '===------ Input Options: ------==='
		puts @options
		puts '===------ Final ENV PATH: ------==='
		puts ENV['PATH']
		puts '===------ TOOL PATHS: ------==='
		puts `echo "---pandoc: $(which pandoc) | V: $(pandoc -v | sed -nE '1 s/^pandoc // gp')"`
		puts `echo "---ruby: $(which ruby) | V: $(ruby -v)"`
		puts '!--ruby version incompatible with new pandocomatic, see https://github.com/iandol/scrivomatic/blob/master/Installing-Ruby.md' if `ruby -v` =~ /ruby 2\.3\.\d/
		puts `echo "---pandocomatic: $(which pandocomatic) | V: $(pandocomatic -v | sed -En '1s/^Pandocomatic version /''/p')"`
		puts '---paru library: V: ' + `ruby -e 'require "paru"; puts Paru::VERSION.join(".")'`
		puts `echo "---rbenv versions:"; [[ -x $(which rbenv) ]] && rbenv versions`
		%w[rbenv rvm gem panzer python xelatex latexmk].each do |c|
			location = `which #{c}`.chomp
			puts "---#{c}: #{location}" unless location.empty?
		end
		puts "\n … running #{@options[:command]}, please wait …\n"
	end

	#-------------------------------build the command line
	def buildCommand
		@cmd += ' --data-dir="' + @options[:data_dir] + '"' unless @options[:data_dir].nil?
		@cmd += ' --output ' + @options[:output] + ' ' unless @options[:output].nil?
		@cmd += ' --to "' + @options[:to] + '"' unless @options[:to].nil?
		@cmd += ' -c ' + @options[:yaml] if !@options[:yaml].nil? && @options[:command] == 'pandocomatic'
		@cmd += ' --debug' if @options[:verbose] == true && @options[:command] == 'pandocomatic'
		@cmd += ' ---debug "panzerlogs"' if @options[:verbose] == true && @options[:command] == 'panzer'
		@toolPath = `which #{@options[:command]}`.chomp
		@cmd = @toolPath + @cmd + ' ' + @options[:input] + ' ' unless @options[:input].nil?
	end

	#-------------------------------run the command
	def runCommand
		`open scrivomatic.log` if @options[:open_log] && File.exist?('scrivomatic.log')
		puts '===------ COMMAND OUTPUT: ------===' if @options[:verbose] == true
		if File.exist?(@toolPath) && !options.dry_run
			puts ":: Running: #{@cmd}\n" if @options[:verbose] == true
			Open3.popen2e(@cmd) do |_stdin, oe, thread|
				while (line = oe.gets)
					puts '::: ' + line.chomp if @options[:verbose] == true
					check_engine = line.match(/pdf-engine=(\w+)/)
					@latexEngine = check_engine[1] unless check_engine.nil?
				end
				exit_status = thread.value
				puts ':: exit status: ' + exit_status.to_s if @options[:verbose] == true
				unless exit_status.success?
					puts "\n!!!---scrivomatic::runCommand() RETURN non-zero value: #{cmd}!!!"
				end
			end
		elsif !options.dry_run
			puts "Tool doesn't exist!!!" if @options[:verbose] == true
			puts "\n!!!---scrivomatic::runCommand() Couldn't find #{@toolPath} to run, please supply a proper path!"
		elsif @options[:verbose] == true
			puts 'Dry run, nothing actually executed...'
		end
	end

	#-------------------------------parse inputs
	def parseInputs(_arg)
		optparse = OptionParser.new do |opts|
			opts.banner = 'Scrivomatic V' + @version + "\n"
			opts.banner += "=======================\n"
			opts.banner += "Scrivomatic is a wrapper around pandocomatic or panzer, that sets up the environment path, enforces UTF8 encoding and other settings so they can be run from any other process that may not do this (e.g. Scrivener).\n\n"
			opts.banner += 'Usage: scrivomatic [additional options] FILE'
			opts.on('-i', '--input FILE', 'Input file') do |v|
				v.gsub!(/(\A'|'\Z)/, '')
				@options[:input] = v.shellescape
				@fileext = Regexp.escape(File.extname(@options[:input]))
			end
			opts.on('-o', '--output [file]', 'Output file. Optional for pandocomatic.') do |v|
				@options[:output] = v.shellescape
			end
			opts.on('-t', '--to [format]', 'Pandoc Format. Optional for pandocomatic.') do |v|
				@options[:to] = v
			end
			opts.on('-y', '--yaml [file]', 'Specify which YAML file for pandocomatic.') do |v|
				@options[:yaml] = v.strip.shellescape
			end
			opts.on('-c', '--command [command]', 'Tool to use: [pandocomatic] | panzer') do |v|
				@options[:command] = v
			end
			opts.on('-p', '--path [dirpath]', 'Additional Path to Search for Commands.') do |v|
				@options[:envpath] = v.strip.shellescape + ':' + @options[:envpath]
			end
			opts.on('-b', '--build', 'For LaTeX output, run latexmk') do |v|
				@options[:build] = v
			end
			opts.on('-B', '--buildclean', 'For LaTeX output, run latexmk and cleanup') do |v|
				@options[:build] = v
				@options[:cleanup] = v
			end
			opts.on('-d', '--dry-run', 'Dry run.') do |v|
				@options[:dry_run] = v
			end
			opts.on('-z', '--data-dir [file]', 'Pandoc data dir.') do |v|
				@options[:data_dir] = v.strip.shellescape
			end
			opts.on('-v', '--[no-]verbose', 'Verbose output.') do |v|
				@options[:verbose] = v
			end
			opts.on('-l', '--[no-]log', 'View log in Console.app.') do |v|
				@options[:open_log] = v
			end
			opts.on('-h', '--help', 'Prints this help!') do
				puts optparse
				exit(0)
			end
		end # end OptionParser

		optparse.parse!

		# make sure we have an input file
		return unless @options[:input].nil?

		# otherwise check if we got passed the file
		if ARGV.nil? || ARGV[0].nil?
			puts optparse
			abort "\n\n!!!---scrivomatic::parseInputs requires valid input file: --input"
		else
			v = ARGV[0].gsub(/(\A'|'\Z)/, '') # scrivener sometimes passes the file wrapped in '
			@options[:input] = v.shellescape # we assume it was passed without -i flag
			@fileext = Regexp.escape(File.extname(@options[:input]))
		end
	end # end parseInputs

	#-------------------------------check for RVM
	def rvm_check
		rvm_home = ENV['HOME'] + '/.rvm'
		return '' unless File.directory?(rvm_home)
		rvm_home + '/wrappers/default'
	end

	#------------------------------check if we want to run latexmk
	def postBuild
		return unless options.build && !(options.dry_run)
		texPath = options.input.gsub(/#{@fileext}$/, '.tex')
		texFile = texPath.gsub(/\\+/, '')
		pdfFile = texPath.gsub(/\.tex/, '.pdf')
		if File.exist?(texFile)
			if File.exist?(pdfFile)
				puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose] == true
				File.delete(pdfFile)
			end
			puts "\n===------ RUN LATEXMK on #{texPath}: ------===" if @options[:verbose] == true
			@latexEngine = 'pdf' if @latexEngine =~ /pdflatex/
			xcmd = "latexmk -logfilewarnings -interaction=nonstopmode -f -pv -time -#{@latexEngine} -f #{texPath}"
			puts ":: directory: #{Dir.pwd}" if @options[:verbose] == true
			puts ":: command: #{xcmd}" if @options[:verbose] == true
			begin
				Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
					while (line = oe.gets)
						if line.chomp.to_s =~ /^(Latexmk:|Run|LaTeX|This is|===|Accumulated|Missing|! )/
							puts '::: ' + line.chomp if @options[:verbose] == true
						end
					end
					exit_status = wait_thr.value
					puts ':: exit status: ' + exit_status.to_s if @options[:verbose] == true
					if exit_status.success? && @options[:cleanup] == true
						logPath = File.basename(options.input, '.*') + '.log'
						FileUtils.cp(logPath, 'latexlog_' + logPath) if File.file?(logPath)
						`latexmk -C -quiet`
						puts ":: Clean-up: used latexmk -c, but kept the latex build log as #{'latexlog_' + logPath}" if @options[:verbose] == true
					elsif !exit_status.success?
						puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
					end
				end
			rescue StandardError => e
				puts e
			end
		else
			puts "!!!---Scrivomatic postBuild: could not find #{texPath}"
		end
	end
end #--------------- end Scrivomatic class

scriv = Scrivomatic.new
scriv.parseInputs(ARGV)
scriv.run

Having updated my script as well, I found myself troubled by the resulting log…

 … running pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /usr/local/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug Dissertation.md 
::: Problematic invocation: unknown argument '--enable'.
:: pandocomatic exit status: pid 18387 exit 243

!!!---scrivomatic::runCommand() RETURN non-zero value: /usr/local/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug document.md !!!

After looking into it, --enable pandoc-verbose is indeed a proper, documented command in pandocomatic. I updated my ruby install and my gems, to no avail.

This is rather puzzling — and I am no closer to understanding why I can use “toc: true” before “pandocomatic” in the frontmatter, but changing the csl does not work at all.
I suspect the command does not overwrite but add to existing instructions.

On a happier note, I managed to fiddle with the rest of the filters you recommended and they are fantastic! Thanks again.

Right in Workflow.scriv I set this sample up to run from the compile directory to make it easier to setup for a new user. I’ve added some more explicit instructions on what to do if you want to convert to using Pandoc data folder in the latest Workflow.scriv.zip – would this text have been helpful for you:

Right this and --log option are new. What is the full output now of your scrivomatic.log, it may be a wrong ruby is being called and thus still an old pandocomatic, scrivomatic does log which versions are being used, like:

===------ TOOL PATHS: ------===
---pandoc: /Users/ian/.pixi/bin/pandoc | V: 3.6.3
---ruby: /Users/ian/.pixi/bin/ruby | V: ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin20]
---pandocomatic: /Users/ian/.pixi/bin/pandocomatic | V: 2.0.1
---paru library: V: 1.4.1
---rbenv versions:
---pixi: /Users/ian/.pixi/bin/pixi
---gem: /Users/ian/.pixi/bin/gem
---python: /Users/ian/.pixi/bin/python
---xelatex: /Users/ian/Library/TinyTeX/bin/universal-darwin/xelatex
---latexmk: /Users/ian/Library/TinyTeX/bin/universal-darwin/latexmk

I’m not sure in your case, but there are some subtle differences in metadata types. some are booleans, some allow single values and some combine values. Also some relate to settings, and others to variables used in templates. What exactly do you want, to specify your CSL in your frontmatter?

Pandoc filters are indeed great for finessing output. With a bit of time, it is well worth learning some Lua as with a bit of Lua + a GPT you can do quite a bit…

I would perhaps tweak it to:

You must edit the Scrivener format used for compiling and change the post-processing arguments in the ‘Processing’ sub-menu: by deleting the -y "pandocomomatic.yaml pointer, pandocomatic will automatically go back to looking for this file in its default directory (instead of the compile target folder).

It just feels a little bit clearer to people who are not as tech-literate.


Got it! It’s running Ruby 2.6.10 — the macOS default and not my brew install. Now to fix this…

===------ TOOL PATHS: ------===
---pandoc: /opt/homebrew/bin/pandoc | V: 3.6.3
---ruby: /usr/bin/ruby | V: ruby 2.6.10p210 (2022-04-12 revision 67958) [universal.arm64e-darwin24]
---pandocomatic: /usr/local/bin/pandocomatic | V: 1.1.3
---paru library: V: 1.4.1
---rbenv versions:
---gem: /usr/bin/gem
---xelatex: /Library/TeX/texbin/xelatex
---latexmk: /Library/TeX/texbin/latexmk

Path to ruby should be /opt/homebrew/opt/ruby/bin/ruby instead (I’m on Apple Silicon).

Edit: Oh, this is indeed an Intel vs. Silicon issue. The paths are different.

After spending hours trying to fix this, I am somewhat at a loss.

My terminal correctly points:

❯ which ruby
/opt/homebrew/opt/ruby/bin/ruby
❯ ruby -v
ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24]
❯ which pandoc
/opt/homebrew/bin/pandoc
❯ pandoc -v
pandoc 3.6.3
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: /Users/me/.local/share/pandoc
❯ which pandocomatic
/opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic
❯ pandocomatic -v
Pandocomatic version 2.0.1

I have tried updating #!/usr/bin/env ruby to #!/opt/homebrew/opt/ruby/bin/ruby at the top of the script, and it is now correctly initating ruby (in fact I can also straight up make the script run using only the “ruby” shell command since my PATH is correctly set), but inside the script makePath needs to be updated — and probably other methods.
I don’t have the skills to do this, even with some friendly AI help.

Thank you @Melis for the syntax.

As a result, I just learnt about the “where” command in Windows (for those that need it). Same syntax it seems.

1 Like

Background knowledge

Scrivener does not use the user’s environment (what you see when you open terminal), and so by default any post-processing tool cannot use the extra paths and tools available in the terminal. Only system paths are available. This annoying limitation was why I wrote scrivomatic in the first place, to add back any possible path that may contain a tool useful to compilation. So I support a bunch of extra paths for package manager tools like brew, rbenv, pixi, TeX etc. should all be supported. But non-standard paths are not added and so…

On Apple Silicon homebrew normally installs all binaries to /opt/homebrew/bin (and on Intel to /usr/local/bin). scrivomatic cannot find your ruby because it is not in either of these. Thus it will use an old pandocomatic etc. I previously recommended using rbenv to install ruby, but I am currently using Getting Started - Pixi by prefix.dev for my packages (i.e. i havent tested homebrew’s ruby since before there were Apple Silicon macs) so I tried removing my pixi package and reinstalling ruby using homebrew, and you are right. This is probably changed behaviour, and the reason is the following (brew tells you this on installing):

ruby is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have ruby first in your PATH, run:
  echo 'export PATH="/opt/homebrew/opt/ruby/bin:$PATH"' >> ~/.zshrc

And so I have added this path to scrivomatic:

Tested and it is working over here:

=== ------------------------------------------------------ ===
=== Scrivomatic V1.0.42 Report @ 2025-02-26 09:21:18 +0800 ===
=== ------------------------------------------------------ ===
 Running shell: /bin/zsh
 Working directory: /Users/ian/Desktop/compile-mmd/workflow-mmd
 Initiating with Ruby 2.6.10
===------ Input Options: ------===
#<struct Scrivomatic::OPT input="Workflow.md", output=nil, to=nil, yaml="pandocomatic.yaml", command="pandocomatic", envpath="/Users/ian/bin:/Users/ian/.pixi/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/Users/ian/bin:/opt/homebrew/bin:/usr/local/bin:/Users/ian/Library/TinyTeX/bin/universal-darwin:/Users/ian/.local/bin", build=false, cleanup=false, verbose=true, dry_run=false, open_log=true, data_dir=nil>
===------ Final ENV PATH: ------===
/Users/ian/bin:/Users/ian/.pixi/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/Users/ian/bin:/opt/homebrew/bin:/usr/local/bin:/Users/ian/Library/TinyTeX/bin/universal-darwin:/Users/ian/.local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/Scrivener.app/Contents/Resources/MultiMarkdown/bin
===------ TOOL PATHS: ------===
---pandoc: /opt/homebrew/bin/pandoc | V: 3.6.3
---ruby: /opt/homebrew/opt/ruby/bin/ruby | V: ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24]
---pandocomatic: /opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic | V: 2.0.1
---paru library: V: 1.4.1
---rbenv versions:
---pixi: /Users/ian/.pixi/bin/pixi
---gem: /opt/homebrew/opt/ruby/bin/gem
---python: /Users/ian/.pixi/bin/python
---xelatex: /Users/ian/Library/TinyTeX/bin/universal-darwin/xelatex
---latexmk: /Users/ian/Library/TinyTeX/bin/universal-darwin/latexmk

Note: it is ok for the system ruby to run the scrivomatic script to begin with (the top of the log can show an old ruby), as scrivomatic then will add the new ruby path and pandocomatic will run with the new ruby…

Thanks, changes made.

1 Like

Thanks again for the answer!

I actually used this comprehensive guide for my install — stuck to homebrew because I use Ruby only for Scrivomatic and my coding is usually in Python.
Side note: this website is great and has a ton of tips for installing environments.


Wonderful! I’ll test it out.

After a whole day of hitting my head against a wall, I finally had what looked like a working solution too:

#!/opt/homebrew/opt/ruby/bin/ruby
# encoding: utf-8

# scrivomatic is a wrapper script that adds tools to the path (as Scrivener
# does not use the user's path), and enables some other tweaks to optimise
# the workflow when Scrivener calls pandocomatic.
# `scrivomatic --help` for details…

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

require 'open3'      # ruby standard library class to handle stderr and stdout
require 'optparse'   # ruby standard option parser
require 'fileutils'  # ruby standard library to deal with files
require 'shellwords' # escapes strings to run in the shell
#require 'debug/open_nonstop' # debugger, use binding.break to stop

##
# main scrivomatic class
class Scrivomatic
  ##
  # Sets up the path and performs other functions to improve running
  # pandocomatic from Scrivener...
  attr_accessor :options
  attr_reader :version, :cmd, :runLog
  VER = '1.0.41'.freeze
  OPT = Struct.new(:input, :output, :to, :yaml, :command, :envpath, :build, :cleanup, :verbose, :dry_run, :open_log, :data_dir)
  DEFENVPATH = ENV['HOME'] + '/bin'

  #-------------------------------class constructor
  def initialize
    # Default command is 'pandocomatic', but force the Homebrew version if present.
    default_command = 'pandocomatic'
    homebrew_pandocomatic = "/opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic"
    if File.exist?(homebrew_pandocomatic)
      default_command = homebrew_pandocomatic
    end
    @options = OPT.new(nil, nil, nil, nil, default_command, DEFENVPATH, false, false, false, false, false, nil)
    @version = VER
    @fileext = ''
    @cmd = ''
    @toolPath = ''
    @runLog = ''
    @latexEngine = 'xelatex'
  end

  #-------------------------------run all options
  def run
    preBuild    # removes old tex file if we are building
    makePath    # build the path
    printInfo   # print some information
    buildCommand  # build the pandoc[omatic] command
    runCommand    # run the pandoc[omatic] command
    postBuildTeX  # run latexmk if we are building
    postBuildTypst  # run typst if we are building
  end

  #--------------------------------checks if a file is less than 3 minutes old
  def isRecent(infile)
    File.exist?(infile) && (Time.now - File.mtime(infile)) <= 60
  end

  #--------------------------------if we will run latexmk, remove old tex file before running pandoc
  def preBuild
    return unless options.build && !(options.dry_run)
    texPath = options.input.gsub(/#{@fileext}$/, '.tex')
    texFile = texPath.gsub(/\\+/, '')
    if File.exist?(texFile)
      puts "\n===------ Delete OLD TeX File: #{texFile}: ------===" if @options[:verbose]
      system("latexmk -quiet -c #{texFile} >/dev/null 2>&1")
      File.delete(texFile)
    end
  end

  #-------------------------------make env path
  def makePath
    home = ENV['HOME'] + '/'
    # Updated order: ensure Homebrew gem bin comes before /usr/local/bin
    pathtest = [
      home + 'bin',
      '/opt/homebrew/lib/ruby/gems/3.4.0/bin',      # Homebrew Ruby gems bin (for pandocomatic v2.0.1)
      '/opt/homebrew/bin',
      '/opt/homebrew/opt/ruby/bin',                # Homebrew Ruby bin
      '/usr/local/bin',
      home + '.pixi/bin',
      home + '.rbenv/shims',
      home + '.pyenv/shims',
      home + 'Library/TinyTeX/bin/universal-darwin',
      '/Library/TeX/texbin',
      home + 'anaconda/bin',
      home + 'anaconda3/bin',
      home + 'miniconda/bin',
      home + 'miniconda3/bin',
      home + 'micromamba/bin',
      home + '.cabal/bin',
      home + '.local/bin'
    ]
    pathtest.each { |p| @options[:envpath] = @options[:envpath] + ':' + p if File.directory?(p) }
    ENV['LANG'] = 'en_GB.UTF-8' if ENV['LANG'].nil?  # Ensure LANG is set
    @options[:envpath].gsub!(%r{(//)}, '/')  # remove double slashes
    @options[:envpath].gsub!(/(::)/, ':')     # remove double colons
    @options[:envpath].gsub!(/:$/, '')          # remove trailing colon
    ENV['PATH'] = @options[:envpath] + ':' + ENV['PATH']
  end

  #-------------------------------print initial report
  def printInfo
    return if @options[:verbose] == false
    puts "\n=== ------------------------------------------------------ ==="
    puts "=== Scrivomatic V#{@version} Report @ #{Time.now} ==="
    puts "=== ------------------------------------------------------ ==="
    puts " Running shell: " + `printf $SHELL`
    puts " Working directory: " + `pwd`
    puts " Initiating with Ruby #{RUBY_VERSION}"
    puts "===------ Input Options: ------==="
    puts @options
    puts "===------ Final ENV PATH: ------==="
    puts ENV['PATH']
    puts "===------ TOOL PATHS: ------==="
    puts `echo "---pandoc: $(which pandoc) | V: $(pandoc -v | sed -nE '1 s/^pandoc // gp')"`
    puts `echo "---ruby: $(which ruby) | V: $(ruby -v)"`
    puts `echo "---pandocomatic: $(which pandocomatic) | V: $(pandocomatic -v | sed -En '1s/^Pandocomatic version /''/p')"`
    puts "---paru library: V: " + `ruby -e 'require "paru"; puts Paru::VERSION.join(".")'`
    puts `echo "---rbenv versions:"; [[ -x $(which rbenv) ]] && rbenv versions`
    %w[pixi rbenv rvm gem python xelatex latexmk].each do |c|
      location = `which #{c}`.chomp
      puts "---#{c}: #{location}" unless location.empty?
    end
    puts "\n … running #{@options[:command]}, please wait …\n"
  end

  #-------------------------------build the command line
  def buildCommand
    @cmd += ' --data-dir="' + @options[:data_dir] + '"' unless @options[:data_dir].nil?
    @cmd += ' --output ' + @options[:output] + ' ' unless @options[:output].nil?
    @cmd += ' --to "' + @options[:to] + '"' unless @options[:to].nil?
    @cmd += ' -c ' + @options[:yaml] if !@options[:yaml].nil? && @options[:command] == 'pandocomatic'
    @cmd += ' --enable pandoc-verbose --log pandocomatic.log --log-level debug' if @options[:verbose] == true && @options[:command] == 'pandocomatic'
    @toolPath = `which #{@options[:command]}`.chomp
    @cmd = @toolPath + @cmd + ' ' + @options[:input] + ' ' unless @options[:input].nil?
end

  #-------------------------------run the command
  def runCommand
    puts "===------ COMMAND OUTPUT: ------===" if @options[:verbose]
    if File.exist?(@toolPath) && !options.dry_run
      puts ":: Running: #{@cmd}\n" if @options[:verbose]
      Open3.popen2e(@cmd) do |_stdin, oe, thread|
        while (line = oe.gets)
          puts "::: " + line.chomp if @options[:verbose]
          check_engine = line.match(/pdf-engine=(\w+)/)
          @latexEngine = check_engine[1] unless check_engine.nil?
        end
        exit_status = thread.value
        puts ":: pandocomatic exit status: " + exit_status.to_s if @options[:verbose]
        unless exit_status.success?
          puts "\n!!!---scrivomatic::runCommand() RETURN non-zero value: #{cmd}!!!"
        end
      end
      `open scrivomatic.log` if @options[:open_log] && isRecent('scrivomatic.log')
      `open pandocomatic.log` if @options[:open_log] && isRecent('pandocomatic.log')
    elsif !options.dry_run
      puts "Tool doesn't exist!!!" if @options[:verbose]
      puts "\n!!!---scrivomatic::runCommand() Couldn't find #{@toolPath} to run, please supply a proper path!"
    elsif @options[:verbose]
      puts "Dry run, nothing actually executed..."
    end
  end

  #-------------------------------parse inputs
  def parseInputs(_arg)
    optparse = OptionParser.new do |opts|
      opts.banner = "Scrivomatic V#{@version}\n"
      opts.banner += "=======================\n"
      opts.banner += "Scrivomatic is a wrapper to set up the shell environment, enforces UTF8 encoding and other settings.\n\n"
      opts.banner += "Usage: scrivomatic [additional options] FILE"
      opts.on('-i', '--input FILE', 'Input file') do |v|
        v.gsub!(/(\A'|'\Z)/, '')
        @options[:input] = v.shellescape
        @fileext = Regexp.escape(File.extname(@options[:input]))
      end
      opts.on('-o', '--output [file]', 'Output file. Optional for pandocomatic.') do |v|
        @options[:output] = v.shellescape
      end
      opts.on('-t', '--to [format]', 'Pandoc Format. Optional for pandocomatic.') do |v|
        @options[:to] = v
      end
      opts.on('-y', '--yaml [file]', 'Specify which YAML file for pandocomatic.') do |v|
        @options[:yaml] = v.strip.shellescape
      end
      opts.on('-p', '--path [dirpath]', 'Additional Path to Search for Commands.') do |v|
        @options[:envpath] = v.strip.shellescape + ':' + @options[:envpath]
      end
      opts.on('-b', '--build', 'For Typst or LaTeX, run Typst and latexmk') do |v|
        @options[:build] = v
      end
      opts.on('-B', '--buildclean', 'For LaTeX or Typst, run Typst and latexmk with cleanup') do |v|
        @options[:build] = v
        @options[:cleanup] = v
      end
      opts.on('-d', '--dry-run', 'Dry run.') do |v|
        @options[:dry_run] = v
      end
      opts.on('-z', '--data-dir [file]', 'Pandoc data dir.') do |v|
        @options[:data_dir] = v.strip.shellescape
      end
      opts.on('-v', '--[no-]verbose', 'Verbose output.') do |v|
        @options[:verbose] = v
      end
      opts.on('-l', '--[no-]log', 'View log in Console.app.') do |v|
        @options[:open_log] = v
      end
      opts.on('-h', '--help', 'Prints this help!') do
        puts optparse
        exit(0)
      end
    end

    optparse.parse!

    # Ensure an input file is provided.
    return unless @options[:input].nil?

    if ARGV.nil? || ARGV[0].nil?
      puts optparse
      abort "\n\n!!!---scrivomatic::parseInputs requires valid input file: --input"
    else
      v = ARGV[0].gsub(/(\A'|'\Z)/, '')
      @options[:input] = v.shellescape
      @fileext = Regexp.escape(File.extname(@options[:input]))
    end
  end

  #------------------------------check if we want to run latexmk
  def postBuildTeX
    return unless options.build && !(options.dry_run)
    texPath = options.input.gsub(/#{@fileext}$/, '.tex')
    texFile = texPath.gsub(/\\+/, '')
    pdfFile = texPath.gsub(/\.tex/, '.pdf')
    if isRecent(texFile)
      if File.exist?(pdfFile)
        puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose]
        File.delete(pdfFile)
      end
      puts "\n===------ RUN LATEXMK on #{texPath}: ------===" if @options[:verbose]
      @latexEngine = 'pdf' if @latexEngine =~ /pdflatex/
      xcmd = "latexmk -logfilewarnings -interaction=nonstopmode -f -pv -time -#{@latexEngine} -f #{texPath}"
      puts ":: directory: #{Dir.pwd}" if @options[:verbose]
      puts ":: command: #{xcmd}" if @options[:verbose]
      begin
        Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
          while (line = oe.gets)
            puts "::: " + line.chomp if @options[:verbose]
          end
          exit_status = wait_thr.value
          puts ":: exit status: " + exit_status.to_s if @options[:verbose]
          if exit_status.success? && @options[:cleanup]
            logPath = File.basename(options.input, '.*') + '.log'
            FileUtils.cp(logPath, 'latexlog_' + logPath) if File.file?(logPath)
            `latexmk -C -quiet`
            puts ":: Clean-up: used latexmk -c, but kept the latex build log as #{'latexlog_' + logPath}" if @options[:verbose]
          elsif !exit_status.success?
            puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
          end
        end
      rescue StandardError => e
        puts e
      end
    else
      puts "===Scrivomatic postBuildTeX: did not find #{texPath}"
    end
  end

  # This method performs post-build actions for Typst files.
  def postBuildTypst
    return unless options.build && !options.dry_run

    typPath = options.input.gsub(/#{@fileext}$/, '.typst').gsub(/\\+/, '')
    pdfFile = typPath.gsub(/\.typst/, '.pdf')

    unless isRecent(typPath)
      puts "===Scrivomatic postBuildTypst: did not find #{typPath}" if @options[:verbose]
      return
    end

    if File.exist?(pdfFile)
      puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose]
      File.delete(pdfFile)
    end

    puts "\n===------ RUN TYPST on #{typPath}: ------===" if @options[:verbose]
    xcmd = "typst compile --pdf-standard a-2b #{typPath}"
    puts ":: directory: #{Dir.pwd}" if @options[:verbose]
    puts ":: command: #{xcmd}" if @options[:verbose]

    begin
      Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
        while (line = oe.gets)
          puts "::: " + line.chomp if @options[:verbose]
        end
        exit_status = wait_thr.value
        puts ":: exit status: " + exit_status.to_s if @options[:verbose]

        if exit_status.success?
          puts ":: Success" if @options[:verbose]
        else
          puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
        end
      end
    rescue StandardError => e
      puts "Error during Typst compilation: #{e.message}"
    end
  end
end  # end Scrivomatic class

#binding.break
scriv = Scrivomatic.new
scriv.parseInputs(ARGV)
scriv.run

And while it did great at loading the correct versions…:

=== ------------------------------------------------------ ===
=== Scrivomatic V1.0.41 Report @ 2025-02-25 16:09:23 -0500 ===
=== ------------------------------------------------------ ===
 Running shell: /bin/zsh
 Working directory: /Users/me/folder/folder/folder/pandocOutput-mmd
 Initiating with Ruby 3.4.2
===------ Input Options: ------===
#<struct Scrivomatic::OPT input="file-entire.md", output=nil, to=nil, yaml=nil, command="/opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic", envpath="/Users/me/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/opt/homebrew/opt/ruby/bin:/usr/local/bin:/Library/TeX/texbin", build=false, cleanup=false, verbose=true, dry_run=false, open_log=true, data_dir=nil>
===------ Final ENV PATH: ------===
/Users/me/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/opt/homebrew/opt/ruby/bin:/usr/local/bin:/Library/TeX/texbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/Scrivener.app/Contents/Resources/MultiMarkdown/bin
===------ TOOL PATHS: ------===
---pandoc: /opt/homebrew/bin/pandoc | V: 3.6.3
---ruby: /opt/homebrew/opt/ruby/bin/ruby | V: ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24]
---pandocomatic: /opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic | V: 2.0.1
---paru library: V: 1.4.1
---rbenv versions:
---gem: /opt/homebrew/opt/ruby/bin/gem
---xelatex: /Library/TeX/texbin/xelatex
---latexmk: /Library/TeX/texbin/latexmk

There was still somehow an issue with the “–enable” argument and verbose option:

 … running /opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /opt/homebrew/lib/ruby/gems/3.4.0/bin/pandocomatic file-entire.md 
::: WARNING: Ignoring the pandoc option "--verbose" because it might interfere with the working of pandocomatic. If you want to use "--verbose" anyway, use pandocomatic's feature toggle "--enable pandoc-verbose".
::: [WARNING] YAML warning (line 1 column 1): Duplicate key: .author
::: [WARNING] YAML warning (line 1 column 1): Duplicate key: .title
::: WARNING: Ignoring the pandoc option "--verbose" because it might interfere with the working of pandocomatic. If you want to use "--verbose" anyway, use pandocomatic's feature toggle "--enable pandoc-verbose".

Frustrated, I decided to wait for your update. :sweat_smile:


Edit:
After testing the new script, it is causing Scrivener to hang.
I had to change back the shell command to /usr/bin/env ruby for it to work (otherwise I got a ruby not found error) and for now Scrivener/pandocomatic is stuck during the compile process.

Logs show the correct tool paths were loaded but something must have happened while pandocomatic was running.

=== ------------------------------------------------------ ===
=== Scrivomatic V1.0.43 Report @ 2025-02-26 13:24:19 -0500 ===
=== ------------------------------------------------------ ===
 Running shell: /bin/zsh
 Working directory: /Users/me/folder/folder/folder/pandocOutput-mmd
 Initiating with Ruby 2.6.10
===------ Input Options: ------===
#<struct Scrivomatic::OPT input="file-entire.md", output=nil, to=nil, yaml=nil, command="pandocomatic", envpath="/Users/me/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/usr/local/bin:/Library/TeX/texbin", build=false, cleanup=false, verbose=true, dry_run=false, open_log=true, data_dir=nil>
===------ Final ENV PATH: ------===
/Users/me/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/usr/local/bin:/Library/TeX/texbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/Scrivener.app/Contents/Resources/MultiMarkdown/bin
===------ TOOL PATHS: ------===
---pandoc: /opt/homebrew/bin/pandoc | V: 3.6.3
---ruby: /opt/homebrew/opt/ruby/bin/ruby | V: ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24]
---pandocomatic: /opt/homebrew/opt/ruby/bin/pandocomatic | V: 2.0.1
---paru library: V: 1.4.1
---rbenv versions:
---gem: /opt/homebrew/opt/ruby/bin/gem
---xelatex: /Library/TeX/texbin/xelatex
---latexmk: /Library/TeX/texbin/latexmk

 … running pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /opt/homebrew/opt/ruby/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug file-entire.md 

1 Like

You may have some metadata issues here.

This is just a warning and I think harmless, though interestingly I don’t see it (and I always keep verbose ON). You can remove the -v verbose flag to scrivomatic and also any verbose metadata in your document or pandocomatic metadata, and that warning should disappear.

The debugging should follow these steps:

  1. Did Scrivener create the .md file in the compile folder?
  2. Use pandoc on the command line and compile the md file directly, does it compile (use a simple format not LaTeX PDF as it has its own complex compilation path etc.)?
  3. Use pandocomatic on the command line, does it work?
  4. Use scrivomatic on the command line, does it work?
  5. If it doesn’t, make a simple md document, does that work?

On the command line if a tool hangs, ctrl+c should unblock it and the error it gives may help in determining the problem.

The latest version of pandocomatic enables the generation of its own log file that can help to get context for a problem (this is enabled by scrivomatic’s -v verbose option, so you should have a pandocomatic.log file available already, when running directly add --log pandocomatic.log --log-level debug).

2 Likes

Ah, but I would like to keep the verbose option on.

Thank you, I will try to debug and get back to you.

Hello,
I am back.

My scrivomatic now simply refuses to compile to pdf or latex, claiming the templates are not found. Yet, when I try to compile to .docx, there is no issue…

Current frontmatter metadata:
# Standard Scrivener + Pandoc[omatic] Project YAML metadata
# IMPORTANT: YAML configuration files use SPACES for indentation to 
# define structure, remember to straighten any smart quotes, and don't
# capitalise metadata keys like "author" or "title", they must be 
# lowercase! Use "Show Invisibles" in Scrivener to see spaces.
title: "<$projecttitle>"
author:
  - name: X
#    affiliation: [1]
    correspondence: Y
#    equal_contributor: true
institute:
  - Z.
keywords:
  - Help
  - Please
lang: en
compiled: <$shortdate>
date: <$fulldate>
# comments: "To err is human; … err?"
# Select which templates pandocomatic will use,
# you can enter overrides to the templates here as well
pandocomatic_:
  use-template:
#    - docx-refs
    - pdf-refs

Current scrivomatic processing script:

Shell arguments: -l -v -b "<$inputfile>" >>scrivomatic.log 2>&1

#!/usr/bin/env ruby
# encoding: utf-8

# scrivomatic is a wrapper script that adds tools to the path (as Scrivener
# does not use the user's path), and enables some other tweaks to optimise
# the workflow when Scrivener calls Pandocomatic.
# `scrivomatic --help` for details…

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

require 'open3' # ruby standard library class to handle stderr and stdout
require 'optparse' # ruby standard option parser
require 'fileutils' # ruby standard library to deal with files
require 'shellwords' # escapes strings to run in the shell
#require 'debug/open_nonstop' # debugger, use binding.break to stop

##
# main scrivomatic class
class Scrivomatic
	##
	# Sets up the path and performs other functions to improve running
	# pandocomatic from scrivener...
	attr_accessor :options
	attr_reader :version, :cmd, :runLog
	VER = '1.0.42'.freeze
	OPT = Struct.new(:input, :output, :to, :yaml, :command, :envpath, :build, :cleanup, :verbose, :dry_run, :open_log, :data_dir)
	DEFENVPATH = ENV['HOME'] + '/bin'

	#-------------------------------class constructor
	def initialize
		@options = OPT.new(nil, nil, nil, nil, 'pandocomatic', DEFENVPATH, false, false, false, false, false, nil)
		@version = VER
		@fileext = ''
		@cmd = ''
		@toolPath = ''
		@runLog = ''
		@latexEngine = 'xelatex'
	end

	#-------------------------------run all options
	def run
		preBuild # removes old tex file if user passed -b 
		makePath # build the path to find our tools
		printInfo # print some information
		buildCommand # build the pandocomatic command
		runCommand # run the pandocomatic command
		postBuildTeX # run latexmk if user passed -b 
		postBuildTypst # run typst if user passed -b 
	end

	#--------------------------------checks if a file is less than 1.5 minutes old
	def isRecent(infile)
		File.exist?(infile) && (Time.now - File.mtime(infile)) <= 90
	end

	#--------------------------------if we will run latexmk, remove old tex file before running pandoc
	def preBuild
		return unless options.build && !(options.dry_run)
		texPath = options.input.gsub(/#{@fileext}$/, '.tex')
		texFile = texPath.gsub(/\\+/, '')
		if File.exist?(texFile)
			puts "\n===------ Delete OLD TeX File: #{texFile}: ------===" if @options[:verbose] == true
			system("latexmk -quiet -c #{texFile} >/dev/null 2>&1")
			File.delete(texFile)
		end
	end

	#-------------------------------make env path
	def makePath
		home = ENV['HOME'] + '/'
		pathtest = [home+'.pixi/bin', home+'.rbenv/shims', home+'.pyenv/shims',
			'/opt/homebrew/opt/ruby/bin', '/opt/homebrew/lib/ruby/gems/3.4.0/bin', # note homebrew installs ruby as keg only!
			'/usr/local/opt/ruby/bin', '/usr/local/lib/ruby/gems/3.4.0/bin', 
			home+'bin', '/opt/homebrew/bin', '/usr/local/bin', 
			home+'Library/TinyTeX/bin/universal-darwin', '/Library/TeX/texbin', 
			home+'anaconda/bin', home+'anaconda3/bin',
			home+'miniconda/bin', home+'miniconda3/bin', home+'micromamba/bin',
			home+'.cabal/bin', home+'.local/bin']
		pathtest.each { |p| @options[:envpath] = @options[:envpath] + ':' + p if File.directory?(p) }
		ENV['LANG'] = 'en_GB.UTF-8' if ENV['LANG'].nil? # Just in case we have no LANG, which breaks UTF8 encoding
		@options[:envpath].gsub!(%r{(//)}, '/') # remove double slash
		@options[:envpath].gsub!(/(::)/, ':') # remove double colon
		@options[:envpath].gsub!(/:$/, '') # remove final colon
		ENV['PATH'] = @options[:envpath] + ':' + ENV['PATH']
	end # end makePath()

	#-------------------------------print initial report
	def printInfo
		return if @options[:verbose] == false
		puts "\n=== ------------------------------------------------------ ==="
		puts '=== Scrivomatic V' + @version + ' Report @ ' + Time.now.to_s + ' ==='
		puts '=== ------------------------------------------------------ ==='
		puts ' Running shell: ' + `printf $SHELL`
		puts ' Working directory: ' + `pwd`
		puts " Initiating with Ruby #{RUBY_VERSION}"
		puts '===------ Input Options: ------==='
		puts @options
		puts '===------ Final ENV PATH: ------==='
		puts ENV['PATH']
		puts '===------ TOOL PATHS: ------==='
		puts `echo "---pandoc: $(which pandoc) | V: $(pandoc -v | sed -nE '1 s/^pandoc // gp')"`
		puts `echo "---ruby: $(which ruby) | V: $(ruby -v)"`
		puts '!--ruby version incompatible with new pandocomatic, see https://github.com/iandol/scrivomatic/blob/master/Installing-Ruby.md' if `ruby -v` =~ /ruby 2\.3\.\d/
		puts `echo "---pandocomatic: $(which pandocomatic) | V: $(pandocomatic -v | sed -En '1s/^Pandocomatic version /''/p')"`
		puts '---paru library: V: ' + `ruby -e 'require "paru"; puts Paru::VERSION.join(".")'`
		puts `echo "---rbenv versions:"; [[ -x $(which rbenv) ]] && rbenv versions`
		%w[pixi rbenv rvm gem python xelatex latexmk].each do |c|
			location = `which #{c}`.chomp
			puts "---#{c}: #{location}" unless location.empty?
		end
		puts "\n … running #{@options[:command]}, please wait …\n"
	end

	#-------------------------------build the command line
	def buildCommand
		@cmd += ' --data-dir="' + @options[:data_dir] + '"' unless @options[:data_dir].nil?
		@cmd += ' --output ' + @options[:output] + ' ' unless @options[:output].nil?
		@cmd += ' --to "' + @options[:to] + '"' unless @options[:to].nil?
		@cmd += ' -c ' + @options[:yaml] if !@options[:yaml].nil? && @options[:command] == 'pandocomatic'
		@cmd += ' --enable pandoc-verbose --log pandocomatic.log --log-level debug' if @options[:verbose] == true && @options[:command] == 'pandocomatic'
		@toolPath = `which #{@options[:command]}`.chomp
		@cmd = @toolPath + @cmd + ' ' + @options[:input] + ' ' unless @options[:input].nil?
	end

	#-------------------------------run the command
	def runCommand
		puts '===------ COMMAND OUTPUT: ------===' if @options[:verbose] == true
		if File.exist?(@toolPath) && !options.dry_run
			puts ":: Running: #{@cmd}\n" if @options[:verbose] == true
			Open3.popen2e(@cmd) do |_stdin, oe, thread|
				while (line = oe.gets)
					puts '::: ' + line.chomp if @options[:verbose] == true
					check_engine = line.match(/pdf-engine=(\w+)/)
					@latexEngine = check_engine[1] unless check_engine.nil?
				end
				exit_status = thread.value
				puts ':: pandocomatic exit status: ' + exit_status.to_s if @options[:verbose] == true
				unless exit_status.success?
					puts "\n!!!---scrivomatic::runCommand() RETURN non-zero value: #{cmd}!!!"
				end
			end
			`open scrivomatic.log` if @options[:open_log] && isRecent('scrivomatic.log')
			`open pandocomatic.log` if @options[:open_log] && isRecent('pandocomatic.log')
		elsif !options.dry_run
			puts "Tool doesn't exist!!!" if @options[:verbose] == true
			puts "\n!!!---scrivomatic::runCommand() Couldn't find #{@toolPath} to run, please supply a proper path!"
		elsif @options[:verbose] == true
			puts 'Dry run, nothing actually executed...'
		end
	end

	#-------------------------------parse inputs
	def parseInputs(_arg)
		optparse = OptionParser.new do |opts|
			opts.banner = 'Scrivomatic V' + @version + "\n"
			opts.banner += "=======================\n"
			opts.banner += "Scrivomatic is a wrapper to set up the shell environment, enforces UTF8 encoding and other settings.\n\n"
			opts.banner += 'Usage: scrivomatic [additional options] FILE'
			opts.on('-i', '--input FILE', 'Input file') do |v|
				v.gsub!(/(\A'|'\Z)/, '')
				@options[:input] = v.shellescape
				@fileext = Regexp.escape(File.extname(@options[:input]))
			end
			opts.on('-o', '--output [file]', 'Output file. Optional for pandocomatic.') do |v|
				@options[:output] = v.shellescape
			end
			opts.on('-t', '--to [format]', 'Pandoc Format. Optional for pandocomatic.') do |v|
				@options[:to] = v
			end
			opts.on('-y', '--yaml [file]', 'Specify which YAML file for pandocomatic.') do |v|
				@options[:yaml] = v.strip.shellescape
			end
			opts.on('-p', '--path [dirpath]', 'Additional Path to Search for Commands.') do |v|
				@options[:envpath] = v.strip.shellescape + ':' + @options[:envpath]
			end
			opts.on('-b', '--build', 'For Typst or LaTeX, run Typst and latexmk') do |v|
				@options[:build] = v
			end
			opts.on('-B', '--buildclean', 'For LaTeX or Typst, run Typst and latexmk with cleanup') do |v|
				@options[:build] = v
				@options[:cleanup] = v
			end
			opts.on('-d', '--dry-run', 'Dry run.') do |v|
				@options[:dry_run] = v
			end
			opts.on('-z', '--data-dir [file]', 'Pandoc data dir.') do |v|
				@options[:data_dir] = v.strip.shellescape
			end
			opts.on('-v', '--[no-]verbose', 'Verbose output.') do |v|
				@options[:verbose] = v
			end
			opts.on('-l', '--[no-]log', 'View log in Console.app.') do |v|
				@options[:open_log] = v
			end
			opts.on('-h', '--help', 'Prints this help!') do
				puts optparse
				exit(0)
			end
		end # end OptionParser

		optparse.parse!

		# make sure we have an input file
		return unless @options[:input].nil?

		# otherwise check if we got passed the file
		if ARGV.nil? || ARGV[0].nil?
			puts optparse
			abort "\n\n!!!---scrivomatic::parseInputs requires valid input file: --input"
		else
			v = ARGV[0].gsub(/(\A'|'\Z)/, '') # scrivener sometimes passes the file wrapped in '
			@options[:input] = v.shellescape # we assume it was passed without -i flag
			@fileext = Regexp.escape(File.extname(@options[:input]))
		end
	end # end parseInputs

	#------------------------------check if we want to run latexmk
	def postBuildTeX
		return unless options.build && !(options.dry_run)
		texPath = options.input.gsub(/#{@fileext}$/, '.tex')
		texFile = texPath.gsub(/\\+/, '')
		pdfFile = texPath.gsub(/\.tex/, '.pdf')
		if isRecent(texFile)
			if File.exist?(pdfFile)
				puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose] == true
				File.delete(pdfFile)
			end
			puts "\n===------ RUN LATEXMK on #{texPath}: ------===" if @options[:verbose] == true
			@latexEngine = 'pdf' if @latexEngine =~ /pdflatex/
			xcmd = "latexmk -logfilewarnings -interaction=nonstopmode -f -pv -time -#{@latexEngine} -f #{texPath}"
			puts ":: directory: #{Dir.pwd}" if @options[:verbose] == true
			puts ":: command: #{xcmd}" if @options[:verbose] == true
			begin
				Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
					while (line = oe.gets)
						if line.chomp.to_s =~ /^(Latexmk:|Run|LaTeX|This is|===|Accumulated|Missing|! )/
							puts '::: ' + line.chomp if @options[:verbose] == true
						end
					end
					exit_status = wait_thr.value
					puts ':: exit status: ' + exit_status.to_s if @options[:verbose] == true
					if exit_status.success? && @options[:cleanup] == true
						logPath = File.basename(options.input, '.*') + '.log'
						FileUtils.cp(logPath, 'latexlog_' + logPath) if File.file?(logPath)
						`latexmk -C -quiet`
						puts ":: Clean-up: used latexmk -c, but kept the latex build log as #{'latexlog_' + logPath}" if @options[:verbose] == true
					elsif !exit_status.success?
						puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
					end
				end
			rescue StandardError => e
				puts e
			end
		else
			puts "===Scrivomatic postBuildTeX: did not find #{texPath}"
		end
	end

	# This method performs post-build actions for Typst files.
	# It checks if the build option is enabled and not in dry-run mode.
	# It converts the input file path to a Typst file path and checks if the Typst file is recent.
	# If the Typst file is recent, it removes any existing PDF file and runs the Typst compile command.
	# The method logs various actions and outputs if the verbose option is enabled.
	# If the Typst compile command is successful and the cleanup option is enabled, it logs success.
	# If the Typst compile command fails, it logs an error message.
	# If the Typst file is not found, it logs a message indicating the file was not found.
	def postBuildTypst
		return unless options.build && !options.dry_run

		typPath = options.input.gsub(/#{@fileext}$/, '.typst').gsub(/\\+/, '')
		pdfFile = typPath.gsub(/\.typst/, '.pdf')

		unless isRecent(typPath)
			puts "===Scrivomatic postBuildTypst: did not find #{typPath}" if @options[:verbose]
			return
		end

		if File.exist?(pdfFile)
			puts "\n===------ Remove old #{pdfFile} ------===" if @options[:verbose]
			File.delete(pdfFile)
		end

		puts "\n===------ RUN TYPST on #{typPath}: ------===" if @options[:verbose]
		xcmd = "typst compile --pdf-standard a-2b #{typPath}"
		puts ":: directory: #{Dir.pwd}" if @options[:verbose]
		puts ":: command: #{xcmd}" if @options[:verbose]

		begin
			Open3.popen2e(xcmd) do |_stdin, oe, wait_thr|
				while (line = oe.gets)
					puts '::: ' + line.chomp if @options[:verbose]
				end
				exit_status = wait_thr.value
				puts ':: exit status: ' + exit_status.to_s if @options[:verbose]

				if exit_status.success?
					puts ":: Success" if @options[:verbose]
				else
					puts "!!!---Scrivomatic: errors on build #{xcmd}, check logs!!!"
				end
			end
		rescue StandardError => e
			puts "Error during Typst compilation: #{e.message}"
		end
	end

end #--------------- end Scrivomatic class
##########################################

##########################################RUN Code
#binding.break
scriv = Scrivomatic.new
scriv.parseInputs(ARGV)
scriv.run


Compiling for pdf-refs gives me the following logs…

Scrivomatic output log for pdf:
=== ------------------------------------------------------ ===
=== Scrivomatic V1.0.42 Report @ 2025-03-25 21:05:42 -0400 ===
=== ------------------------------------------------------ ===
 Running shell: /bin/zsh
 Working directory: /Users/X/Dropbox/X/X/pandoc-mmd
 Initiating with Ruby 2.6.10
===------ Input Options: ------===
#<struct Scrivomatic::OPT input="X.md", output=nil, to=nil, yaml=nil, command="pandocomatic", envpath="/Users/X/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/usr/local/bin:/Library/TeX/texbin", build=true, cleanup=false, verbose=true, dry_run=false, open_log=true, data_dir=nil>
===------ Final ENV PATH: ------===
/Users/X/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/usr/local/bin:/Library/TeX/texbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/Scrivener.app/Contents/Resources/MultiMarkdown/bin
===------ TOOL PATHS: ------===
---pandoc: /opt/homebrew/bin/pandoc | V: 3.6.3
---ruby: /opt/homebrew/opt/ruby/bin/ruby | V: ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24]
---pandocomatic: /opt/homebrew/opt/ruby/bin/pandocomatic | V: 2.0.1
---paru library: V: 1.4.1
---rbenv versions:
---gem: /opt/homebrew/opt/ruby/bin/gem
---xelatex: /Library/TeX/texbin/xelatex
---latexmk: /Library/TeX/texbin/latexmk

 … running pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /opt/homebrew/opt/ruby/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug X.md 
::: No such template: 'pdf-refs'.
:: pandocomatic exit status: pid 62293 exit 243

!!!---scrivomatic::runCommand() RETURN non-zero value: /opt/homebrew/opt/ruby/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug X.md !!!
===Scrivomatic postBuildTeX: did not find X.tex
===Scrivomatic postBuildTypst: did not find X.typst
Pandocomatic output log for pdf:
2025-03-25 21:14:36 INFO : ------------ START ---------------
2025-03-25 21:14:36 INFO : Running /opt/homebrew/opt/ruby/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug X.md
2025-03-25 21:14:36 DEBUG: Validating command-line arguments:
2025-03-25 21:14:36 DEBUG: ✓  Option '--input' not used:  treat all arguments after last option as input files or directories.
2025-03-25 21:14:36 DEBUG: ✓  Convert single input file or directory.
2025-03-25 21:14:36 DEBUG: ✓  Input files and directories exist.
2025-03-25 21:14:36 INFO : -  Checking feature toggles:
2025-03-25 21:14:36 INFO :    ✓ Enabling feature 'pandoc_verbose'.
2025-03-25 21:14:36 DEBUG: Start conversion:
2025-03-25 21:14:36 ERROR: No such template: 'pdf-refs'.
2025-03-25 21:14:36 INFO : ------------  END  ---------------

However, if I instead comment out the pdf-refs and compile for docx-refs…

Scrivomatic output log for docx:
=== ------------------------------------------------------ ===
=== Scrivomatic V1.0.42 Report @ 2025-03-25 21:17:26 -0400 ===
=== ------------------------------------------------------ ===
 Running shell: /bin/zsh
 Working directory: /Users/X/Dropbox/X/X/Xpandoc-mmd
 Initiating with Ruby 2.6.10
===------ Input Options: ------===
#<struct Scrivomatic::OPT input=X.md", output=nil, to=nil, yaml=nil, command="pandocomatic", envpath="/Users/X/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/usr/local/bin:/Library/TeX/texbin", build=true, cleanup=false, verbose=true, dry_run=false, open_log=true, data_dir=nil>
===------ Final ENV PATH: ------===
/Users/X/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/3.4.0/bin:/opt/homebrew/bin:/usr/local/bin:/Library/TeX/texbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Applications/Scrivener.app/Contents/Resources/MultiMarkdown/bin
===------ TOOL PATHS: ------===
---pandoc: /opt/homebrew/bin/pandoc | V: 3.6.3
---ruby: /opt/homebrew/opt/ruby/bin/ruby | V: ruby 3.4.2 (2025-02-15 revision d2930f8e7a) +PRISM [arm64-darwin24]
---pandocomatic: /opt/homebrew/opt/ruby/bin/pandocomatic | V: 2.0.1
---paru library: V: 1.4.1
---rbenv versions:
---gem: /opt/homebrew/opt/ruby/bin/gem
---xelatex: /Library/TeX/texbin/xelatex
---latexmk: /Library/TeX/texbin/latexmk

 … running pandocomatic, please wait …
===------ COMMAND OUTPUT: ------===
:: Running: /opt/homebrew/opt/ruby/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug X.md 
::: [WARNING] YAML warning (line 1 column 1): Duplicate key: .author
::: [WARNING] YAML warning (line 1 column 1): Duplicate key: .title
::: [INFO] Running filter citeproc
::: [INFO] Loaded /Users/X/.local/share/pandoc/csl/apa.csl from /Users/X/.local/share/pandoc/csl/apa.csl
::: [INFO] Loaded /Users/X/.local/share/pandoc/cite-abbr.json from /Users/X/.local/share/pandoc/cite-abbr.json
::: [INFO] Loaded /Users/X/.local/share/pandoc/Core.json from /Users/X/.local/share/pandoc/Core.json
::: [WARNING] Citeproc: citation RegulationEU20232023 not found
::: [INFO] Completed filter citeproc in 147 ms
::: [INFO] Running filter /Users/X/.local/share/pandoc/filters/prependAll.rb
::: [INFO] Completed filter /Users/X/.local/share/pandoc/filters/prependAll.rb in 103 ms
::: [INFO] Running filter /Users/X/.local/share/pandoc/filters/simplifyMetadata.rb
::: [INFO] Completed filter /Users/X/.local/share/pandoc/filters/simplifyMetadata.rb in 36 ms
::: [INFO] Running filter /Users/X/.local/share/pandoc/filters/convertIndex.lua
::: [INFO] Completed filter /Users/X/.local/share/pandoc/filters/convertIndex.lua in 43 ms
::: [INFO] Running filter /Users/X/.local/share/pandoc/filters/pagebreak.lua
::: [INFO] Completed filter /Users/X/.local/share/pandoc/filters/pagebreak.lua in 7 ms
::: [INFO] Running filter /Users/X/.local/share/pandoc/filters/abstract-section.lua
::: [INFO] Completed filter /Users/X/.local/share/pandoc/filters/abstract-section.lua in 12 ms
::: [INFO] Running filter /Users/X/.local/share/pandoc/filters/pretty-urls.lua
::: [INFO] Completed filter /Users/X/.local/share/pandoc/filters/pretty-urls.lua in 62 ms
::: [INFO] Loaded PastedGraphic6.png from PastedGraphic6.png
::: [INFO] Loaded PastedGraphic1.png from PastedGraphic1.png
::: [INFO] Loaded PastedGraphic8.png from PastedGraphic8.png
::: [INFO] Loaded PastedGraphic7.png from PastedGraphic7.png
::: [INFO] Loaded timeline.png from timeline.png
::: [INFO] Loaded General.png from General.png
::: [INFO] Loaded /Users/X/.local/share/pandoc/templates/custom.docx from /Users/X/.local/share/pandoc/templates/custom.docx
::: [INFO] Not rendering RawInline (Format "tex") "\\n"
:: pandocomatic exit status: pid 64907 exit 0
===Scrivomatic postBuildTeX: did not find X.tex
===Scrivomatic postBuildTypst: did not find X.typst
Pandocomatic output log for docx:
2025-03-25 21:17:27 INFO : ------------ START ---------------
2025-03-25 21:17:27 INFO : Running /opt/homebrew/opt/ruby/bin/pandocomatic --enable pandoc-verbose --log pandocomatic.log --log-level debug X.md
2025-03-25 21:17:27 DEBUG: Validating command-line arguments:
2025-03-25 21:17:27 DEBUG: ✓  Option '--input' not used:  treat all arguments after last option as input files or directories.
2025-03-25 21:17:27 DEBUG: ✓  Convert single input file or directory.
2025-03-25 21:17:27 DEBUG: ✓  Input files and directories exist.
2025-03-25 21:17:27 INFO : -  Checking feature toggles:
2025-03-25 21:17:27 INFO :    ✓ Enabling feature 'pandoc_verbose'.
2025-03-25 21:17:27 DEBUG: Start conversion:
2025-03-25 21:17:27 INFO : (2) + converting /X/X/X/X/X/Xpandoc-mmd/X.md 1 time:
2025-03-25 21:17:27 INFO : (1)   - convert X.md -> X.docx
2025-03-25 21:17:27 DEBUG:   #  Using template 'docx-refs'.
2025-03-25 21:17:27 DEBUG:   #  Selected template mixed with internal template and pandocomatic metadata gives final template:
                                  extends:
                                  - docx
                                  - refs
                                  glob: []
                                  setup: []
                                  preprocessors: []
                                  metadata:
                                    notes-after-punctuation: false
                                    link-citations: true
                                    link-bibliography: true
                                    csl-hanging-indent: true
                                  pandoc:
                                    verbose: true
                                    citeproc: true
                                    bibliography: Core.json
                                    csl: csl/apa.csl
                                    citation-abbreviations: cite-abbr.json
                                    reference-links: true
                                    from: markdown
                                    to: docx
                                    standalone: true
                                    filter:
                                    - filters/prependAll.rb
                                    - filters/simplifyMetadata.rb
                                    lua-filter:
                                    - filters/convertIndex.lua
                                    - filters/pagebreak.lua
                                    - filters/abstract-section.lua
                                    - filters/pretty-urls.lua
                                    reference-doc: templates/custom.docx
                                  postprocessors: []
                                  cleanup: []
2025-03-25 21:17:27 DEBUG:   →  Reading source file: '/X/X/X/X/X/Xpandoc-mmd/X.md'
2025-03-25 21:17:27 DEBUG:      | FileInfoPreprocessor. Adding file information to metadata:
                                     pandocomatic-fileinfo:
                                       from: markdown
                                       to: docx
                                       path: '/Users/X/X/X/X/X/Xpandoc-mmd/X.md'
                                       src_path: '/Users/X/X/X/X/X/Xpandoc-mmd/X.md'
                                       created: 2025-03-25
                                       modified: 2025-03-25
2025-03-25 21:17:27 DEBUG:      | MetadataPreprocessor. Adding mined YAML blocks to metadata:
                                     notes-after-punctuation: false
                                     link-citations: true
                                     link-bibliography: true
                                     csl-hanging-indent: true
2025-03-25 21:17:27 DEBUG:      #  Changing directory to '/Users/X/X/X/X/Xpandoc-mmd'
2025-03-25 21:17:27 DEBUG:      #  Running pandoc
2025-03-25 21:17:27 DEBUG:      |  pandoc	--verbose \
                                           	--citeproc \
                                           	--bibliography=/Users/X/.local/share/pandoc/Core.json \
                                           	--csl=/Users/X/.local/share/pandoc/csl/apa.csl \
                                           	--citation-abbreviations=/Users/X/.local/share/pandoc/cite-abbr.json \
                                           	--reference-links \
                                           	--from=markdown \
                                           	--to=docx \
                                           	--standalone \
                                           	--filter=/Users/X/.local/share/pandoc/filters/prependAll.rb \
                                           	--filter=/Users/X/.local/share/pandoc/filters/simplifyMetadata.rb \
                                           	--lua-filter=/Users/X/.local/share/pandoc/filters/convertIndex.lua \
                                           	--lua-filter=/Users/X/.local/share/pandoc/filters/pagebreak.lua \
                                           	--lua-filter=/Users/X/.local/share/pandoc/filters/abstract-section.lua \
                                           	--lua-filter=/Users/X/.local/share/pandoc/filters/pretty-urls.lua \
                                           	--reference-doc=/Users/X/.local/share/pandoc/templates/custom.docx \
                                           	--output=//Users/X/X/X/X/X/Xpandoc-mmd/X.docx
2025-03-25 21:17:29 INFO : ------------  END  ---------------

I have no idea what could be going wrong.
I deinstalled my BasicTex and installed the full MacTex (I was getting an error about a unicode character previously), to no avail.
I have not changed my pandocomatic.yaml file at all.

While I do not know how to test templates from a yaml config using only pandoc, I was able to run simple pandoc conversions to latex from the command tool.

Any help is welcome!

No such template: 'pdf-refs' means the pandocomatic.yaml does not contain this recipe. In my pandocomatic.yaml this is on line 369:

Double check there isn’t an error in your pandocomatic.yaml in your pandoc data directory (~/.local/share/pandoc) — you can zip it and post it here or paste it as a gist or other pastebin so I can check. YAML can be fussy in that indentation (white space) is important to define structure etc.

Did you try the latex-refs recipe? This generates just the LaTeX before it is run through the latex engine.

BasicTeX should not have any unicode issues as long a xelatex or lualatex engine is working (they should be), I use the similar TinyTeX without issue. I do install some extra packages[1] to ensure Pandoc can compile OK (as I don’t want to install the whole 4GB caboodle)…

The error is that pandocomatic can’t find the recipe. But pandocomatic is just a recipe manager, so you can take the settings from the latex recipe and refs recipe and these are what gets sent to Pandoc. As a simple example here is my indexing recipe: dotpandoc/pandocomatic.yaml at master · iandol/dotpandoc · GitHub — as a pandoc command that would be pandoc -L convertIndex.lua -M doindex=true. I think your pandoc and latex install are probably just fine, lets fix the yaml issue first.


[1]

tlmgr update --self
tlmgr update --all
tlmgr install lualatex-math luatexja abstract \
	latexmk csquotes pagecolor relsize mdframed needspace sectsty \
	titling titlesec preprint layouts glossaries tabulary soul xargs todonotes \
	mfirstuc xfor wallpaper datatool substr adjustbox collectbox \
	sttools wrapfig footnotebackref fvextra zref \
	libertinus libertinus-fonts libertinus-otf threeparttable \
	elsarticle algorithms algorithmicx siunitx bbding biblatex biber \
	stackengine xltabular booktabs orcidlink \
	ltablex cleveref makecell threeparttablex tabu multirow \
	changepage marginnote sidenotes environ fontawesome5 tcolorbox framed pdfcol \
	tikzfill luacolor lua-ul xpatch selnolig \
	lua-visual-debug lipsum svg newfile
3 Likes

Thank you, that’s the odd thing – I’m using your pandocomatic.yaml only slightly edited for .docx compiles.
pandocomatic.yaml.zip (7.5 KB)


I did, same result: the template could not be found.


I also thought it strange, re- latex. I used your proposed install method, yet ran into some issues.
First of all, I had difficulties grabbing all the packages and had to manually route through different repositories to grab almost all of them (one remained elusive).
Secondly, when I tested the pandoc .tex output just now after the full reinstall using TexShop, I encountered the same UNICODE error I used to have through Scrivomatic when I tried to typeset using LaTex but no error using XeLaTex, which is what the Pandocomatic compile is supposed to be using in the first place according to the logs.