Scrivener → Squarto

[cavalierex]

Screenshot update | 2025-01-20

For anybody interested, here is how Squarto is set up to facilitate creation of a Quarto Book project. All writing takes place in Scrivener, using all the normal tools (like the binder, styles, etc.). The Squarto template for Scrivener gives a good starting point for structure of the binder.

Zooming in on the binder, there are sections for Quarto files, the book itself (including front matter, parts/chapters, and appendices), and figures.

Building the Quarto Book is initiated through Scrivener’s Compile command. The Squarto template for Scrivener has built-in section layouts to auto-assign the structure of the binder.

The Squarto compile format includes a processing step that will pass important arguments, such as where the COMPILED_BY_SCRIVENER.md output file is located, what you want to name the root directory for the Quarto book (in this case, squarto_output), and where your references .bib file is located (which will be included in the project as a copy or as an alias).

Of course, most of the processing itself is done via the embedded bash script.

There are some pretty cool features in the bash script, such as redirecting STDOUT and STDERR to a log file, pasting the path to the .scriv project file (since Scrivener doesn’t have a native way to pass this path to the Compile step, as detailed in this thread), prepending key parameters into the COMPILED_BY_SCRIVENER.md document for later reference, and finally, executing squarto build at the command line. The script is executed in a virtual environment (in my case, using conda).

The post-processed COMPILED_BY_SCRIVENER.md looks something like this:

The top section contains prepended metadata that will be used by Squarto later. Each item from the binder is separated by a delimiter (that Scrivener adds via the Section Type and Section Layout functionality). Squarto uses this later to parse the sections of the compiled text and save the content into separate files, ordered hierarchically within the Quarto Book project, as was visually laid out within the binder.

The Quarto Book project is created on the filesystem as a hierarchy of folders and files. Text documents within the Scrivener ‘DraftFolder’ were parsed from the COMPILED_BY_SCRIVENER.md file and then stored in the appropriate location. (Quarto uses the .qmd extension for Quarto Markdown format.) Other files (e.g., _quarto.yml, fonts and scripts, and images) are copied from the deep internal recesses of the .scriv project package into the proper location. You might notice that the filenames for the text documents are “slugified” versions of the titles in the binder. Emojis are removed lol.

A Quarto Book requires a properly formatted _quarto.yml configuration file to tell Quarto what to do. This file is dynamically created by Squarto to show the correct paths to all the parts, chapters, and appendices. You don’t have to edit this portion of _quarto.yml yourself – all you do is rearrange/rename things in the Binder, and the YAML is auto-generated at Compile/post-process time!

With the Quarto Book project properly saved in all its bits-n-pieces, it is now ready for rendering using quarto render into whatever format you want (e.g., PDF), or quarto preview for a preview in the browser.

Of course, there are tons more details about the internals of the project. I continue to refactor it to make it more robust against edge cases. And I continue to work on the Squarto Scrivener project template itself, in which all the styles and compile settings are defined. Squarto is quite modular and extensible now, so I can continue to develop pipeline steps that are relatively easy to add – for example, pre-processing steps that I want to run on the COMPILED_BY_SCRIVENER.md file before parsing and saving the content to files. For example, this is how I correct image tag attributes.

I’m now at the point where I can use Squarto in “real life” (with “real projects”); that said, it’s still beta software, and I continue to improve it. Let me know what questions/suggestions you have about it!

Many thanks to @nontroppo, @AmberV, @kewms, @KB, and others for the great tips along the way.

[nontroppo]

Wow, looks awesome!!! Does your bash script have any other dependencies other than Quarto?

[cavalierex]

It pastes from the clipboard, so it requires POSIX (possibly macOS).

The last line in the script is squarto build, so that uses Python — mainly standard library, but I opted to add dependency on typer for CLI and loguru for logging. All this is available in the virtual environment selected by the bash script.

I actually don’t run Quarto via the bash script. The script ends with Squarto.

The workflow that I’ve gravitated toward is:

• Compile and squarto build (via the bash script) the first time.
• In Terminal, I manually run quarto preview once — which opens a browser window with a preview of the Quarto Book, so I get a nice view of what I’m working on.
• Then, I continue writing in Scrivener.
• When I want to refresh the preview (for example, to check how some math equations look, or to check if a Python or R figure is rendered correctly from code), I trigger the re-compile, Squarto re-builds again, and the preview automatically refreshes.
• Once I’m finally done working, I would manually run quarto render to get the final products.

Perhaps I’ll add options for triggering Quarto into Squarto’s CLI or add a TUI using textual. For right now keeping the Quarto preview/rendering separate from the Squarto build is working pretty nicely.

[cavalierex]

Cookiecutter for Scrivener+Quarto projects

I am considering adding cookiecutter functionality into Squarto.

For people who have never used it, Cookiecutter is a command-line program that will create a new project based on a project template. For example, to create a new research project, one could copy the Cookiecutter template that contains hierarchically nested folders and files. What makes Cookiecutter special, though, is that it will customize placeholder fields throughout the project – such as naming folders, naming files, and even inserting content within files – based on your answers to any prompts that were set up in the Cookiecutter template. So instead of getting the generic, blank template, it might start out partially customized to the specifics of the new project.

cookiecutter is a very popular choice for instantiating new projects based on a template. Brett Terpstra created his own program called planter, and we discussed some of the similarities and differences between Planter and Cookiecutter on his forum. The new kid on the block is copier. It seems to have some nice features. The docs provide a handy comparison table to see similarities and differences relative to Cookiecutter.

I am considering integrating some Cookiecutter-like functionality within Squarto so that it would facilitate creating a new Scrivener project using a template and filling in basic details like project name, author name, metadata, etc. From a user’s perspective, one would simply type

squarto new myproject.scriv

to use the internal default template, or

squarto new myproject.scriv --template=/path/to/mytemplate.scriv --bibliography=/path/to/myreferences.bib

and any other arguments that are desirable.

From an implementation perspective, this will require deciding where {{placeholders}} appear inside the .scriv project and its contents (.scrivx file, etc.). Since Scrivener itself creates UUIDs for various elements, the squarto new command would generate UUIDs dynamically as well.

Has anybody ever tried something like this?

It goes without saying that this will be an unorthodox method for creating a new Scrivener project, likely to face technical limits and problems, so this remains a research-oriented side project that will not be applied to “real work”.

[cavalierex]

Oh, and just to clarify the workflow better…

I would create a new project with
squarto new myproject.scriv. This would create my Scrivener project file using the Squarto template that is designed to eventually produce the Quarto book. Internally, all the metadata would have been set up appropriately for the new project according to the prompts.

From that point forward, all writing occurs in Scrivener in the usual way.

When done, the project is compiled, and the post-compilation script will run squarto build to create the Quarto book folder/file hierarchy.

Finally, run quarto render to produce the Quarto book PDF or web site.

[nontroppo]

It seems like quite a bit of work for what amounts to only a minimal gain? For my own work, editing my Pandoc metadata in a new template takes a few minutes, and tweaking my compile output choices using the GUI a minute or so. Answering questions wouldn’t be much faster than editing my metadata, the main benefit would be possibly less errors (as YAML can be fussy)?

[cavalierex]

Oh, you are certainly right… but you (of all people!) can surely sympathize with the DIY urge to tinker. It is the age long quest: can it be done?

Besides, the main benefit of automation is to my future-self. I tend to work on one or two “big” projects for a while. If some time passes before starting a new project, my future-self will have to re-discover what my past-self had done to get the last project off the ground. With a cookiecutter approach, I will be able to simply type squarto new, and start working in my new Scrivener project without any fuss.

[nontroppo]

@cavalierex

Oh, you are certainly right… but you (of all people!) can surely sympathize with the DIY urge to tinker. It is the age long quest: can it be done?

I can only wholeheartedly agree with this of course

[cavalierex]

I haven’t updated this thread in a while, and I need to do so soon. But here is a sneak peak at the progress that Squarto has made… It now has an integrated TUI (text user interface, courtesy of the magnificent Textual project). My Scrivener project’s binder is replicated on the left side. All the content files are read from within the .scriv package and converted from RTF to Markdown, and a preview of that is presented in the center content pane. In the upper right is a metadata inspector, and the bottom right panel shows the synopsis and the notes.

I was fooling around with adding a Markdown editor also, but there’s really no point, as I prefer to keep the editing in Scrivener itself.

This TUI will be the interface to an alternate compiler, so I can process the RTF file in various ways (e.g., extract embedded graphics and save to the /Figures/ directory; implement styles properly; clean up whitespace or stylistic preferences for Markdown; all sorts of things). The main goal is to export everything as separate files in the correct hierarchical structure for Quarto to then work its magic. I will also see if I can enable plain Pandoc and newer Typst workflows via the Squarto compiler.

I know this isn’t much of an update, but I will see if I can give more thorough explanation sometime.

[cavalierex]

Squarto update 2026-02-07

For readers unfamiliar with this thread, Squarto is my Python application that is designed to permit writing in Scrivener, outputting the appropriate folders and files for a Quarto project, and then exporting the final products such as PDF. Quarto itself is a system for scientific and technical writing that integrates Markdown, LaTeX or Typst, and executable code (R, Python, Julia, etc.).^[1]

Squarto has evolved quite a lot, most significantly in architectural design, but also in approach. While I do use it for some “real work,” Squarto remains a work-in-progress and hobby project.

Here’s an update to the workflow in early 2026:

Squarto has a CLI and TUI

The main way to run Squarto is via the command line, such as squarto new and squarto build. When installed by uv as a tool, Squarto can be run anywhere and its dependencies are taken care of for the user (no virtual requirement need be created).

I also created a text user interface (TUI) using the excellent Textual library to facilitate visualization of certain features.

A Squarto project has a standardized structure

Inspired by Cookiecutter/Copier and even Scrivener itself, I designed the squarto new command to create a new Squarto project as a hierarchy of nested folders and files:

📁/squarto_project_title/
│
├── 📁/_squarto_settings/

├── 📁/scrivener/
│
├── 📁/quarto_project/
│
├── 📁/products/
│
└── 📁/temp/

Writing takes place in Scrivener

All writing takes place in Scrivener, i.e. in my_project_title.scriv in the file listing below.

📁/squarto_project_title/
    │
    ├── 📁/scrivener/
    │   ├── 📁/my_project_title.scriv/
    │   └── 📁/compiled/           

The Scrivener implementation is a custom template (with Styles, metadata, and compile format).

I’ll describe the Scrivener template itself in another post. It is similar to what I posted previously and also similar to (inspired by) the approach taken by others, such as @nontroppo.

Because all writing is done in Scrivener, you can use all of its tools that help you write. Squarto doesn’t change any of that. It is only used when it comes time to compile.

Important settings are stored outside of Scrivener

In an earlier iteration of Squarto, I was trying to store everything in the Binder and then extract it all at compilation time. This had several downsides: (1) the size of the Scrivener package would balloon if you were storing fonts and Quarto extensions in there; (2) it was rather frustrating to edit some plaintext documents in Scrivener (such as YAML or JSON files). It made sense to store these assets outside of the Scrivener package itself. This is the main reason that I transitioned to making a “Squarto project” have a dependable structure.

Now, the /squarto_settings/directory contains the important bits, which includes Squarto’s settings, Quarto extensions, “presets” (which includes Quarto YAML configuration, LaTeX or Typst partials, CSS, font files, etc.), and templates (like DOCX templates).

📁/squarto_project_title/
│
├── 📁/_squarto_settings/
│   ├── 📄_squarto_settings.toml
│   ├── 📁/quarto_presets/
│   ├── 📁/quarto_extensions/
│   └── 📁/templates/

Presently, all these files need to be locally stored, but my plan is to allow for linking to external assets.

Specifically, I think that Quarto extensions might simply be specified in the settings TOML file and installed during compilation (instead of storing them in each project’s folder). The inspiration for this comes from uv, which does a great job at dynamically installing dependencies when a tool is run.

So I will continue to streamline this.

The Squarto settings TOML file is very powerful

A huge step forward for me was to bite the bullet and create a Squarto settings file that would be totally separate from the Quarto config. Quarto prefers using YAML, but I find it a bit annoying to write YAML; so I decided to use TOML instead. Here is what the _squarto_settings.toml file looks like:

[squarto]
squarto_project = "path/to/sample_squarto_project"

# Scrivener flags
build_from_scrivener_compiled_md = false
check_scrivener_includeincompile = true
exclude_scrivener_folders = [
  "PLANNING",
  "RESEARCH",
]

# Squarto flags
empty_quarto_project_dir_if_already_exists = true
empty_products_dir_if_already_exists = true
enable_citation_and_bibliography = true
copy_bib_file_locally = true
copy_csl_file_locally = tru
logging = true
verbose = true

# Quarto flags
quarto_preset = "default"
quarto_project_type = "book"
autorun_quarto_render = true

[processors]
quarto = "/usr/local/bin/quarto"
pandoc = "/opt/homebrew/bin/pandoc"
tex = "/Library/TeX/texbin/tex"
latex = "/Library/TeX/texbin/latex"
xetex = "/Library/TeX/texbin/xetex"
lualatex = "/Library/TeX/texbin/lualatex"
pdflatex = "/Library/TeX/texbin/pdflatex"

[bibliography]
bib_file = "/path/to/Zotero/MY_LIBRARY.bib"
csl_file = "/path/to/Zotero/styles/apa.csl"

In the [squarto] section, there are flags related to Scrivener, Squarto itself, and Quarto. For example, build_from_scrivener_compiled_md = false means that Squarto will not depend on a previously compiled Markdown document that Scrivener created; rather, Squarto will compile the Scrivener document from scratch (on its own). But the flag check_scrivener_includeincompile = true means that Squarto will still respect the Binder/DraftFolder items that were designated (within Scrivener) to be included at compile time.

The parameter exclude_scrivener_folders is a nice one. I was frustrated that Scrivener would only include DraftFolder in compilation, allowing at most Frontmatter and Backmatter to be added. Any other root-level 0 folders could not be included in the compilation. For example, if you wanted to store folders ‘Appendices’ alongside ‘Draft’, this couldn’t be included in compile in an easy way. Now, Squarto includes everything in the Binder unless it is excluded explicitly in the exclude_scrivener_folders parameter. In the example above, I listed ‘Planning’ and ‘Research’ to be excluded.

The Squarto settings file also allows specification of paths to any processor that you want. This gets stored internally as a key:value dictionary in the format { command: path }. So if you want to refer to different tools from the usual, you can.

Similarly, this is the location where you can specify the location to the bibliography .BIB file and .CSL file that you want to use. If the flag copy_bib_file_locally is set to true, then Squarto will copy the .bib file into the Quarto project directory later; if set to false, then a symlink alias is created instead.

I think the TOML settings file was critical in development of Squarto. It makes Squarto very, very extensible.

In the future, I think I will also list Quarto extension dependencies here, so they can be installed at compilation time.

Any necessary virtual environments might be specified here as well – A virtual environment is not necessary for Squarto anymore, but if Quarto needs to execute any R or Python code in a virtual environment, this may be the mechanism to specify that. Quite simply, Squarto will finish the compilation, activate the virtual environment, and then handoff to Quarto (which will continue to run within the virtual environment).

Squarto compiles the Scrivener package directly

In its past iteration, Squarto relied on Scrivener to compile the items in the Binder, outputting a singular compiled.md file that Squarto would then chop up again into pieces. (This is actually the reason for the name squarto, a pun from the Italian squartare, meaning to dismember or to rip into pieces.) This approach still works, but it relied on a few hacks to pass the proper file paths and other parameters to Squarto. That story is too long to reiterate but may still be present in the posts above or other threads.

Presently, Squarto is capable of doing the compilation independently. It does this by introspecting the .scriv package, which is itself a standardized hierarchy of folders and files. Thankfully, KB and the Scrivener designers based everything on plain text, so things were straightforward once I got used to it. The main .scrivx file is an XML document that specifies the ordering of the items in the Binder (among a few other things). Each binder item has a unique identifier (UUID) that is used to locate the data files related to what you type in the Editor. This includes the writing itself (content.rtf) but also information about Styles, Notes, Synopsis, etc.

📁/squarto_project_title/
│
└── 📁/scrivener/
    │
    └── 📁/my_project_title.scriv/
        │
        ├── 📄my_project_title.scrivx
        │
        ├── 📁/Files/
        │   ├── 📁/Data/
        │   │   └── 📁/<UUID>/
        │   │       ├── 📄content.rtf
        │   │       ├── 📄content.comments
        │   │       ├── 📄content.styles
        │   │       ├── 📄notes.rtf
        │   │       ├── 📄synopsis.txt
        │   │       └── ...
        │   ├── 📄binder.autosave
        │   ├── 📄binder.backup
        │   └── 📄styles.xml
        │
        └── 📁/Settings/
            │
            ├── 📁/Compile Formats/
            │   └── 📄Squarto.scrformat
            ├── 📄compile.xml
            ├── 📄projectpreferences.xml
            └── ...

Squarto basically performs a big ETL (extract-transform-load) process:

• Extract raw data from Scrivener
• Convert raw RTF to raw Markdown using an async Pandoc subprocess.
• Extract embedded images (storing files in /Figures/) and substitute with Markdown link
• Transform Style information into standard Markdown
• Transform myriad other details into clean Quarto Markdown format
• Load everything into a temporary SQLite database

This process is done asynchronously, and it is satisfyingly very fast!

Being its own compiler allows Squarto to do cool things

Before, when I relied on a Scrivener > Scrivener Compiler > compiled.md > Squarto workflow, there were many friction points where I wished that I could “preprocess” something before Scrivener compiled it, then “postprocess” it in a certain way before handing off to Squarto. Somethings were possible with workarounds, but other things were not possible.

Now that Squarto does the compilation instead of Scrivener, it opens up the door to many cool features. For example, now Squarto can differentiate between Scrivener Styles and regular RTF styles. That allows me to use regular command-B (bold), command-I (italic), command-U (underline), superscript, subscript, strikeout, etc. while I am writing, and it will be converted correctly to Markdown. I no longer need to create an alternative Scrivener style like strong or emphasis and then key-bind it to command-B and command-I. I only use Scrivener styles for the markup features like heading levels, marginalia, raw Markdown or LaTeX, citations/crossrefs, etc.

Another thing that being its own compiler enables is specifying the order of processing steps in a pipeline. For example, my default pipeline unescapes symbols, removes comments, optimizes bullets and tables and figure Markdown, etc. Specific pipelines could be specified per file (per item in the Binder), although I have found the default pipeline to work for most everything. The one exception is something marked ‘data’ or ‘raw’, in which case minimal processing occurs.

Squarto’s main output is the Quarto project

From the converted data, Squarto then creates the canonical structure of the Quarto project. A Quarto Book project might look like this:

📁/squarto_project_title/
    │
    └── 📁/quarto_project/
        │
        ├── 📄_quarto.yml
        │
        ├── 📄index.qmd
        │
        ├── 📁/draft/
        │   ├── 📁/frontmatter/
        │   │   ├── foreword.qmd
        │   │   └── preface.qmd
        │   ├── 📁/chapters/
        │   │   ├── chapter_1.qmd
        │   │   ├── chapter_2.qmd
        │   │   └── chapter_3.qmd
        │   ├── 📁/backmatter/
        │   │   ├── glossary.qmd
        │   │   └── references.qmd
        │   ├── 📁/appendices/
        │   └── ...
        │
        ├── 📁/figures/
        ├── 📁/tables/
        ├── 📁/data/
        ├── 📁/code/
        │
        ├── 📄references.bib 
        │
        ├── 📁/.venv/
        ├── 📁/_extensions/
        └── 📁/_fonts/ 

The structure of the Quarto Book is a direct translation of the structure of folders and documents contained in the Scrivener Binder. (From the ‘Draft’, folders become Parts, documents become Chapters, and nested subdocuments are aggregated into the parent Chapter. ‘Frontmatter,’ ‘Backmatter,’ ‘Appendices,’ ‘Figures,’ ‘Tables,’ etc., are other root-level folders parallel to ‘Draft’.)

The content of the _quarto.yml file is automatically updated with the appropriate metadata, such as paths to the various .qmd files.

All embedded images are extracted into /figures/ automatically. In addition, the Scrivener template has Binder folders named ‘Figures’, ‘Tables’, ‘Data’, and ‘Code’ where the user could choose to store files directly, and anything in those Binder folders would be extracted into the respective folders in the Quarto project directory.

Once the Quarto Project is created, it is processed as usual

Once the folder/file hierarchy for the Quarto project is created, it is simple to run quarto render on that project to create any output you want; e.g, quarto render --to=pdf' will produce a PDF document.

If the autorun_quarto_render = true flag is set in Squarto’s settings (TOML file), then squarto build will hand off to quarto render automatically.

Quarto products are moved to a dedicated directory

One annoying thing about Quarto is that it wants to do everything inside the same Quarto project directory – mixing temporary files and output files right there alongside your source (input .qmd) files. It makes for a big mess. Quarto allows you to rename some things, but not to move things outside of the main Quarto project directory.

Therefore, Squarto looks for the Quarto output products and moves everything to a nice clean directory:

📁/squarto_project_title/
│
├── 📁/quarto_project/
│
├── 📁/products/
│   ├── 📁/docx/
│   ├── 📁/md/
│   ├── 📁/html/
│   └── 📁/pdf/
│       └── 📄my_project_title.pdf

The Quarto project can be versioned using git

Since the canonical Quarto project is just a hierarchy of folders/files, it can be versioned via git commit. This allows tracking changes (via file diff), undoing changes,^[2] etc.

What I’m working on now…

Now that I have this pipeline working nicely, it would be really cool to improve the preset functionality so that I can use LaTeX or Typst classes/partials.^[3] I might also like to generalize beyond Quarto so that I could use a Pandoc-only workflow when I don’t need to have code be executed.^[4] There are certain limitations in the existing Quarto ecosystem that I may be able to work around – for example, using Typst (and its nice templates) in a Quarto Book project is not presently feasible.

Well, that’s enough of an update for now. I look forward to hearing your thoughts or suggestions for features to improve/add.

---

1. There is a lot of overlap between Quarto and Pandoc. The main reason that I use Quarto instead of just Pandoc is because of executable code. ↩︎

2. To be clear, “undoing changes” is at the level of the Quarto .qmd files, not at the level of the Scrivener RTF documents. I have not implemented (and am unlikely to try to figure out how to do) a round-trip back into Scrivener. ↩︎

3. I am really, really, really eager to be able to use the kaobook (LaTeX) or haobook (Typst) templates. Cf. GitHub - fmarotta/kaobook: A LaTeX class for books, reports or theses based on https://github.com/kenohori/thesis and https://github.com/Tufte-LaTeX/tufte-latex. and GitHub - ParaN3xus/haobook: A Typst book template inspired by kaobook ↩︎

4. Then I might be able to utilize other post-processing workflows for Pandoc that many others have developed. ↩︎

[eliamartin65]

Please make it available for Windows, @cavalierex!

[cavalierex]

I don’t think there is anything Mac-specific. But it is a work-in-progress that I can’t distribute until I feel it works flawlessly. It’s not ready for “production.” I don’t want anybody to have trouble and lose work. There are also important features to add, such as making a strict Pandoc workflow possible.

But I’m glad to hear of your interest, as that encourages me to keep working on it.

Please share a bullet-point list of friction points in your workflow right now, and I’ll see how Squarto could address those.

[eliamartin65]

@cavalierex: It’s just that I’m trying to create a Quarto Book using VSCode and I have no idea of what to do. Using Squarto might be easier?

[cavalierex]

If you are using VSCode to write (rather than writing in Scrivener), then why not use Positron instead? It was designed by Posit specifically for Quarto. Quarto – Positron

There is a Getting Started guide here: Tutorial: Hello, Quarto – Quarto

The point behind Squarto is to write in Scrivener and compile into a Quarto project, that then can be rendered.

[eliamartin65]

@cavalierex

If you are using VSCode to write (rather than writing in Scrivener), then why not use Positron instead? It was designed by Posit specifically for Quarto. Quarto – Positron

I guess I could keep VSCode for Typst and general coding and install Positron for Quarto. I do most of my writing in Scrivener, but when it comes to Markdown and other plain text, I prefer text editors.

[cavalierex]

Positron is a full IDE — it is a fork of VSCode. So you should be able to do everything in it. It handles writing Markdown, Typst, R, Python, etc.

[eliamartin65]

I prefer having one single editor for every language LOL.

[nontroppo]

Amazing and super cool!!!

@cavalierex

For example, now Squarto can differentiate between Scrivener Styles and regular RTF styles.

Do you preprocess the raw RTF before you pass it to Pandoc or use some other trick? Very neat either way!

As a more general point, formatting templates (classes/partials) should be fairly easy to integrate as they are really just packaged pandoc templates (Quarto has a few already avilable, including Tufte’ish ones Quarto Extensions – Quarto). I have converted a few TeX/Typst templates to work with Pandoc (thus should work with Quarto), it isn’t much work once you understand how pandoc templates work.

[nontroppo]

@eliamartin65

Please make it available for Windows, @cavalierex!

Do you use WSL, as this will make everything much easier I think for running this on Windows?

[cavalierex]

I start with the RTF, which read as plaintext looks like this:

\pard\tx360\tx720\tx1080\tx1440\tx1800\tx2160\tx2880\tx3600\tx4320\sl264\slmult1\pardirnatural\partightenfactor0
\cf0 This is normal text. This is 
\f6\b bold
\f3\b0 . Back to normal text.\
\pard\tx360\tx720\tx1080\tx1440\tx1800\tx2160\tx2880\tx3600\tx4320\sl264\slmult1\pardirnatural\partightenfactor0
\cf0 \
This is normal text. This is 
\f7\i\b bold and italics.
\f3\i0\b0  Back to normal text.\
\
This is normal text. This is 
\f8\i italicized
\f3\i0 . Back to normal text.\
\
This is normal text. This is \ul underlined\ulnone . Back to normal text.\
\
This line has a superscript\super 1\nosupersub  in it.\
\
This line has a subscript\sub 2\nosupersub  in it.\
\
This is normal text. This is \strike \strikec0 Strikethrough\strike0\striked0 . Back to normal text.\
\
This is normal text. This is 
\f0 \cf11 <$Scr_Cs::22>
\f9 SmallCaps
\f0 \cf0 <!$Scr_Cs::22>
\f3 . Back to normal text.\
\
This is normal text. This is 
\f0 \cb12 <$Scr_Cs::23>
\f3 Highlighted Text
\f0 \cb1 <!$Scr_Cs::23>
\f3 . Back to normal text.\
\
\
This is a 
\f0 \cf2 <$Scr_Cs::16>
\f3 Markdown span with some symbols [] that should not be escaped
\f0 \cf0 <!$Scr_Cs::16>
\f3 . And here is what the span 
\f0 \cf2 <$Scr_Cs::16>
\f3 *
\f0 \cf0 <!$Scr_Cs::16>
\f3  would look like if it contained only an asterisk. In contrast, this is a regular sentence that has an asterisk * in it (and it should be escaped).\
\

The first step is Pandoc conversion to raw Markdown, which looks like this:

  
This is normal text. This is **bold**. Back to normal text.  
  
This is normal text. This is ***bold and italics.*** Back to normal text.  
  
This is normal text. This is *italicized*. Back to normal text.  
  
This is normal text. This is <u>underlined</u>. Back to normal text.  
  
This line has a superscript^1^ in it.  
  
This line has a subscript~2~ in it.  
  
This is normal text. This is <s>Strikethrough</s>. Back to normal text.  
  
This is normal text. This is \<\$Scr_Cs::22\>SmallCaps\<!\$Scr_Cs::22\>. Back to normal text.  
  
This is normal text. This is \<\$Scr_Cs::23\>Highlighted Text\<!\$Scr_Cs::23\>. Back to normal text.  
  
  
This is a \<\$Scr_Cs::16\>Markdown span with some symbols \[\] that should not be escaped\<!\$Scr_Cs::16\>. And here is what the span \<\$Scr_Cs::16\>\*\<!\$Scr_Cs::16\> would look like if it contained only an asterisk. In contrast, this is a regular sentence that has an asterisk \* in it (and it should be escaped).  
  

You’ll notice that bold and italics already are marked-up using asterisks, and some things like underline and strikethrough are marked-up using HTML.

The Scrivener markers are still in there, such as <$Scr_Cs::23>, which indicates Scrivener character style #23.

A few problems exist at that point:

(1) Pandoc escapes a zillion things that shouldn’t be escaped. Some things are escaped in HTML and some things are escaped in the rest of the text. For example, the Scrivener style marker should be <$Scr_Cs::23>, but it appears as \<\$Scr_Cs::23\>.

(2) If a Scrivener style employed regular styling (like bold, italics, etc.) so that it looked a certain way on your screen, that mark-up was also present in the initial Markdown conversion. So let’s say I have a style that I call ‘Yelling’ because it uses bold, italics, and underline. In Scrivener, it would make sense for the style to look that way. But then when it converts, it will not only preserve the Scrivener style number, but it would have the ***<u>…</u>*** markup within the Scrivener style markers.

Thus, the processing pipeline was born to sequentially remove all the issues. Order sometimes matters, sometimes not. The default pipeline unescapes HTML, removes <!-- --> comments, cleans up white space around Markdown features, replaces HTML spans like <u>…</u> with Markdown spans like […]{.underline}, cleans up and converts the Scrivener styles into Markdown, and then does some of the other cleanup that you and I have discussed before, like repositioning the image attribution tags and using fig-alt as the caption for the image.

Escaping/unescaping remained a frustrating challenge in this. The last step, unescape_symbols_in_image_tags(), was necessary because some escaping persisted in the paths to images wherever there were symbols like underscores in the filename.

DEFAULT_PIPELINE = [
        partial(unescape_html_symbols),
        partial(remove_empty_html_comments),
        partial(remove_leading_spaces_in_normal_paragraphs),
        partial(clean_bullet_lists),
        partial(remove_empty_tables),
        partial(replace_html_tags_with_markdown_spans),
        partial(clean_scrivener_style_spans, styles_used=styles_used, scrivener_styles=scrivener_styles),
        partial(clean_scrivener_tables),
        partial(replace_image_tags_from_reference_to_inline, output_dir = settings.paths.quarto_project_dir, figures_dir = settings.paths.quarto_figures_dir),
        partial(reposition_image_attributes_in_curly_braces),    # Do this for images before handling the general case
        partial(replace_image_title_caption_with_alt_text),
        partial(reposition_general_attributes_in_curly_braces),  # Do this general case after handling images
        partial(unescape_symbols_in_image_tags),
        # partial(clean_whitespace_and_style),
    ]

Sooo…. after the processing pipeline, the clean version is this:

This is normal text. This is **bold**. Back to normal text.

This is normal text. This is ***bold and italics.*** Back to normal text.

This is normal text. This is *italicized*. Back to normal text.

This is normal text. This is [underlined]{.underline}. Back to normal text.

This line has a superscript^1^ in it.

This line has a subscript~2~ in it.

This is normal text. This is ~~Strikethrough~~. Back to normal text.

This is normal text. This is [SmallCaps]{.smallcaps}. Back to normal text.

This is normal text. This is [Highlighted Text]{.mark}. Back to normal text.


This is a Markdown span with some symbols [] that should not be escaped. And here is what the span * would look like if it contained only an asterisk. In contrast, this is a regular sentence that has an asterisk \* in it (and it should be escaped).

Very satisfying.

And the best part is that it is super fast. It takes microseconds for a single document and less than 1 or 2 seconds for a big book. It is really Quarto and the PDF conversion process that is slow.