As someone dabbling in coding, I’d love to be able to use Python scripts alongside Scrivener. I have a few ideas for scripts I want to create, but I have no idea how to combine Python + Scrivener. I’d appreciate ideas and tips.
PS: If this is in the wrong section, admins can move it.
Others can comment but as an experienced and enthusiastic long-time user of both Python and Scrivener, despite Python’s power and sophistication since L&L does not provide an API for Python to use it is not possible. Yes, you can dig into the files inside the Scrivener project package/folder but that expressly not recommended or supported by L&L.
I suspected something like this, but I wanted to confirm.
Do you have any workflow tips? I didn’t mean using Python to directly manipulate a Scrivener file (though it would be nice), but rather using Python in the post-processing compile pane.
Although I don’t know what you would like to do nor do I have very high skills in process automation, I think you could use Scrivener’s function of syncing an external folder to make changes there that can be automatically imported into your project. Avoid, however, any operation that involves changing the encoding or altering the structure of the XML files.
Regards.
All down to what you now do manually that you want to automate, what you wish to do, or even what would be fun to try and do.
I have no “work processes” to suggest other than to use Python in a virtual environment to keep it and the libraries you import to be independent of anything that the system (macOS or Windows) has and expects.
I’ve automated quite a bit using Vite + Typescript (basically vibe-coded three apps to help with the outline process). I’m now trying to do the same with Python, but for worldbuilding and note-taking specifically.
You might like to read this thread here: Scrivener → Squarto
I haven’t updated the thread in a long time, but I started working on an alternative compiler called squarto (coded in Python). Long story short, all the composition is done in Scrivener, but then I do the processing via Squarto to create the Quarto Markdown files necessary for a book project (cf. quarto.org for details about Quarto].
In the thread, I was debating between having Squarto be totally independent of the Scrivener Compiler vs. allowing Scrivener to start the compilation step and then handing off to Squarto for post-processing. I have the latter version (post-processing only) working pretty nicely, but I continue to work on a more robust alternate compiler. It’s still very much a work-in-progress. I will have to update the thread.
You can also look for posts in other threads by @nontroppo , who is a guru at integrating Scrivener with post-processing steps. In his case, he uses Ruby scripts.
I did look at your Squarto thread, but since I run Windows, I wouldn’t know how to replicate that.
The main thing in my workflow is that I am reading the source data files from within the .scriv package. On macOS, the .scriv file is really a “package” – i.e., a folder named with a .scriv as an extension, and inside the package is the .scrivx XML document and all the supporting data files. On Windows, the Scrivener package has the same structure, except it looks like a directory instead of a single file, but inside the directory you will find the same .scrivx file and data files. You can create a new test document in Scrivener (one that you don’t care about!) and then poke around a bit to figure it out.
Now, poking around the internals was my approach. But others like @nontroppo allow Scrivener to do the normal Compile step, and then process the compiled document using Ruby (or Python) to do various things. This is a safer way to go, as you don’t mess around with the actual Scrivener document, but rather only the exported compiled Markdown file.
Anyway, I suggest you create a new test document and play around a bit. But be forewarned that it’s easy to fall into the rabbit hole!
Noted.
Any suggestions for something easy I can do?
To continue on in that vein, of automating via working with the project format itself, you can certainly do things at even a feature level, I would say, and not all would be terribly involved or difficult to learn. We made the XML files deliberately easy to understand and work with.
For example, say you would like to search by custom icons. Project Search doesn’t have this capability, but if you look at the .scrivx file for how custom icons are set it would be pretty straight-forward to create a tool that lets you search for a word/phrase against icon assignment in the binder, and print out a list of search results using their binder titles, maybe even tab indenting the output based on the original outline hierarchy.
Once you get that working, you could go a step further and make this useful in Scrivener, too. Instead of printing a list to STDOUT, you could fetch the UUID from each XML node that uses the icon, and append it to a Collection you create with the script, inserting it into the .scrivx file in the right place. You could even set it to load that collection in the binder the next time the project is loaded. Create a simple collection in a test project to see how easy it would be to inject a collection structure from scratch and populate it with a list of binder items.
<Collection Type="Arbitrary" ID="C630324E-4EE4-4CCA-BA02-A3400A27747C" Color="0.458824 0.313726 0.482353">
<Title>Purple Flag Results</Title>
<BinderUUIDs>D3A2A7C8-9E0B-4E8C-AB85-0F5D8ED73B04,27E89E8D-E832-4E5C-8E17-EC90A5EEE476</BinderUUIDs>
</Collection>
The caveat is that for things like that, where you are modifying the project, you must close the project to run your script. But if your scripts are passive only, they can be used while it is open (a Mac tool called Marked does this, assembling a full-draft preview for Markdown-based writing, to a pretty CSS output, that updates as you write). Even if your ultimate aim is to modify the project, during the testing phase it is good to know you can open .scrivx files in coding editors and monitor how the software updates it against your actions in the GUI.
Thank you once again @AmberV!
Do you have any planning/outline specific script ideas?
I think @cavalierex and @AmberV have given pretty comprehensive answers related to tweaking the project bundle itself. Squarto is really cool!
But for me at least native Scrivener is more than capable in the writing and compile phase; I personally don’t need to tweak this part. But I do output to many formats and this is where post-compile automation kicks in. I use Ruby scripts to do some initial fixes as what Scrivener wants to output may not align with my final layout tools. For example moving cross-reference labels from where it is easy to place them in Scrivener to where Typst or Quarto may need them. This translation allows me to write using a single interface and use native tools like Scrivener metadata.
But probably my greatest benefit has been writing Pandoc filters. Pandoc creates an abstract syntax tree, and this means each chunk of a document has a structure and that structure can be manipulated in many ways. You can write pandoc filters in Python or Ruby[1]. Lets say you want an index, you can AST scan for a unified marker for index items that you use in Scrivener (an inline style rule) then translate to HTML/TeX/Typst/DOCX/ODT native indexing. Or you could parse all citations and create multiple bibliographies related to specific categories (or create compile specific databases of references to share with colleagues). Or build mini TOCs per chapter. The world is your oyster! As long as you can tag it you can tweak it. The AST helps as it minimises brute regex replacing which is what you would normally do with Python or Ruby directly. Scrivener’s styles + section types + AST can do just about anything. For example you could make a Python runner, so you make a Scrivener Section type that would wrap python code. Then the post processor would take the code and run it, inserting the results of the python code into the final doc. This is what things like Juypyter do but made by you using Scrivener as a frontend.
TLDR:
though Lua is better as it is already embedded in Pandoc and it works at a lower level than Python or Ruby. ↩︎
I had no idea you could write Pandoc filters with Python. WHAT?
I wonder if AI could help out, since I’m not a coder.
Neither is “A.I.”. It will give you some half-baked solution and you won’t know why the other half is broken.
I’ve successfully written Vite web apps using AI, so I’d disagree. It makes mistakes sometimes, but I can correct them easily.
Well, then I’m an architect. I’m experienced in Lego.
I do use AI for coding tasks. Because LLMs exhibit no actual intelligence, and are just smart statistical compression and recollection of large amounts of data, they are great at tasks that have lots of training data, so making web pages is certainly well served in their training. AI has stunned me occasionally with superb breakdowns of a complex problem and spot on solutions in domains that I know enough about to be able to judge (and probably because previous solutions existed and were trained on). But then, for other seemingly simple problems, they stumble into “oh yes you are right my solution is incorrect, I have found the error and it should work” which keeps repeating endlessly and it is obvious they don’t know what they are doing. My preferred shell language is Elvish (https://elv.sh not LOTR), and the “best” and “most smart” last gen LLMs stumble around like overconfident idiots writing th simplest of functions…
Having said all that, at least for Lua, I had some success writing functional pandoc filters (using agent mode and lots of code context).