Compiling To LaTeX -- One file per section?

Scrivener Scrivener for macOS
,
1

Hi there,
I’m using scrivener to write my bachelor thesis, using MMD->LaTeX->PDF workflow. I write my draft in Scrivener 3.01 and compile that into LaTeX files. While the hierarchical structure in Scrivener works best for writing thesis, the compiled LaTeX file always results in a single file that contains all of the chapters, sections, and subsections. It’s hard to debug LaTeX syntax in a single file.

So I want to ask that is there any method to compile each section (or chapter) into individual files?

Thanks.
NiwTR

2

I recently put together a little demonstration of how Scrivener could be used to generate a bunch of WikiMedia formatted .txt files from a single compile job. The premise is very simple, and only took a few minutes to convert to a LaTeX splitter instead:

  • Have a compile format with a Layout that generates a token as a separator, to indicate where the next file should begin.
  • Build a shell script that splits the compiled output up by this separator and prints each chunk of text into its own temp file.
  • Process each temp file using MultiMarkdown/Pandoc/etc. to create the .tex file and save the output into the compile target folder.
  • Don’t worry, with Scrivener we can automate all of this. :slight_smile:

File splitter sample project (226.5 KB)

The attachment includes a sample project and a quick and dirty Ruby script that handles the splitting and conversion:

  • Scroll through a few of the sample documents. It’s all pretty basic MMD-formatted stuff. There are a few footnotes and images for testing to make sure they all end up properly referenced in each .tex file that needs them. Of particular note here is that you shouldn’t need to change much if anything of how your project is currently set up. You don’t even have to use folders and files if that’s not how you work—the whole Section Type <> Section Layout binding system gives you that flexibility. I’m using a Section Type called “LaTeX File” but that’s just for clarity—more likely that would be “Chapter”; the name doesn’t really matter.

  • Go to File ▸ Compile... and take a look at the preview column in the middle, where the layout being used by “LaTeX File” is of most interest.

  • The rest is all done in the “MultiFile Output” compile format on the left. Double-click that to edit it.

  • The “New File” Section Layout is what is assigned to folders. Click on the “Prefix” tab to see where the split marker is inserted. It’s nothing special, only sufficiently unique enough to probably not be encountered naturally.

    I am using the <$title_no_spaces> placeholder to generate file system friendly names from the folder titles, but it might be better to clean filenames in the post-processing script if your chapter titles have potential problematic characters in them. The serial number will help keep these files in binder order on the disk.

  • Next, in the Text Layout pane we’ve inserted a manual file split marker at the very bottom of the document. This will directly precede the chunk of text where Scrivener inserts figure and footnote references. The way this works is that Scrivener appends all markers at the very bottom of the document. Our script will lop that off and append it to each temp file. The unused reference tokens from other sections are no matter, MMD ignores them.

  • Last but not least is the Processing pane. We enable post-compile shell scripting, with the script itself embedded in the compile settings (Click the Edit Script button to view it. A copy of the script has also been provided as a stand-alone file, for your reference). Lastly we delete the temporary Markdown file used to generate the output, leaving only the .tex files. You can switch that off if you want to “debug” the process or modify the script.

    The script itself doesn’t take any arguments other than the compile output filename, which Scrivener automatically supplies if you do not otherwise.

Save the Format with the updated path to the script and give it a test compile. Note that since this includes images, you’ll be giving a folder name when asked for a name at the top, all of the .tex files and images will end up in that folder. If your work doesn’t have images, then you might want to create your own folder to save into, as this will of course be generating a bit of clutter.

You’re going to need your own copy of MultiMarkdown installed for two reasons:

  1. The Ruby script assumes ‘multimarkdown’ is on your path.
  2. Everything is set up for MMD6, including the .tex boilerplate files for “article”, which is what the demo settings currently are. If you want something else, edit the “LaTeX Config” metadata key in the compile overview screen to “memoir-book”, or whatever you need.

I think that’s everything that might need explanation. Let me know if you can’t get it working or run into any other problems or areas that are confusing. If you want to import this into your project, it might be best to integrate the ingredients into a duplicate of your main compile format if you do a lot of custom stuff with it. If you’re using a pretty stock setup though, easiest thing might be to just export the format from this demo project as a file (gear button below Format sidebar in compile overview) and import into your project—or edit the format and save it to “My Formats” instead of being project-based. That’ll open it up to every project you have. Each compile Format has its own layout assignment settings, meaning you can switch between a singular PDF and a bunch of .tex files with a single click once you get it set up.

3 Likes
3

Thanks and what a clever job you’ve done! It’s very fascinating and fairly intuitive. I’ve managed to follow your demo project and your script works just fine. I also thought about the idea of making a placeholder that indicates where to cut files, but I screwed up while you made it work.
It’s a cool ruby script but I still need to modify something to make it apply to my work. Your method generates LaTeX headers and footers for each .tex files, while I need them only for the 1st chapter file (the headers) and the final chapter (the footer). Besides, the sequential natures between .tex files have been broken and I need to manually add \input{next_chapter.tex} in each file. – Of course, those are my own jobs, and thanks again.
You know what, that’s why I LOVE relatively small companies like LiteratureAndLatte, which has friendly communities, helpful supports and enlightening discussions. It won’t be so difficult for you to write that script for me, but your introductions are so detailed and I can tell you’re trying your best to help your customers. I’d be very happy to hear about that’s your principle of doing business, or something else. Just hold on, keep calm and grow Scrivener!! :smiley:

4

Ioa, this is awesome. When I saw this question come up I was going to answer, “You would need to write a script to do this.” And then, voilà, the script appeared. Thank you!

5

A neat trick here is that if a document lacks a metadata block, MMD won’t turn it into a full typeset-ready file, and rather export a “snippet”, suitable for \input into another file. So to get that behaviour you’d only need to modify the script slightly so that metadata_block isn’t printed into each temp file, rather only to the first. You could do that simply by setting metadata_block to an empty string at the end of the loop. That way only the first chunk to reach the end of the loop gets anything added to the top of the file.

You could also automatically assemble that from the tempfile array as well, since that has the filenames in it, then the list would adapt to whatever folders you compile. It would make sense to have the script work that way if it is only going to generate one full .tex and a bunch of snippet files following it, too.

And thanks for the kind words to both of you. :slight_smile: Glad to have helped—and yes that is something we try to do around here!

6

Yes, great answer from AmberV, as usual! And a very elegant script, Ruby can be so concise yet powerful…

A bit offtopic, but if you are not a Ruby expert, ruby has a great debugger than can debug scripts even if they are run by other programs like Scrivener: viewtopic.php?f=14&t=52158

7

A nice tip! I vacillate between old-school “puts” style debugging and formal testing suites for larger projects. I should really take the time to learn a better ad hoc method like this—that’s a neat trick running the interface through a server for non-interactive contexts.

8

Thanks, I think I nearly understand how to make this do something I might like!

If I may be so bold, this example would be 10x better if the writeup shown here in the forum were part of the text of the sample .scriv file.

9

Amberv: actually one thing that would make these kind of sample projects easier to use is if you could store the script in the Binder, and it got exported to the compile folder during compile. In general the idea is that we have some “accessory” files that get automatically exported on each compile. We could put scripts, LaTeX or other templates, CSS, YAML metadata or anything else really in a research folder in the binder, and this just gets auto-exported. Then you create a project.scriv that would require no manual setup, the script gets copied to the compile folder and the post-processing pane uses the current folder for its path.

1 Like
10
  • 35 to that idea!
11

You can use Scrivener to compile individual chapters and then use the LaTeX \include command to debug individual chapters. So your base document is something like:

\begin{document}
\include{chapter1.tex}
\include{chapter2.tex}
\end{document}

Remember not to include \begin{document}…\end{document} in the individual chapter files.
Here’s more: https://tex.stackexchange.com/questions/187677/how-to-include-chapters-in-latex

Keep in mind also that moving \end{document} to different places in the document is a good way to find where the problems are in a LaTeX compile.

12

I gave some consideration to the notion that a script will be located within the project in a predictable folder under its internal UUID, but what is not predictable, nor obtainable, is the path to the project itself.

My main issue with the idea you bring up is two-fold: you always get a compile folder (it’s nice when you need one, but not if you really don’t), and if you aren’t intimately aware of all the processes involved here, the result would be terribly mysterious. What is this geek code that keeps coming out of Scrivener when I compile? Do I have a virus? I think it’s a fair point, since the whole idea here is to package up these more complex implementation into easy to use Formats. Overall it just feels a bit messy to me for this particular purpose (leaving aside the whole: exporting support files in general conversation).

Another idea is an approach similar to Cover images, whereby you can select a script from a dropdown menu an alternative to typing in someting in the path field. Problem there is: what’s a script? Any text item in the binder? That could make for a massive and 99.9% useless dropdown menu. “Research” files with certain extensions? The Mac doesn’t have a convenient umbrella “Kind” for them, rather they are all identified by language, and using inconsistent nomenclature at that: “Ruby source”, “Shell script”, “Python Source”, “Lua Source File”… at least with the Cover dropdown we have an “Image Document” kind, that handles everything that is valid for selection. So that one doesn’t strike me as a good alternative either. I don’t know about you, but I’d rather type in a few dozen characters into a field than spending five minutes poking through a 1,000 item dropdown.

The best idea I’ve enccountered is something I’ve seen done in a similar situation. Hazel, the Mac folder and file automation tool, has the ability to embed a script right into the settings as an alternative to pointing to one on the disk. In the Processing pane, a radio selection could allow one to select between typing in a path and embedding a script. The embed option would have a button that loads a popup editor (hazel has fancy language detection and syntax highlighting, so maybe there is a library out there that could do that). The script is stored within the project as a file, rather than embedded into the compile XML, and in fact using this approach internally sets the path to /path/to/project/settings/<FORMAT_UUID>-compile_script.

In the context of the script being used here to split a draft into multiple parts, I would consider automating that process. The script stores an array of .tex filenames in the first loop. From that one can easily assemble a master .tex file from this array, between the first and second loops.

Now if you change a chapter name, or shuffle things in the binder, you don’t have to go back and update this master file each and every time.

13

The Hazel way sounds cool. Alfred also allows editing of scripts that are bundled in its workflows, something similar.

I suppose because I do think that having “Support files” for compile would be a useful addition anyway, I imagined this would be the most “gain” per developer time spent :smiley: The user would select a research binder folder, and its contents would end up in the compile folder. I think if the Support Files were clearly shown in the compile interface, there would be no “surprise” when looking in the compile folder. For paths, it would mean for Scrivener to be able to use the current folder (./ would be set to the compile path — what I do for my pandoc filters in my scrivomatic sample project which currently requires manual export + manual editing of the post-processing path).

1 Like
14

The support file thing is another topic entirely from this I think. It’s a stretch to put internal compile logic, used to produce an output of some form, into the same group as files used to support a file you’ve compiled (like CSS). The script in nearly all cases is as unnecessary once you have compiled as the table of Replacements are.

15

Yes, I think.

And yet, it seems to me that we are dancing around an important topic, which is that while Scrivener “wants” to create one large compiled document, the universe often “wants” an ordered collection of smaller documents, be they HTML files or LaTex files or whatever.

Scrivener happens to have triggers already, based on indicators in the files and folders, such as the inserted >>>> that Amber uses. It seems to be possible to insert constant plus variable text at beginning (or end?) of any designated component, based at least on structure, maybe on other criteria I don’t understand.

So we know there is a moment inside Scrivener when it is “aware” that something needs to be done. All it presently knows how to do just now is “insert this text”. It could also know things like "close this file and open another one named ". That change would require some code, but we know from the program’s behavior that there is a place for that code to stand.

Another action, of course, could be “run the script of this name in the output folder”. Whether the script itself were stored in Scrivener is a separate topic.

As it is Scrivener insists on making my file into a single glob, and then I can, if sufficiently skilled, run a script to take it back apart, despite the fact that it was already “apart” in Scrivener’s binder!

Scrivener remains very opinionated, but its structure clearly would allow it to be less so. I’d like that, because I like writing in Scrivener’s fashion – I think – and I am constrained by the universe I live in to produce output that Scrivener could help with, but does not.

It’s a puzzlement …

16

It’s not really such a puzzlement. Scrivener was always designed - from the very beginning - to produce book-length work, a single document built from multiple components. That is the very core of the app. Requests for multiple files from Compile are few and far between. Creating multiple files is an entirely different process that would require its own logic (continuing page numbers from a previous file and such like for RTF/DOCX), and I have no plans to do add anything like it.

17

Understood, if not enjoyed. Rock on!

18

:smiley:

19

A suggestion. Since, on the one hand, many publishing paths work best with separate files (e.g. leanpub, pragmatic, web pages), and since Scrivener is philosophically opposed to catering to these heretics and fanatics, it might be a service to provide a packaged “splitter” solution, along the lines of Amber’s, with instructions in the tutorial how to do it.

It might not be too difficult to set up a template (if that’s the word), a portable splitter program, and then folks like me, who have fallen into the dreadful trap of wanting separate files would have a clean, neat, and most important packaged and working solution.

As it is, all us poor devils here in the outer darkness are condemned, not only to the standard and well-known weeping and gnashing of teeth, but also to redundant yak-shaving and rolling of our own solutions, foregoing temporarily or perhaps even permanently, the joy and pleasure that accrues to the True Scrivener folk. Think how wonderful!

:smiley: but serious

20

Ioa’s script will only work with MultiMarkdown (or plain text as of 3.0.3). It won’t work for other formats.