Separate Files, DocBook, Pragmatic Programmers

Scrivener Scrivener for macOS
1

Hi All,

I’m loving Scrivener for creating my book. Lets me organize, move things around, all kinds of good things. Best writing tool I’ve found in decades. That’s the good news.

More good news is that Pragmatic Programmers have accepted the book I’m working on and will publish it.

The bad news is that their input form is an XML structure, rather like DocBook. Every chapter in a separate file, things like that. They do also accept MarkDown, and since my formatting will be pretty simple, I’m good to go with that, I think. So at least I won’t be typing all over. Or at least not much …

One complication is that I would like to use the notecard titles and synopsis as the chapter title and epigraph for each chapter. So I’d really like to COMPILE the document, not export it. I can of course just retype this info into the chapters if I must.

So I’m looking for some advice and discussion on options for keeping the work in Scrivener as long as possible, and producing a bunch of chapter files at the last minute, ideally by including Title and Synopsis, but that’s not absolute, since I can just type them in.

So … thoughts, please?

2

Congrats! I’m a fan of PragProg, they have consistently good tech books.

What I would recommend is using the Separators panel to insert a “cut” sequence wherever appropriate (where a page break would otherwise occur)—just something unique like “%%%”, then write a little script that mows through a file line by line, dumping out a sequence of documents by starting a new one whenever the cut sequence is located.

Or if that isn’t something in your skillset, another way would be to drop the compiled document with these cut separators into a temporary folder in the project, using File/Import/Import & Split. That will chop everything back up into chapter-length files which you can then turn around and bulk export. :slight_smile: Not terribly elegant, but it should work, and it also leaves you free to plan out the book in an outline that doesn’t necessarily resemble the output file-for-file. To put it another way, since you can insert a cut separator at say the text/folder boundary, that means you can use any number of files to represent the chapter. That will all get merged when you compile and a “%%%” is only inserted once around them all.

3

Yes, Amber, I’m thinking I might do something like that. The script could get more and more powerful / convoluted … inserting their boilerplate, and so on. And there is a Scriv2Doc script out there from the guy who wrote the Google App Engine book for O’Reilly that I could scarf up.

I’d really like to keep writing in Scriv. In the end, I’m sure my editor will start editing the individual files and after that the Scriv -> Files path will be iffy. But in the meantime I prefer keeping “control” inside Scrivener.

I’ve noticed that Scriv to Leanpub, and now this path, are not as well supported as they might be. I hope that as those publication paths become more popular, we’ll see more options. I’m getting quite fond of Scrivener.

Thanks,

4

Yeah, the point where you start going back and forth with your editor is going to be difficult no matter what workflow you use. Some people can manage (some can convince their editors to use Scrivener) but I think most consider the work done in Scrivener at that point and continue on in whatever end-product tools they need.

One tool you can consider is syncing binder documents with an external folder. This dumps out the editor content to a sequence of files that can be shared with your editor (a shared Dropbox/similar folder works well for this). When they are done with a section you can hit the sync button and pull their changes back into the project, complete with automatic snapshots and all revised documents gathered into a Collection for you. The changes you make in Scrivener are likewise updated on the filesystem. So if you keep your sections long enough, your editor might be keen on that.

In my opinion, I think the best path for these technical formats in Scrivener is MultiMarkdown. I don’t just say that because I use that myself for everything, but because it has a scriptable back-end and works in a full semantic space in the editor. Another alternative, I have heard, is Pandoc. I’ve never looked too deeply into it myself, because I’m happy with MMD and it has all of the formats I need to get work done, but Pandoc has a much larger array of file formats supported, and does even offer a mostly complete conversion from MMD to Pandoc. One could either write in Scrivener using Pandoc entirely, or use MMD as per normal and then convert. One thing Pandoc does support is native DocBook output. So if it would be easier to get to the format you need from DocBook, that may be something to consider.

5

Thanks … syncing looks useful. And I definitely want to use markdown, which they can support to some degree.

So far, the only other Scrivener user I’ve found is using Scrivenings mode and pasting stuff into a text file. I’d like to do better than that but the point is to write, not do tool work.

Where should I look for scriptability on the MultiMarkdown stuff? There may be some easy stuff I could do to ease this transition …

Will also check out Pandoc, thanks!

6

What would be REALLY useful would be an outline that lets one sync down to some level, and include all the files below.

So you could have, say, a section, with lots of paragraph texts underneath, but it would sync the section as the append of the paragraphs.

Oh well, gotta have a dream …

7

Agreed, a tool is only useful if it saves you time, and I would argue that at its most basic, the compiler is drop-dead simple and the only reason some people don’t use it is because they don’t know that. All you need to do is pop it into Original mode to get close to the same result as copy and paste. For this argument, the question is whether MMD or another form of format generation is easier than typing in XML during the creative phase. For myself, it nearly always is, even if I need to do a little scripting now and then. I’d rather spend a day or two getting a workflow built than sacrifice a natural writing environment—personal taste though.

Anyway, if you want to play with post-compile scripting, you will need to install the Mac support installer. This will install shell scripts into [b]~/Library/Application Support/MultiMarkdown/bin[/b].

To make Scrivener use these, you will need to switch on “Use XSLT post-processing” in the Compatibility compile option pane.

Now examine the mmd2tex-xslt shell script and modify the contents to do whatever you want—maybe even not use MMD at all. Here is where you could insert Pandoc automation if you wished. Scrivener just uses that script when you choose LaTeX, pipes data to it via STDIN and waits for STDOUT. Once it has that it proceeds with creating the file. So it doesn’t care what you do in between input and output.

#! /usr/bin/ruby STDOUT.puts "There were #{STDIN.readlines.count} lines."

This example will cause Scrivener to create a text file that merely consist of a line count of the original compiled output. If you output anything to STDERR, it will abort the file write and dump the error string to a UI window in the software for handy reference.

It’s a bit of a kludge, but Keith didn’t want the compatibility pane to be that fiddly, so there isn’t a way to supply a custom shell command. Working with the “LaTeX” script to make it do stuff other than make LaTeX files is easy to do though at least.

So if you already have some tools built, you may be able to just click them into place.

Could you clarify that? I think Scrivener can do what you are describing, but I’m not sure what you mean by “sync down”. It is capable of handling mixed roles based on depth. For instance levels 1–3 of the outline can be topologically significant in the book, but then every level four or greater would emit no heading for itself, thus obscuring the outline from the output allowing you to work with a finer level of detail than is required for the reader. I use this technique myself for the Scrivener user manual. I use Part-Chapter-Section-Subsection-Subsubsection, and below that point all outline text is output without headings, causing them to be essentially grouped with the last major heading.

8

Thanks for this, it’s a good hook for starting. I’ll try to keep a link to this.

The output I need is all specialized XML that the Pragmatic Programmers use to generate all their books. Once one’s book is marked up, they run it though various tools that apply the styles they want to get the various forms, paper, ePub and so on.

I’ll compare this to Leanpub format at some points as I go along.

Both Leanpub and Prag want a top level control file, called Book or something, that lists the files to be included. These are supposed to be chapters. Leanpub just lists the file names, and Prag has some kind of XMLish “include” statement, but same idea.

In each, there are markers for parts, chapters, sectiions. Nominally one writes a bunch of chapter files, each wrapped in suitable XML and the system assembles them and voila! you have a book. Prag also supports some Markdown syntax that you can invoke by a suitable XML thingie that says “Markdown inside here”.

Leanpub basically understands Markdown directly. It is much less capable than Prag.

So I hacked Leanpub by putting just one file name in the Book file, and compiling the whole book, with all its headers and such, into that one file. So the one file has all my chapters, all my images, all in one go. Leanpub’s accepting markdown and the book file index lets me pretty much do everything with compile.

I am not sure whether I can coerce Prag into accepting one big file. One issue is that one’s development editor would like to look at and edit the files one at a time. So I really need to split out the chapters, and ideally, I’d like to generate the Book file from Scrivener’s structure. One possibility is to get close enough and then just cut over to a text editor. There is some OK support for Prag’s language and I won’t need much anyway as the book is mostly words.

So one possibility is to embed enough info in the compiled file to split it up with Ruby. That would be nearly good for output and I could manually import edits back. I really don’t want to do a bunch of programming – it would be a distraction from what I’m supposed to be doing, namely writing.

The other thing I had been thinking was to use export to break out all the files. Each of these would have enough XML in it to do the job. But … one of the things I sometimes do with Scrivener is work below the chapter level, with a card for each paragraph or couple of paragraphs. So …

the Book file would have to say, not just
chapter1.txt
chapter2.txt
chapter3.txt

but …
chapter1.txt
chapter2.txt
subchapter21.txt
subchapter22.txt
chapter3.txt

in order to pick up all the little bits. That would be a pain. So what would be neat would be if you could tell the Export to export level 1 and 2, but to include all level3 files inside the owning level 2. But this is not to be.

So the one way you have to break the big compiled file apart on some markers. Doable, but a digression. The other way you have to roll it back up, or deal with way too much manual creation of book file. Also doable, also a digression.

I suppose I could recruit a tame Ruby programmer so that I could keep my head down on the work.

Anyway, your hook info for post processing this will be useful should I go that way, and makes me think that postprocessing compile is better than preprocessing an export.

Thanks, you’re a star!

9

Quick followup question: are you using Scriv’s built-in multi markdown or is it necessary to install the real thing, in order to use the post processing?

Thanks,

10

Hi (Agile) Ron !
I am considering LeanPub for a technical book (on Haskell), and of course would like to use my beloved Scrivener as a Front end.
As I understood, you already travelled this path.

A Few Questions, if I may :

  • do you have now a smooth process . other references [1] I found are now oldish ( 2 Y old). has th process progressed since ?
  • do you write in MD in scrivener ? ie do you write bold or bold , and let scriverner make some conversions ? same for ref, images , etc
  • do you use tables ? or images ?
  • do you plan to have two outputs, ie one plain for Leanpub, and one fancier for Latex/ Web for a better User Experience ?
  • this one has been more recent . saw that ? [2][3]

1
[2](rightthewriter.wordpress.com/201 … -together/)
[3](natedickson.com/blog/2013/05/30/ … -one-step/)

11

Hi Luc,

I’ve been focusing mostly on my Pragmatic Programmers process so my Leanpub chops are a bit old. However:

The basic process, keeping chapters in separate documents (with possible sub documents), then exporting, seemed to work just fine.

I used Markdown. bold etc. Scrivener does not seem to have the ability to correctly convert bold text to bold and so on.

I use images with the title syntax and it works. If you put them into the file for viewing, I’m not sure they export correctly but I think they do. In my case they were not included in the text I had so I just converted the HTML links to MD. Scriv seems only to allow them to be in the document folder, not in a subfolder, which was what I need for Prag.

I am not planning to use LaTeX / Web at this time, as I’m mostly on the Prag path at the moment and don’t know anything about how I’d do that.

My book has chapter titles and little italicized précis at the beginning of each chapter. I was using the notecard title and contents for those, checking the right boxes. I moved away from that and just put the precis in by hand, but don’t think I had to. I did like working that way, but when composing I like to use the notecard for notes not content.

Keep us posted on your experience. And based on my experience and comments of others, do consider submitting it to Prag.

Good luck!