Scrivener -> Sigil/Calibre Workflow Recommendations

[NorbVor]

I am trying to find my workflow for using Scrivener with Sigil (or Caliber Epub Editor) as post-processing for a rather technical e-book. As some people here seem to use that as well, I would like to hear your recommendations and practical tips with regard to the following questions:

• Is it possible to use a repetitive Scriv → Sigil cycle? Meaning: Does everybody just finish your book completely and then import it to Sigil and post-process just once. Or is there a way to go through that cycle repetitively and in order to import also intermediate stages of a book? I am thinking along the line of doing a first import, doing all the settings and CSS stuff. Then for any further runs to delete the chapter files from the Sigil project and re-import the current ones from Scrivener. Is that a viable way of operating?

• How do you normally import from Scrivener? Do you run a compile in Scrivener with EPUB output then open that in Sigil and change whatever needs to be changed? Or is there a better way, perhaps “MultiMarkdown → Webpage (.html)”. Or is File → Export Files → .html the best way to transfer files to the Sigil project?

• Do you split up the imported file in Sigil into chapters again or just leave it as a single file?

• What is your preference: Sigil or Calibre? I have found the Calibre editor easier when it comes to make corrections. You can type away in the Code window while the preview is being updated in a separate thread. The Sigil editor just hangs for a few seconds while updating the preview – apparently doing it in the UI thread of the application. I don’t want to start a big Sigil vs. Calibre discussion here, understanding that there are tons of reasons to use one or the other. Just wanted hear how many of you use what.

I am planning to use Scrivener styles extensively for some special paragraph and character formats and also probably some special markdown. Still experimenting with those. Any hints here highly appreciated.

I have been trying to learn as much as possible from the Scrivener manual, chapter 24. Unfortunately, there is very little about possible workflows together with Sigil or LaTex. Perhaps a separate chapter about post-processing would be appreciated (-> @AmberV ?). As a relatively new Scrivener user I think that Scrivener is a great tool for organizing your content, but pretty much leaves you alone when it comes to generating professional quality output. While understanding that Scrivener doesn’t try to rival professional layout tools, I would like to see the manual (and tutorial) point out some paths of how to arrive at a professional end product, print and e-book.

Thanks for recommendations and tips!

[philjaq]

I’m usually writing non-fiction stuff of a technical nature where page layout plays an important part in helping information transfer. I haven’t found Scrivener very helpful for anything concerning page layout. I’ve been repeatedly frustrated trying to achieve my own paragraph styles and nested mixes of unordered and ordered lists don’t go down at all well. (To be honest, they often need a bit of lubrication to pass inspection within epub as well).

My workflow has settled around using the excellent features in Scrivener for gathering research materials together, doing my early plotting/outlining and writing a first (or even second) draft. There are some features that could be improved, in my opinion – for example, image files placed into the research folder don’t display their complete filename so I can’t tell by glancing whether a file is a png or a jpg and sometimes that’s important for me.

When I need to get to page layout, I export to LibreOffice Writer where I use my prepared style templates to get things into order. I usually try to publish paper versions at the same time as ebook versions so the page layout is important to me. I did once spend a lot of time trying multiple setups of Scrivener’s compiler but I never did master it in a way that gave me a satisfactory layout. So I just do a simple compile that the word processor can read.

When the paper book version layouts are finalised, I use Sigil to prepare the epub version taking advantage of a plugin. I ran a few trials some time ago to see whether I liked Calibre or Sigil better for the conversion from Writer output to epub and decided that Sigil caused me a little less cleaning up. I have my own CSS style sheets but in any case, some cleaning up is required. It’s a matter of personal preference but I find more unwanted “spans” in Calibre’s output than in Sigil’s.

I would definitely break a large single file into several chapters - it improves user experience with faster loading in the ebook reader. On occasion I have found chapters split into smaller fragments and that too is easily corrected in Sigil.

Once I have my project out of Scrivener, I don’t go back for revisions because by then I have all the work in the word processor with full Wysiwyg and any updates are carried out there. I can copy and paste any modified text directly into Sigil and adjust any code as needed. Recently, I had occasion to change all the illustrations in an earlier ebook from pngs to gifs - the advantage was a much smaller file size and substantially reduced delivery charges by Amazon. This only concerned the ebook version and so I carried out the mods directly in Sigil without changing the paperback master document files.

That, in outline, is my work flow.

[NorbVor]

Thank you @philjaq! I highly appreciate your detailed description and think that I might finally end up with a similar workflow. Too bad that this intermediate step through LibreOffice is necessary for layout purposes. I had hoped that Scrivener was the all-in-solution. Perhaps it will get there over the years.

[devinganger]

@NorbVor

Perhaps it will get there over the years.

KB doesn’t seem to feel the need for Scrivener to be an all-in-one tool, so I would not count on it. Layout programs are complex software.

[NorbVor]

You are probably right; I won’t hold my breath for it. But while paper layout is really complex, generating “professionally viable” HTML for an epub is by far simpler. And that should actually belong into the realm of Scrivener. After all, what are all the nice research and ordering features good for if I finally have to step over to a different tool for layout and production and can never go back to my Scrivener project, because I have made so many changes in the layout stages. That’s like saying to a programmer: You may compile your program once. But as soon as you have run the debugger, you may never return to your source code again.

[AmberV]

You are probably right; I won’t hold my breath for it. But while paper layout is really complex, generating “professionally viable” HTML for an epub is by far simpler. And that should actually belong into the realm of Scrivener.

I certainly agree with that, and have said as much in the past.

“The only main exception is its ebook output system, which is probably right up there in terms of what other tools are capable of. You have full control over the CSS, and with a knowledge of how content is produced, deep control over the HTML structure itself. One could do just about anything they want for an ebook, rivalling even lower-level tools like Sigil, or assembling book manifest content by hand in a coding editor.”

The important caveat here is that what I described there is not yet in place in the Windows version, as you’ve noticed yourself. You can’t for instance insert a block level element correctly because it assumes you want a <p> around everything.

So for the moment, you’ll hit a brick wall trying to use Scrivener like a word processor if you want detailed ebook control. There should be settings which remove most (all if you’re careful) forced styling, giving you full control. At the moment no combination of settings will remove serialised classes, or worse, hard-coded inline CSS in the elements themselves.

As for whether “everybody” takes their ebook into Sigil/Calibre after compiling—I highly doubt that. While there are issues with the more advanced approaches to using the compiler, it was pushed to launch for the reason that it’s basic “bold is bold” level formatting works fine for most.

That’s like saying to a programmer: You may compiler your program once. But as soon as you have run the debugger, you may never return to your source code again.

If you’re going to make a comparison like that, then you need to use like methods for the production of books: where the product is one layer away from the source, like source ⇒ software is. I.e. don’t use Scrivener as a tool that speaks to editors (word processing files), but as a tool that speaks to publishing tools (LaTeX, Pandoc conversion to DocBook, InDesign, etc.).

If you want to use Scrivener like a programmer would, where you can compile and get straight to the end product, you need to compile to something other than what is intended for a different part of the publication process entirely.

So to come back to what you can do, the good news is that Scrivener for Windows can produce super clean HTML with total control over the structural and formatting aspects of design. But to get there, you’re going to need to install Pandoc to gain access to its Pandoc ⇒ ePub integration. Whether that means adopting Markdown from top down, or making use of Scrivener’s conversion tools for producing Markdown from rich text conventions is up to you. Each has their own learning curve. For myself, I prefer to write using Markdown itself, double-spaced paragraphs and all, but that isn’t a necessity with Scrivener.

Learning that process will be in part learning how Scrivener integrates (that part is relatively simple), but also how Pandoc produces ebooks, external to what Scrivener does.

The ideal here, in my opinion, is front-loading everything, or as much as possible, into the production process (compile) and reducing post work in Sigil to zero. As intimated above, that should be possible with all forms of using Scrivener, once these issues are sorted out—but right now it is only possible with Markdown. And honestly if clean HTML is your grail, that should almost always be on the table for consideration, as it takes a lot of work to get clean fully-semantic HTML out of an RTF converter, and will probably never be perfect.

I have been trying to learn as much as possible from the Scrivener manual, chapter 24. Unfortunately, there is very little about possible workflows together with Sigil or LaTex. Perhaps a separate chapter about post-processing would be appreciated…

Whew, yeah, honestly that would be a short book in and of itself. As I often say, the documentation is long enough just describing what the software does, let alone the diverse ways in which you can take what it does and form workflows out of it. It’s a common documentation problem with toolkit-based software like Scrivener, where you are given tools to create workflows out of smaller integrated tools. The permutations of how you use them become endless, in part because the wide variety of things we need to do is essentially endless. Software that doesn’t give you that ability is easier to document and learn, but then if you need to do something it’s little carefully fenced in features provide, you’re stuck.

[philjaq]

@AmberV

So to come back to what you can do, the good news is that Scrivener for Windows can produce super clean HTML with total control over the structural and formatting aspects of design. But to get there, you’re going to need to install Pandoc to gain access to its Pandoc ⇒ ePub integration. Whether that means adopting Markdown from top down, or making use of Scrivener’s conversion tools for producing Markdown from rich text conventions is up to you. Each has their own learning curve. For myself, I prefer to write using Markdown itself, double-spaced paragraphs and all, but that isn’t a necessity with Scrivener.

Learning that process will be in part learning how Scrivener integrates (that part is relatively simple), but also how Pandoc produces ebooks, external to what Scrivener does.

The ideal here, in my opinion, is front-loading everything, or as much as possible, into the production process (compile) and reducing post work in Sigil to zero. As intimated above, that should be possible with all forms of using Scrivener, once these issues are sorted out—but right now it is only possible with Markdown. And honestly if clean HTML is your grail, that should almost always be on the table for consideration, as it takes a lot of work to get clean fully-semantic HTML out of an RTF converter, and will probably never be perfect.

A super interesting reply, AmberV . I’m always open to new ideas on workflow and I do like super clean efficient html code. I’m going to have to look into Pandoc, Markdown and their use and integration with Scrivener.

My concern is whether the investment in time will pay off later in an improved workflow. At present, it is not onerous to do the page layout in a word-processor because my requirements do not require DTP precision. Furthermore, I have already a certain investment in Sigil (and Calibre) which means that I can handle any cleanup or reconfiguration with little effort.

Your ideas do present me with a challenge, though and I shall look into them.