Documentation for how to create your own Format?

I’m working with a publisher using AsciiDoc. I’d really like to continue to use Scrivener. I’ve already got a custom output format that solves the most basic problems done and working. At this point I need a bit more of the internals documented so I can create something others can use.

The problems I’m facing right now are:

  1. How to create a blank line between paragraphs on the output.

  2. How to split on each chapter in the output (I understand you have this working for iBook)

  3. How to reimport a changed file to Scrivener

As this is going to be really similar to MultiMarkdown, I don’t think this should be a big project. I’m also intending to give it back to you for everyone to use. Just please point me at any docs for creating a Format that I’m not finding in my searches today.

Right now I’ve copied Scrivener.App/Contents/Resources/MultiMarkdown to Resources/AsciiDoc and am crashing around like a bull in a china shop. I’d appreciate any help to focus my efforts better.

Although I realized this was a technical query that wouldn’t get fast response time, I had expected an answer in less than a year :open_mouth: :question:

I have another need that likewise I need more control of formatting than provided – specifically the ability to define a closure on a section. I’m even willing to pay you to work on these issues for me.

On a related note, if you want to know how I got my first O’Reilly book done entirely in Scrivener while delivering to O’Reilly in AsciiDoc, see this github project github.com/jorhett/scrivener-asciidoc

Well, it’s not much of an excuse, and it’s my fault for missing it, but this was posted to the “Wish List” section, which doesn’t get the same level of attention that the tech support sections do, when things get busy.

In case you never found the answers:

In the Transformations pane, there is plain-text option for converting rich text formatting to literal spaces. Thus paragraphs with at least a half-line of spacing between them will get a literal empty line between them.

I do this with a script and a custom separator, when I need it. Since you are using a plain-text output format, it’s easy to just insert “%%%%” or something where a file break should occur, and then write a quick script that slices that out into files. The string itself can be inserted as a Separator—or just typed into the editor where needed.

That is where using MultiMarkdown during the editing phase could help. That has nothing to say about what you type into the editor, the method is purely based upon the outline structure as converted to MMD headings, and then turning those headings back into outline structure on import. The content can be anything, even raw XML—Scrivener doesn’t care.

So the trick is to compile to plain MMD with all “Title” checkboxes enabled in the Formatting pane. Then use File/Import/MultiMarkdown Files… to bring that back in and convert the headings in the file back into Binder outline (with the content spliced back in where it should be). It’s a seamless round-about for MMD users (or those just taking advantage of MMD headings while using some other base format for the text itself).

The main drawback to that method is going to be the same as all export & import based methods: your original outline isn’t actually updated. This can be important depending on how many Scrivener features you use. If you make heavy use of References or Scrivener Links to keep things organised in your head, or keywords and other meta-data—then blowing away your original outline with a content updated outline will probably be more work than it is worth—and simply copying and pasting the updated sections back into the files where they should go (probably with a snapshot prior to doing so), will be more efficient.

Likewise there is the option of automating that entire process with the File/Sync/with External Folder tool. That works if your Binder items are not too small.

I’m not sure what that looks like in your output, but experiment with Separators as well as the advanced title options in the Formatting pane, where you can insert a prefix or suffix—which needn’t really have anything to do with the title, as it can be on its own line. Another approach is to just add it to the main content—perhaps as an individual Binder item that specifically does this one thing, closing off a section, that is set to “Compile As-Is” so it doesn’t get a heading.

Anyway, again, sorry for missing your query when you posted it.

I think we’re talking past each other a bit, so let me clarify.

I’ve used and abused the formatting options to the best of their ability. I need more features that you can get out of formatting, for example if I start a section with or in some cases then I need to close that section with or . Right now formatting has things before the title and after the title, but not at the end of the section.

Yes, I’m doing the trick put a tag in place and split on that tag. It’s inflexible, it sometimes won’t work for where the provider expects a split, etc. If you are already doing file splits for iBook, why not provide this same option for other formats? A simple checkbox to start a new file (identical to the section break) would meet the need.

Finally, I don’t care very much about meta data when working with my publisher. And all the text I use is plain text, fixed font, etc. The delivery method is pure text marked up. But right now, as it stands (and is well documented by others) I can write the book in Scrivener, but as soon as editing starts I have to stop using Scrivener. On the last project we did some major changes in the last 3 months, and not being able to use Scrivener was very frustrating. Hand typing all the changes back into Scrivener (which I did for much of it) was a painfully bad use of my time.

You are touting your ability to work with editors now, but I don’t see how that happens if one cannot reimport the same text for a section with minor changes.

Finally, my need now is for outputting HTMLBook, which is not dissimilar to ePub. I’m willing to pay you to create the format, or for some assistance in creating the compile format. Please don’t blow this off with “look at the formatting options” – if you look at the links I posted and the output I’ve created, you’ll know I’m using and abusing those to the best of my ability.

The current need for HTMLBook output is being handled by this script: github.com/jorhett/scrivener-htmlbook

It gets us a simple chapter split and formatted headers. I need a lot more than basic formatting prefix-only settings.

Mainly because we weren’t setting out to create a multi-file compile feature, but a quick output for iBooks Author specifically—it just turned out that was the best approach for working with that program, so a very simple implentation was put in place, with the idea that if we did implement multi-file splitting in the future, there would be a little work going toward making that broadly useful, first.

I can’t say if we do intend to add that. I know I could certainly use it myself, but ultimately that decision is up to Keith.

For more power than what you get in the Compile feature, I use MMD’s scripting potential. You can optionally download (or extract out of the Scrivener.app package), the MultiMarkdown folder (the one a “bin” and “XSLT” subfolder) and put that in a ~/Library/Application Support/MultiMarkdown folder. Scrivener will check for the existence of this folder and use it instead of the built-in copy, for cases where you’ve set the XSLT flag in the Compatibility compile pane. Hence: a fully scriptable post-processing workflow can be achieved.

For example, setting XSLT usage with the LaTeX option selected will trigger the ‘mmd2tex-xslt’ shell script—which you can alter to do whatever you want (you can basically pretend that the “.tex” option in Scrivener is AsciiDoc instead, at that point). You don’t even have to use XSL, naturally, but there are some examples provided in the XSLT folder. You can drop your own XSLT files in here and reference them by adding an appropriate MMD meta-data key (“LaTeX XSLT: filename” for that).

That’s how I would insert wrappers around sections, with a full scripting engine, rather than trying to get the compiler to do this.

Yeah, and that’s why we often say that the editing phase is about when people should consider stopping use of the Scrivener project and transitioning into other tools that work better in an collaborative environment (I’m not sure where we “tout” otherwise, we are generally very conservative in our summaries of the software, even going so far as to say Scrivener is meant for first drafts, which wouldn’t include any collaborative work with an editor, for sure). Some find ways to get around it, some find editors that use Scrivener, some implement their own round-trip import/export protocols—but ultimately our focus here is on the writing, not addressing each and every component of the production process.

I won’t say that’s the final answer—we actually have a lot of ideas for broadening Scrivener’s scope here, but the ultimate answer here is probably something that will take a little thinking in order to get it right.

I’m happy to provide any pointers you may need, where I can, but we don’t have a service for creating compile formats, ourselves. You may find someone willing to contract, though.

I have and already use for various stuff the MultiMarkDown. I do not see how to effectively write HTMLBook in a way which outputs to MMD (which would lose information) such that I can use MMD tools to render back to HTML. That’s a pretty serious sideways loop.

I want to create something equivalent without using MMD in the middle. Where is the documentation for creating what they did with MMD?

For example, setting XSLT usage with the LaTeX option selected will trigger the ‘mmd2tex-xslt’ shell script—which you can alter to do whatever you want (you can basically pretend that the “.tex” option in Scrivener is AsciiDoc instead, at that point). You don’t even have to use XSL, naturally, but there are some examples provided in the XSLT folder. You can drop your own XSLT files in here and reference them by adding an appropriate MMD meta-data key (“LaTeX XSLT: filename” for that).

That might work 50 years ago… before Macs and before Scrivener. Modern writing is a collaborative exercise.

I love Scrivener. I need to be able to finish the book in it.

I grok that going from one rich text format to another and back (e.g. Scrivener to Word and back) may be difficult – but you’ve already got options to do that. What I’m looking for is pure text. Pure ascii characters and nothing else. This is magnitudes simpler.

As a programmer myself, I fail to understand why you would provide documentation and/or training to someone I would contract with, rather than giving me the same documentation and training you are providing them? If you support me, I’m going to do things that will make Scrivener more useful to a wider audience, getting you more customers.

To clarify, I didn’t mean to suggest actually using MMD to produce HTMLBook, but rather just to use Scrivener’s existing utility of piping text to a shell script and waiting for STDOUT. That’s all this described feature is doing—hence you can do whatever you want in the middle, it doesn’t need to be MMD or XSLT.

I also mentioned it might be useful as a text-based outline approach (similar to an org-mode file) that you can import back into Scrivener as an outline—but like I say, the stuff in between the headings can be whatever you want.

From earlier, I meant to remark on this. I know you don’t want to hear about the Formatting pane, but keep in mind that headings follow sections, and thus the prefix of a heading for a following section can close off an XML element. Thus you could have something like this for a file title:

[code]

{TITLE INJECTED HERE}

[/code]

And remember also that Separators can insert code in useful areas of the document, too. I experimented with some compile settings that added containment elements around text, and if I remember right I had to trim the “ends” in an editor but it worked well.

Oh you mean the actual MMD entries in the Compile For list? We added those ourselves—that’s a hard-coded aspect of Scrivener (that actually goes deeper than just compiling).

If you aren’t using a bunch of meta-data or linking, and plain-text is acceptable—as you mentioned earlier, then what is wrong with the method I described, where you can export a plain-text file with MMD style headings in it, then import that back into Scrivener as a full outline, to replace the prior draft?

The main problem with extending Scrivener into editing has little to do with formatting actually—but rather finding a compromise between a program that is a huge outliner, with the draft cut down into topical sections, vs. an industry that nearly entirely revolves around the use of monolithic documents in word processors (or plain-text editors, but still big files, not 500 three paragraph .txt files). These two working models do not mesh together well.

That’s probably true in some fields, but Scrivener was never targetted at that type of writing to begin with. It was always designed for the solo author working on a first draft. The goal is to produce a document that would be taken into other programs for final work (collaboration, etc.) and submission/publication.

That said I think it has plenty of tools that can be used for managing collaboration, both with other Scrivener users and beyond. We just posted a blog article on the topic, actually. It’s subject is fiction, but the mechanisms themselves have broad usages—some of them may be applicable to the editorial process too.

I think we still might be talking past each other on this point. I thought you were looking for someone to set up your compile settings and maybe put together a post-processing script or two. There is no formal SDK for Scrivener—no way that anyone other than Keith can add a new compile format. Thus, no documentation for creating a new format. The best I can offer is what I already have. A combination of:

  • Formatting Pane
  • Separators
  • Replacements
  • Post-processing via jury rigging the the XSLT mode switch in one of the MMD outputs.

You can do quite a bit with that.

Thanks for posting the workflow on Github by the way. If you don’t mind, I could forward people to it. Everyone once in a while someone asks about DocBook or similar.

That sounds good, so long as I can create something new that is distinct from the MMD as that is something I also use. So docs on how to utilize my own scripts like MMD does… that would be great.

I’m not following that statement at all, sorry. Can I borrow a clue-by-four? :laughing:

This doesn’t work for layered sections. If I am in a chapter and I start a second level section, I don’t want to close the previous section. However if the previous section is the same level or a higher level, then I do…

[code]
Chapter intro text

First section text. Second section text. next chapter...[/code] [quote="AmberV"] And remember also that Separators can insert code in useful areas of the document, too. I experimented with some compile settings that added containment elements around text, and if I remember right I had to trim the “ends” in an editor but it worked well. [/quote] Again, clue by four would be great. In particular, the ability to wrap paragraphs in

...

tags would be incredibly useful. Right now I'm having to hand-type all the interior html. [quote] Where is the documentation for creating what they did with MMD? [/quote]

Well, I’d really like to do something similar for HTMLBook. I’m willing to write all the processing scripts. That’s the kind of help I’m looking for.

Except the fact that it’s not MMD format. And I’d like to see and approve differences if possible.

Well in this case my output is likely to be 50 text files. It’s not impossible to make it 500 files, but O’Reilly prefers one file per chapter. I think what you have and what they have is a lot closer than the monolithic Word problem.

In this scenario there is no other program other than text file editors. This opportunity is the chance for Scrivener to become O’Reilly’s preferred and recommended writing tool.

You really need to look at my github links :laughing: I’ve already got a custom compile setting and post-processing scripts. I’m trying to make it work better.

From this answer, it sounds like I want a new XSLT switch for something without having to gut the MMD output, and some documentation for what will be passed to those scripts.

Sure. I’m hoping that this conversation will allow me to make it work for more use cases, as right now you have to do a lot of things Just So…

Amber, I am trying to work on this path and I think your description here leaves a lot unsaid. Can you please provide some clear documentation on how this works?

You said

I see no such flag, only one checkbox for MultiMarkDown 2 compatibility. Does this enable XSLT mode?

Is this required for replacing MMD? If I don’t enable this, what is run?

I’ve now lost about 4 hours on this, and I’m just running into a crazy maze of misinformation spread across 8 years of Scrivener and MMD support.

First of all, none of the MMD-Support packages provide the same directory structure used in Scrivener 2.6. All of the stuff from MultiMarkDown/bin is in xslt-bin/ in your version. Is using the MMD-Support package really going to work? Does scrivener really figure this out?

Second, the really key thing missing here is the most basic map. If I use X compile choice, which script/s is/are run? This would be really key to hacking up the support.

Okay, the more I look at this, and compare against the HTML output, the more I think I’ve wasted many hours.

I’m back to my previous point. There’s a standard for HTMLBook documented at oreillymedia.github.io/HTMLBook/

How much would you charge me to create this as an output option? This would actually be a much simplified version of your current HTML output. You’d have to do much less.

If you won’t do it, is the source you publish complete enough for me to create a patch and submit it back to you for this?

As noted, you need to select one of the formats that MMD itself has an alternate XSLT post-processing route for: either LaTeX or HTML.

If you want it to run straight out of Scrivener when you punch the Compile button, sure this is required. If you do not enable the XSLT flag, then Scrivener uses the normal MMD binaries to produce the output file, rather than a script that initiates the use of XSL processing, and there is no way to step into that process and customise it (by design, that is just how MMD works). It is with the XSLT option that you have a shell script being executed by Scrivener, complete with a STDIN dump of the compiled text. You should be able to take it from there with a modicum of shell scripting knowledge.

You mean inside the software .app package itself? That isn’t a good reference as we have combined both the binary installation and the scripting support folder into one place. This isn’t meant to be used as a working folder or example of anything, it’s merely an internal fall-back copy of the entire MMD subsystem so that the user need not have anything external installed.

As mentioned earlier, the script ‘mmd2tex-xslt’ is used when selecting LaTeX and enabling XSLT post-processing. But if you need a second one, the only other ‘-xslt’ file in the bin folder is the alternate for the HTML selection in Scrivener.

I’m not sure if it would be as easy as you think. Right now we call a Cocoa method, requesting that the rich text document be converted to HTML. Scrivener then takes the HTML data that the appropriate Mac framework produces and saves that data into a file as specified by the user. So, anything we do beyond that is always going to be more, not less, and by beyond that, I don’t mean much more than what amounts to “search and replace” on the output HTML data to coerce it where possible to do so. Keith does a little of that for ePub/Mobi outputs and a few other things, but this isn’t a very good method for large-scale transformations.

What do you mean by charging you to create an output option, though?

We don’t publish any source at all. Are you thinking of modifying MultiMarkdown (which is open source)? If so, that project is on Github, if you have a dev account, feel free to fork it, or develop your format and submit it to Fletcher to see if he’ll include it.

Sad face :frowning:

Given how much I use this, it may be worthwhile for me to pay you to create the HTMLBook output mechanism rather than keep developing my script to convert it. Furthermore, it may be possible to get O’Reilly to pay or co-pay for that since it would be very useful for them.

My bad, I thought I saw you recommending someone to look at the Linux source in another thread.

Yeah, I agree with the sad face. It would be really awesome if the text engine made it easier to modify how its conversion routines work. I suppose a small consolation is that we’re working on some new features for the future that may help you out here, indirectly, via the ability to inject wrapper text around selected ranges during compile.

Have you looked into Pandoc at all? It can take a MultiMarkdown document and convert it to AsciiDoc or DocBook.

We don’t really have a format-for-hire approach though. If there is a huge demand for a format, Keith will do it, but we’re talking .epub levels of demand, which HTMLDoc is most decidedly not in the same league as. :slight_smile: This would be in particular difficult, however, since the base engine is incapable of producing HTML5 (main reason we don’t support ePub3).

Pandoc needs Markdown input. Given the need for me to use HTML elements directly that don’t exist in Markdown and other elements that provide much greater specificity – remember, HTMLBook is for final form publishing – that’s not going to work for me.

I am totally okay with having to type much of HTMLBook syntax directly into Scrivener. I’d just like to see decent support for basic conversions == italics, bold, preformat -> code blocks == the exact same stuff you do for Markdown internally right now. 1:1 ratio of requirements, just a slightly different output.

Like, if you gave me the code which did Markdown I could produce code which does a decent wrapper for HTMLBook.

But for here and now: I realized that perhaps your “Replacements” functionality could give me much of what I’m looking for. However it seems to be very underdocumented. The default Replacements for text output imply that perhaps there’s a way to mark up figures that were inserted, but there’s nothing in the manual explaining this.

If you take a look at the attached screenshot – is that Replacement match on the left going to match whatever syntax is in your file if I were to insert an image inline? Likewise can I use the second line to convert a Scrivener table to an HTML table?

If that is what it seems, could you please provide a list of the Scrivener internal syntaxes for various things like bold, italics, underline, preserve formatting, etc such that I can use replacements to output the right syntax for them?

Also note that the manual repeated refers to using $n and $R for auto-numbering but doesn’t document anywhere what variables are available and what they mean. It’s just tossed out there, you’re left to try out and see what you get. Definitions of these variables would greatly improve the manual.
Screen Shot 2014-12-08 at 4.02.35 PM.png

FYI I know my requests are pretty demanding, but know that if I am successful you’ll be the only writing tool in the market to support HTMLBook and there will be lots of advantage for you.

Have you tried blending the two together? The original Markdown, and both Pandoc and MultiMarkdown are designed to allow for HTML to be blended with the Markdown stuff. I’m not familiar enough with HTMLBook to really test it out myself. I’ve found very little on the ’net about HTMLBook.

No, these are just simple counters, actually. All of the placeholder counters are described in the Help menu, under “Placeholder Tag List…”. I suspect most of them would not be terribly useful to you however. I’m assuming your system has some sort of mechanism for automatically numbering captions, headings and other such things? These counters are more useful to those working in rich text.

On images, you could actually do what you were thinking of, however. Scrivener has an image code syntax that you can use to refer to images in text. It’s very simple, and documented in 15.5.4, Image Placeholder Tags, pg. 220. There are definitely some neat things you can do with that concept.

I’m not sure where you were looking, but flip to page 399 in the user manual. In particular you may be more interested in 24.18.1, on page 402, where advanced usages such as the one you see with the figure and table numbering, are described. If you want to see some real-world uses of Replacements, check out the Scapple documentation project. That is what I use to compile the Scapple PDF. Check in the Replacements pane, under the preset tab.

If you google it, the first response (result also of “I Feel Lucky”) is the definition of the standard: oreillymedia.github.io/HTMLBook/

Not helpful here. The images have to be relative to the book’s GitHub repo, which means nothing for Scrivener. It might allow me to compile with the images on my own, but there’s no point since it’s better if I use O’Reilly’s build system.

This would be very useful if you’d provide the internal markers for italic, bold, etc such that I could use a pattern like “(bold)$@(/bold)” and replace it with “$@

In addition to markdown input, Pandoc also reads (subsets of) Textile, reStructuredText, HTML, LaTeX, MediaWiki markup, Haddock markup, OPML, Emacs Org-mode and DocBook (at this time of writing).

Pandoc can write (at this time of writing) plain text, markdown, reStructuredText, XHTML, HTML 5, LaTeX (including beamer slide shows), ConTeXt, RTF, OPML, DocBook, OpenDocument, ODT, Word docx, GNU Texinfo, MediaWiki markup, EPUB (v2 or v3), FictionBook2, Textile, groff man pages, Emacs Org-Mode, AsciiDoc, InDesign ICML, and Slidy, Slideous, DZSlides, reveal.js or S5 HTML slide shows.

I use Scrivener to compile to web page (.html), with XHTML 1.0 Strict set as the HTML document type in the Import/Export pane of the Scrivener Preferences. I make sure to apply HTML style header numbers to all the heading levels of my document in the Formatting pane of Scrivener’s Compile window, and to apply whatever other HTML settings I need in Scrivener’s Compile window. After compiling, I feed the HTML output from Scrivener to Pandoc for whatever other formats I need. It works beautifully.

I do not use the HTMLBook schema, but I am curious to know how other authors who use the HTMLBook schema produce their markup: whether all authors produce their HTMLBook markup by hand or whether they use some software tool.