S2: Structural Writing Solution

I have been giving a lot of thought to structural definition within Scrivener. As you know, I am a proponent of structure over style. When given the choice between using a mark to define a type of text, over a font change to describe the type of text, I will take the mark any day. This is one of the primary abilities that drew me to Ulysses, back when I first stumbled across it. It provides a very simple way (for the most part) to mark up your text document with structure, which can then during the export phase, be defined as style.

While I enjoy that ability, I am not completely convinced that the only format a writer needs is text only. Font styles can indeed be useful. I find use for styles as another layer of meta-data, in a similar vein to how Ted G. of About This Particular Outliner fame, supports the usage of named styles. In this school of thought, style then becomes a marker for visually expressed meaning; meaning which does not need to be conveyed at the publishing phase. A simple example would be commentation, and types of commentation within that category. Scrivener’s use of adornment and colour to set apart commentation and citation is a good example of an automated “named style” system. One could also argue that the highlighter tool is another example of meta-data styling. Beyond the built-in tools, Scrivener’s use of the RTF format allows further use of this system in an informal sense, as the author sees need for it. Ulysses lacks this ability completely, having opted for plain text.

Where Ulysses lacks an infinite style potential, it gains in its fairly strong structural export model. Paragraph level mark-up can be transformed into a number of different types, such as headers, lists, block quotes, or deleted entirely; treated as comments. Character level mark-up can be ignored, or transformed into font based styles. Conversely, where Scrivener has an open book on style, the exporter has zero awareness of internal document structure, opting to merely pass format rules to the export engine, where it can be converted into various formats. Even more detrimental, it is impossible to discard what may be wholly irrelevant styling, reducing the effectiveness of its infinite styling potential.

Back in the SG days, I campaigned a little for some form of structural support, with little luck. And lacking any good method (short of a massive departure from Scrivener’s philosophy and a complete “from the foundations” design of the exporter), I let it drop. But just the other day I had an idea. Something that would actually be very easy for you to implement, which would allow structural writers and format writers to coexist. You would not even have to research XHTML, let along LaTeX! Here is the idea, and its ancillary features:

In a word: Markdown. For those not familiar, Markdown (or Textile), is a drop-dead simple method for creating full mark-up from a text file which has had minimal structure applied to it, using (mostly) standard conventions for ASCII embellishment that have existed for decades. Anyone that has been sending email before the advent of HTML, recalls doing simple things like putting asterisks around things you wish to emphasise. Markdown is no more complicated than that. I can make a chapter title like this:

Markdown Usage ==============

Or, if I preferred to take a more flexible approach, like this:

# Markdown Usage #

So, how could Markdown integrate with Scrivener? There are several issues to be considered. The key issue with integration is that documents which have been structured with style changes have to go through a different exporter than documents which have been styled with Markdown. The documents which go through the Markdown fork will have all of their font styling completely ignored. This fully restores Scrivener’s infinite meta-data potential, so you certainly would not want to pass standard documents through that layer. On the other hand, the method for conversion that Keith is using for exporting is somewhat of a black box, so he could not (without a ton of work) hack a Markdown filter into the standard exporters, and besides, integrating both philosophies into one exporter would reduce the potential of a structurally designed document.

Alternative 1) A battery of Markdown based exporters is added to the export dialogue. Thus, if you used Markdown to structure your novel, you would choose “Markdown to RTF” in the exporter, or “Markdown to XHTML,” whatever. This method would result in a very long, and potentially confusing export format list.

Alternative 2) Rather than make individual choices to the export format, leave that list the way it sits, and instead have a checkbox in the export dialogue which informs the exporter to use the Markdown versions of these exporters. When this checkbox is clicked, the export engines are silently switched behind the scenes, but the list of choices remain the same. It is much less confusing for the user. However, there is a flaw with this and alternative 1: It forces the user to choose between philosophies for every single document in Scrivener that will be exported. This would, even for a structural fiend such as myself, be restricting. While I am a big supporter of structural for most types of documents, there are certainly cases where I do not see any need for it at all, and simple font based styling is all I want. If the project were toggled one way or the other, I would have to sacrifice flexibility.

Alternative 3) While this would require a bit more work on Keith’s end, it would be the most flexible and useful option. In this scenario, a checkbox is added to each document in the Meta-data section of the Inspector, right above the date stamp. This checkbox would enable Markdown formatting for that document. Optionally, as a time-saver, each document could reference its parent, and inherit the checkbox automatically. This could of course be over-ridden. Then, during the export phase, the export system would check for that flag, and if it was enabled, that document would be passed through the Markdown version as opposed to the standard version. Say the user selected RTF format, they would end up with a directory full of RTFs, completely ignorant of which system it went through by that point. The except would be plain text export, which would not do anything with Markdown (the syntax would be preserved).

The particulars of the Markdown export engine are quite simple. It would require two functions (I believe). The first would convert the working stream to plain text, and in the process convert annotations and footnotes (and potentially links) into appropriate Markdown syntax or plain text alternatives (such as how brackets are used, though brackets have special meaning in Markdown). Layer two would make use of MultiMarkdown.pl, which allows a variety of export types. Standard Markdown only converts your text to XHTML, but with MultiMarkdown, we can get functional LaTeX, and RTF documents.

To handle LaTeX, which can only be done with Markdown and not the RTFD style documents in Scrivener, perhaps a checkbox option in the export dialogue to over-ride the export format choice to LaTeX for all Markdown based documents in the export. That way, you can choose a format for basic export.

The actual export process would basically just be piping the documents through the shell commands, very little development would be required.

That is basically it. So what do you think? This could all be made very simple for the user. Versions of the Markdown Perl scripts could be distributed inside of Scrivener. See TextMate for examples. TextMate has a full Markdown bundle, with a bundled version of it and MultiMarkdown, some syntax aids, and a full range of export options. The Bundle editor reveals the syntax used to create these various exporters.

Hmm… Sounds a lot more work than it might seem. :slight_smile: If you could save me some time on research and give me some links to markdown and MultiMarkDown with some very short documentation, I’ll certainly take a look. No promises, though, especially for 1.0…

Sure, I can put some links together. And yes, I did title the thread S2 for a reason. :slight_smile:

Author’s website for Markdown. The “Basic” link has a quick overview of Markdown’s syntax. This page also has SmartyPants, which comes in useful with MultiMarkdown, to create typographically correct punctuation.

MultiMarkdown. Brief explanation of the project, and documentation on how to use it. The primary reasons for using Multi instead of standard is the range of documents that can be produced from it (rather than just XHTML), and it also has support for footnotes. Something that would be important for Scrivener. This document was created using MultiMarkdown, and has a download link to it, and the OmniOutliner file that was used to originally create it. Lastly, a link to the XSLTs used below.

Here are the shell scripts that TextMate uses to create all of these formats from MultiMarkdown. The $TM_ variables are TextMate specific, used to grab data from the open file and such:

RTF

[code]# first figure out a name for the result
NAME="${TM_FILENAME:-untitled}"
BASENAME="${NAME%.*}"
DST="/tmp/$BASENAME"

MultiMarkdown.pl|SmartyPants.pl >"$DST.html"

textutil is included with Tiger

require_cmd textutil
textutil -convert rtf -extension rtf -font Times -fontsize 11 “$DST.html”

open -a TextEdit.app “$DST.rtf”[/code]

XHTML

MultiMarkdown.pl|SmartyPants.pl

LaTeX

cd "$TM_BUNDLE_SUPPORT" MultiMarkdown.pl|SmartyPants.pl|xsltproc -novalid -nonet "$TM_BUNDLE_SUPPORT/xhtml2article.xslt" -

And, there is also one for PDF

[code]# first figure out a name for the result
NAME="${TM_FILENAME:-untitled}"
BASENAME="${NAME%.*}"
DST="/tmp/$BASENAME"

MultiMarkdown.pl|SmartyPants.pl|"${TM_RUBY:-ruby}" >"$DST.html" -KU -pe ‘$_.gsub!(/[^\x00-\x7F]/) { |ch| “&##{ch.unpack(“U”)[0]};” }’

show the generated HTML file

cat “$DST.html”

convert to PDF with htmldoc and open in Preview

require_cmd htmldoc
htmldoc -f “$DST.pdf” --header “” --footer “” --webpage “$DST.html”
open -a Preview “$DST.pdf”[/code]

Thanks - I will look into this as soon as time permits. Appreciated.

Here’s what I’m worried about:

If I did implement a MarkDown export, then surely the user would expect RTF tables to be converted to MarkDown tables, Scrivener footnotes to be converted to MarkDown footnotes and so on. Converting tables would be impossible, as currently Apple’s RTF table system is largely opaque. When writing a MarkDown document, you would have to adopt to the idea that although you could apply footnote and annotation ranges, and although you could add RTF tables, all of these would be lost in favour of MarkDown formatting. Now, you might say that this is fine; but what about other users who start using Scrivener further down the line. I fear that I would be setting myself up for user-demands for large amounts of work that would be taking me in a direction I don’t really want to go.

I will continue exploring MarkDown (it is a nice syntax); but these are my worries…

del

?

I hope there isn’t that much to do. The current beta is working up towards 1.0, and I hope to get there before Christmas. And I have long said that after 1.0 there will be a development lull (aside from bug-fixes) whilst I use Scrivener for what I built it for in the first place - my Great Novel ™. Once I’ve beat that into shape, I’ll start thinking about Scrivener features again. In the long-term, this should benefit everybody - Scrivener will be a lot better if its developer has had a chance to use it for a long period of time.

So, it is probably best that I review most suggestions now, before the post-1.0 lull. “Review” only, mind; I’ll only start work on anything like this if it seems achievable in a short period of time. Obviously beta 2 takes precedence over everything else at the moment. So - don’t be worried! :slight_smile:

You raise some valid concerns, but perhaps with proper introduction and warning, these can be somewhat avoided. Whatever route taken would require careful thought. Something like this could easily come off feeling like a hack. For the rest of this response, I am just thinking out loud.

There are two things in favour: The first is that it could be considered a relatively advanced feature. The only place where it is visible is in the Meta-data Inspector, and it must be turned on manually to function. So, a person would have to know roughly what it is, to expect anything of it. And one who knows roughly what it is, will know that it is a text based syntax for creating a document structure, and will be willing to use that, most likely for the benefit of getting LaTeX or high quality XHTML output in the end. Not the average user, in other words.

The documentation would just have to be very clear that when that flag is on, all RTF formatting will be lost, including pictures, and tables, and highlights. At least, certainly, for the initial implementation.

As for the Scrivener specific features such as footnotes and annotations, I had been thinking that those would be fairly simple to tweak, since you have already got them converting into a “flattened” format right now, so they are getting intercepted already. Perhaps I was mistaken. Making a footnote in Markdown is as simple as this:

[code]…the range of documents that can be produced from it (rather than just XHTML), and it also has support for footnotes[^1]. Something that would be important for Scrivener.

[^1]: Contents of the footnote.[/code]

The pointer is the only part that is location important. The actual footnote can be placed anywhere convenient. Footnote IDs need to be unique. Internal document number and linear counter should suffice for that. 08-01, 08-02, et cetera.

How about this for user orientation. There is a proper introduction and warning in the tutorial, and a brief explanation of what its purpose is. There could be a tiny question mark button next to the checkbox item, itself. If you click that, it loads a page in Scrivener help that has a syntax cheat sheet, and another warning at the top. And lastly, when the user checks the Markdown box, a warning sheet pops up stating that all RTF formatting and the listed impossible features will be ignored upon exporting, blah blah, with an “Okay” button and a little “Never show this warning again” checkbox.

Actually, yes, and I wrote an RTF exporter using Tinderbox templates, too. But if you are wondering if Keith will have to muck about in that, he won’t. The provided scripts take care of all the dirty work. He would only have to use UNIX pipes to get converted data. I think the biggest challenge with this is an intuitive interface, but I might be overlooking something. :slight_smile:

Here is a sticky situation:

User wants a big LaTeX file, so they go to export. Only problem is, part of their draft was not written in Markdown. What happens to those sections? Currently the exporter produces one big file, will these sections just get forced into text only mode and processed as Markdown anyway? Dropped? Warned about and asked on an individual basis with a “Make this decision for all” checkbox?

The export process would have to go:

  1. Verify Markdown flag OR user selected force
  2. Pre-process text to Markdown compliant footnotes & annotations
  3. Combine all documents
  4. Add necessary MultiMarkdown headers to top (“Format: complete”)
  5. Pipe result through requested filters and shell scripts
  6. Write file

I actually like the mark-down idea. I still use // for italics and :: for bold in my Scrivener writing — it’s the easiest way to mark formatting permanently. I’ve lost a lot of formatting moving my text between different programs, but this way I’ll never lose it. The basic set of mark-ups is enough for me.

The only problem is the Scrivener export… I sort of think I may have to copy-paste my text to Ulysses and then export it from there.

Well, Scrivener will always be a “rich text” editor where as Ulysses is plain text. If Ulysses had been rich text, I probably would never have even started in on Scrivener, I don’t know.

I have to admit that I don’t really understand the problem some people have with rich text, or why losing formatting would ever be a problem. You only lose formatting like bold or italic if you move your text to an app that is plain-text only. RTF is nearly as well-established as plain text these days. It is a cross-platform standard, and I do not know of any word processor that does not support it. I can’t see why the average writer would ever choose a mark-up syntax, which requires extra work and looks a lot more distracting on the screen, over rich text. Personally, I like to see how my text will look on the page, which is a major advantage of rich text.

This is not to say that I’m not interested in adding the feature - I will look into it - merely that I have never quite understood the need, especially for writers of prose. Perhaps someone could enlighten me. :slight_smile:

Well, preference certainly comes into the equation. I actually prefer a syntax look to a pseudo-page look. Depending on the font, both bold and italic can often be hard to spot, whereas their syntax counterparts are easy to spot. To me, they are not distracting. Using something like HTML or LaTeX would be, sure, but something flexible and simple like Markdown? There is no more clutter on the average page than punctuation provides, yet most people are utterly used to subconsciously processing grammar mark-up like quotation marks, question marks, and so on.

Portability is a little concerning. Actually, there are a number of problems with RTF and portability. We’ve already identified a number of issues with footnotes and comments, just amongst different applications on a single operating system. There are image issues between systems. Links. No two word processors use proper styles the same way. Even the way fonts are declared changes between platforms.

That is just part of the portability coin, however. When you use RTF, you are discarding thousands of utilities which can be useful, all based on manipulating or analysing plain text. How about:

grep -r !EXPAND project/* > expand_me.txt

—to find all of the spots in all of the files of your project that have been marked as needing important expansion. Using these tools, bulk changes can be made to documents. Say I’m done expanding, I can run a quick multiple file search and replace that switches it to “_EXPAND.” If you use text based links, you can actually search for links. You can manipulate them in bulk.

But more important to me than portability is flexibility. I like having one single document that can be printed as a manuscript, published as a web page, or typeset into a beautiful looking PDF for leisure reading. Font choice, heading styles, whether or not to honour certain types of the structure, and so forth can all be changed and decided in one central spot. A 300 page book, filled with elaborate structure can be completely mutated into a new look in a matter of minutes. Yes, there are RTF to HTML converters, but they do a horrible job of it generally, because they lack a central style. Each paragraph is styled individually. There is no true heading structure, just font sizes.

And LaTeX. Nothing I own comes even remotely close to the level of quality that typesetting engine puts out. Beautiful pages with subtle attention to detail.

Wow, I feel humbled. My needs are so much simpler than I thought! I like rtf just fine and have no significant compatibility issues. As long as I stick with certain fonts, I never lose formatting. I personally would hate to have to deal with mark-ups—to me it would slow me down and feel cumbersome.

BUT, this dicussion was helpful to me in understanding why anyone would want to deal with all of that. So thanks. I suppose it could be useful to address it in the future for those that have the need. After Keith has had a chance to write that novel, of course!!!

Alexandria

That right there is the ideal reached with this system. Unlike Ulysses, where you are forced to use simple mark-up to describe style, Scrivener would have the choice, and if you never wanted or needed structure in your documents, that little checkmark could remain forever unflagged.

I do like the whole idea of MarkDown as an option quite a lot; it deals with the numerous LaTeX requests I get quite nicely. I’ll be honest, though: the main obstacle to my implementing this at the moment is my own lack-of-familiarity with running shell-scripts and suchlike. I’ve looked into it and sort of know what Cocoa classes I’d have to use to put a wrapper around the shell scripts. But that doesn’t help if I don’t know how to run the shell scripts in the first place!

So… where did you find those shell scripts in Mellel? What do you save them as? How would you go about running those shell scripts on the command-line, with no GUI app running them? If I can get these running and working on the terminal, I can probably get it going in Scrivener. I know it may sound funny that I can program something as big as Scrivener but don’t have a clue about how to run a shell script, but hey. :slight_smile:

We are opposites on that score. :slight_smile: I’ll make the shell scripts. It would take me maybe five minutes to do this. Just let me know what formats you want to support.

Very kind of you, thanks. As for formats - all the basics, I guess… RTF, LaTeX, HTML, XHTML, PDF - same as the ones you posted, and anything you can think of that would be useful.

Where did you find the TextMate scripts? Presumably they are somewhere inside teh TextMate.app bundle, but I couldn’t find them.

Thanks!
Keith

P.S. If you could tell me how actually to use the scripts as standalone before I try to do anything else, that would be very useful. Now I feel like a dumbass. :slight_smile: