MultiMarkdown implementation - How?

[AmberV]

Well, given the way it currently generates an RTF file, that might be hard. At least, I am taking into consideration your explanation from an earlier comment in this thread. It would be awesome if you could. By default MMD’s RTF exporter is slightly more functional than Scrivener’s RTFD export, mainly in that footnotes are cross linked between each other. Click on a reference number, and you get sent to the footnote. But they are essentially just flattened footnotes like RTFD does.

I think the main question that should perhaps come from the user base is: Will anyone using MMD be interested in a high performance RTF file? If RTF is your goal, wouldn’t the use of Scrivener’s rich text features be the best route to take? I guess I kind of just saw MMD->RTF as a nice quick preview for us LaTeX junkies who wanted to print a few pages before heading out for the day. I could be entirely alone in that, though. The main thing is that RTF has no structure, really. XHTML and LaTeX both take advantage of structure. XHTML especially so, since it is XML at its core. RTF, you lose all of that structure you carefully placed into the document with MMD.

[fletcher]

@AmberV

Please pardon my bluster. I did not realise that I was actually talking with the designer of MultiMarkdown. I am a “skim” learner. I read through things rapidly and draw presumptions based on that. So, I do not always get everything right the first time around.

No problem at all

@AmberV

Re: Cross-referencing, I understand that you can cross-link to the labels that MMD generates (though I have not actually tested it yet). But what I was referring to was the ability to manually create a label anywhere in the document (say, preceding a paragraph on the toxicity of the box jellyfish), and then link to it using LaTeX’s “See p.121” output pointers. From what I can tell, there is no way to do this, I’d love to hear that I am wrong though.

You are correct. Currently you can link to headers (e.g. chapters, sections, etc), tables, and figures. There is not a built in way to link to an arbitrary paragraph. I, like Keith, am trying to avoid feature bloat, but am willing to consider a mechanism to do this. What I love the most about Markdown was the fact that reading a markdown document is like reading plain text. And I cringe a little every time I had a new feature for fear of “bastardizing” Gruber’s philosophy.

@AmberV

Re: Indexing. Since, as you put it so well, fully implementing this across format types would be quite a lot of work, I was really only interested in using it to generate LaTeX indexes, which only require idx markers around key words, and an index generation command at the end of the book. Using the custom tag idea, where a word is surrounded in tags, would allow it to silently fail in XHTML (nothing would happen, unless it were styled), yet allow the XSLT to detect that tag and place the appropriate LaTeX command around the word. Since it would only be a single format solution, it was never my intention to have it be anything more than a tool for myself and anyone else who was interested in the concept for LaTeX only exporting from Scrivener.

For right now, I would prefer to leave this as a user add-on. You can tweak the XSLT to recognize the tags, and a quick perl script pre-processor could look for the markup convention of your choice to indicate that a word should be indexed. Again, this is something I have thought about with MMD, and am willing to discuss it. But I don’t have something in mind that I am willing to incorporate into the main MMD release at this moment.

@AmberV

Another thing I would like to experiment is a way to detect and convert Scrivener’s annotations, and turn them into something more meaningful in LaTeX. I am waiting to see how Keith gets them exported to start looking into that, though. Definitely the use of the \color command, perhaps in conjunction with \marginpar. shrug Something like that would have little use in MMD proper, I fear.

I had looked into a syntax for pull-quotes, so something like this is not totally out of the ballpark for MMD. Again, I welcome discussion.

[KB]

Hmm, interesting. It would be possible to get it working, but it would be a lot of work. I would have to write out the RTF file and then insert the comments and footnotes directly into the RTF stream. This is what I do already, but things are complicated by another layer of abstraction: it is one thing to write out an attributed string, remember where the footnotes and comments were, and then read back in the RTF and insert them; it is another to take out all of these things and then pass it all through MMD and then try to reinsert them.

And, like you say, I doubt many users who would want to use MultiMarkdown would really be interested in exporting to RTF as their final format. If RTF is your ultimate goal, it would make little sense to take this roundabout way to get there when you could just format using RTF attributes.

So, I think the answer to this is Occam’s Razor: take the easier path. If there is a massive demand for true comments and footnotes in MMD RTF, then I can always take a look at it later. But for now, the priority should be to get it working as is.

Thanks,
Keith

[AmberV]

@fletcher

You are correct. Currently you can link to headers (e.g. chapters, sections, etc), tables, and figures. There is not a built in way to link to an arbitrary paragraph. I, like Keith, am trying to avoid feature bloat, but am willing to consider a mechanism to do this. What I love the most about Markdown was the fact that reading a markdown document is like reading plain text. And I cringe a little every time I had a new feature for fear of “bastardizing” Gruber’s philosophy.

I am definitely on your side with this one. If MMD had not been as clean as it is, and easy to use, I would not have suggested the idea of implementing it into Scrivener in the first place. Textile, the only other major alternative is, in my opinion, a bit “heavy” on the syntax. An MMD document truly does look fantastic in plain text, as well as being highly functional (a reason for not using plain MD, as simple XHTML export would not be incredibly useful – already have an XHTML export method). That is very important for a writing program like Scrivener. When we are in the middle of an inspiration, nothing is more annoying than having to recall and spend time typing syntax rules to get our ideas across. Anyone who has communicated heavily via BBS and Email in the “early days,” will find MMD’s syntax second nature.

For right now, I would prefer to leave [indexing] as a user add-on…

Agreed. When I made my initial comments, they were aimed at another potential script tweaker. I did not know they would be taken as feature requests – they shouldn’t be.

I had looked into a syntax for pull-quotes, so something like this is not totally out of the ballpark for MMD. Again, I welcome discussion.

Well, a potential problem here is that whatever the output implementation looks like, the input mechanism will be rather specialised to Scrivener. Keith is going to make it so annotations get exported inside of a special tag. I’m not sure how much you have played with Scrivener yet, but within the application, there is no special mark-up for annotations (well, that is not actually true, but maybe 2 people will actually be messing with the mark-up), they are entered and edited using GUI controls. So there would not be any syntax for the MMD script to check for. It would just be passing the raw HTML through the parser. Footnotes are handled very similarly to annotations, but I believe Keith is making it so they get converted into actual MMD syntax for that part – since there is an existing feature.

[AmberV]

@KB

But for now, the priority should be to get it working as is.

Cheers to that!

[KB]

Right, so now I’m a little confused on the annotations thing. For footnotes I will just parse the attributed string for the footnote attribute and swap them out for MMD syntax, which can be converted for LaTeX etc…

So, just to clarify, how should I best handle annotations?

Thanks,
Keith

[AmberV]

All Scrivener would have to do during export is:

<annotation>This is an annotation, stripped of extraneous white-space.</annotation>

And then, of course, the CSS style you already listed which will make these red and underlined. All of this you already know… just collecting everything into one spot for easy future reference.

But, like I said a few posts above, the tricky part is the way it will look in LaTeX. Using this method, the XSL parser will just ignore tags it doesn’t understand, like , and export the annotation as a part of the regular text stream. They will become a part of the text. This is not good.

So something (and fletcher, do step in and tell me I am a fool at any point here. I am relatively ignorant of XSLT) along the lines of this would need to be added to the file ‘xhtml2latex.xslt’:

<!-- Scrivener annotation -->
<xsl:template match="annotation">
	<xsl:text>\color[rgb]{1.0, 0.0, 0.0}{[</xsl:text>
		<xsl:call-template name="clean-text">
			<xsl:with-param name="source">
				<xsl:value-of select="."/>
			</xsl:with-param>
		</xsl:call-template>		
	<xsl:text>]}</xsl:text>
</xsl:template>

As for where in the file, I’d search for “strong” and just insert it above or below that section that handles bold. It should look very similar to the above. That would probably be safest. I had no problems in my testing.

And then, in the same file, search for “latex-header”, and you should see (around line 700) a text block that gets included in each LaTeX file. Modify that by adding the line:

\usepackage{color}

…anywhere in that block where you see a group of \usepackage commands would be safe. The LaTeX file will need this declaration, because the \color command is not a part of the core syntax. These modifications will convert the XHTML tag to:

\color[rgb]{1.0, 0.0, 0.0}{[This is an annotation, stripped of extraneous white-space.]}

Which will, when rendered by LaTeX, look identical to RTFD style annotation exports. Caveat! This will fork the default MMD distribution and make it Scrivener specific. It is up to you if you wish to take that path or not. It is two very simple changes to the XSLT only, so it is pretty safe – it just means a little more work if you ever wish to upgrade the MMD internals in Scrivener.

[KB]

Eek. Frankly, being no MMD or LaTeX expert, this is a rather difficult one. Personally, I don’t want to dabble with the XSLT file. And if that has to be done, I would rathe the annotations get exported to proper LaTeX annotations - comments or margin notes or whatever you would normally use here.

Frankly, I would feel a lot more comfortable leaving this to the experts - ie. any poster to this thread except myself.

[AmberV]

Fletcher, thinking about this a bit further, would perhaps having a third xslt file be the best way to go about this? Then, Keith could have Scrivener call ‘xhtml2memoir_scriv.xslt’ or whatever, which would import ‘xhtml2memoir.xslt,’ which in turn would include ‘xhtml2latex.xslt’. That way, the specialised Scrivener functions would remain outside of the default MMD release. Upgrades to the MMD internals would (most likely) not conflict. Additionally, this would allow users like myself, a safe place to add new syntax parsing.

Keith, would that make you more comfortable? Basically it already cascades. Your script calls xhtml2memoir.xslt, which utilises the main LaTeX converter. This would simply be adding a small layer on top of the cake that adds annotation features – and the ability to easily expand Scrivener’s export vocabulary in the future, without forking and messing around with the original distribution.

If that is okay, I’m willing to put this file together. I can basically see how it is done in the Memoir layer. If anyone else more skilled cough has the volition to do a more professional job of it – go for it.

[AmberV]

@KB

I would rathe the annotations get exported to proper LaTeX annotations - comments or margin notes or whatever you would normally use here.

The reason I went with the RTFD embedded text method is two-fold. First, as far as I know, there is no such thing as an inline comment, or comment in LaTeX at all, that is not literally a comment. In other words, it is more like a programmer’s comment. It will never appear in export. The LyX inline notes that we actually examined as a model for Scrivener’s inline notes, are proprietary to LyX, they are not part of the LaTeX specification. They never appear in exports or renderings.

That leaves margin notes, which raises the second problem. Margin notes – we all know the many problems with them, and even considering the “soft” margin notes featured in JNW which have helpful pointers. The margin notes in LaTeX are really designed as a typographical solution to the need for “blurbs” and side-bars in printing. They are not at all well suited for contextual commentation. They will get placed in the column adjacent to the location of the declaration, but no marker will be drawn to its reference point, multiple adjacent notes will run together – and large notes will become completely impossible to decipher at a glance – what they are referring to.

So, that is the two-pronged reason for endorsing an embedded system. I suppose a third reason would be: Why would anyone export annotations into their final copy? If they are exporting annotations, it is probably for their own use, or for an editor. In that case, having them remain contextual would be ideal – even if it does break up the flow of the printed document. Same story as RTFD.

Debate welcome, but short of coding our own LaTeX macro (where I would be completely over my head), there is no easy way to address annotations. LaTeX was just not designed for editing, so it has few tools to accomodate writers. It is all about “print this book.” If there are any LaTeX experts here that know of a ready-made class we could use, or some better way to handle this – I am not expert. I know enough to get a manuscript looking good for my own purposes.

[fletcher]

@AmberV

I think the main question that should perhaps come from the user base is: Will anyone using MMD be interested in a high performance RTF file? If RTF is your goal, wouldn’t the use of Scrivener’s rich text features be the best route to take? I guess I kind of just saw MMD->RTF as a nice quick preview for us LaTeX junkies who wanted to print a few pages before heading out for the day. I could be entirely alone in that, though. The main thing is that RTF has no structure, really. XHTML and LaTeX both take advantage of structure. XHTML especially so, since it is XML at its core. RTF, you lose all of that structure you carefully placed into the document with MMD.

I haven’t asked around a whole lot, but I wonder how many people actually use MMD->RTF. 4-5 yrs ago, I thought RTF was great. Then I realized that it’s not actually cross-platform or cross-application - every app seems to interpret the RTF slightly differently. I have now settled into the belief that it’s a crappy format, though marginally better than .doc.

There is not a lot of built-in support for RTF within MMD, and I used Tiger’s built in tools to handle the export (so that does not work for Windoze or *nix MMD users, but they won’t be using Scrivener either).

I’ll have to hear from others about the relative benefits of MMD->RTF vs. Scrivener->RTF. Is RTF something that a lot of MMD users are interested in? To me, it’s a throwaway format for those instances when XHTML and PDF don’t work (I figure anyone interested in LaTeX would cringe at RTF, but perhaps I am wrong).

That said, for anyone using MMD, I suspect that the MMD->RTF export pathway would be more consistent with what they were expecting. Let me know what doesn’t work, and we can look for an alternative means of patching Scrivener and MMD (we could try and find a way to convert Scrivener footnotes to MMD footnotes before processing, and could look into annotations as well…)

I sound like a broken record - but feel free to bring up any RTF issues on my wiki as well…

[KB]

In which case, the other - simpler though less visually attractive - approach is just for Scrivener to place square brackets around annotations before converting to plain text, and leave it like that. The (big) drawback would be that they would not be coloured in your LaTeX file; the advantage would be that you would not need a separate XSLT file to do this, it could all be done in Scrivener at the pre-processing stage. Are there not MMD tags that assign a colour to a section of text?

Thanks,
Keith

[fletcher]

@AmberV

All Scrivener would have to do during export is:

<annotation>This is an annotation, stripped of extraneous white-space.</annotation>

discussion snipped…

For now, this sounds like a reasonable approach. I welcome interested users to continue the discussion as to whether there is something comparable to an annotation that should be in MMD. More specifically, what would it look like in plain text. That discussion can take place via email, or on my wiki, if it is felt to be too off-topic here.

If such a syntax is developed, we could add change the scrivener export mechanism in the future.

As for the XSLT part, I would suggest creating a bare bones scrivener specific XSLT that then calls the other XSLT. I have done this for myself - I have a basic memoir class XSLT that calls the xhtml2latex basic XSLT. Additionally, there is a dissertation specific XSLT that I used to transform MMD into a PDF for printing at Lulu.com. It adds a couple of templates, and then calls the memoir XSLT, which then calls the latex XSLT. This building block approach is very powerful, customizable, and allows reusable code.

To recap, I suggest a scrivener2latex XSLT file, built upon either xhtml2memoir or xhtml2article or whatever (or all of the above). This file can add support for annotations, and any other Scrivener specific information.

[fletcher]

@AmberV

Fletcher, thinking about this a bit further, would perhaps having a third xslt file be the best way to go about this? Then, Keith could have Scrivener call ‘xhtml2memoir_scriv.xslt’ or whatever, which would import ‘xhtml2memoir.xslt,’ which in turn would include ‘xhtml2latex.xslt’. That way, the specialised Scrivener functions would remain outside of the default MMD release. Upgrades to the MMD internals would (most likely) not conflict. Additionally, this would allow users like myself, a safe place to add new syntax parsing.
…

I replied to soon - great minds think alike…

[fletcher]

@KB

In which case, the other - simpler though less visually attractive - approach is just for Scrivener to place square brackets around annotations before converting to plain text, and leave it like that. The (big) drawback would be that they would not be coloured in your LaTeX file; the advantage would be that you would not need a separate XSLT file to do this, it could all be done in Scrivener at the pre-processing stage. Are there not MMD tags that assign a colour to a section of text?

No. Markdown, and therefore MMD is really designed at the structure of the text (chapters, sections, lists, tables, links), and not the appearance. For one, I believe that most users get so focused on the appearance, they forget the content. Second, if you want to alter appearance, you can change the CSS (for XHTML) or LaTeX code/preamble.

The reason I became so enamored of LaTeX is that it basically assumes that users don’t know how to handle the appearance of text and uses the approach that the software can do a better job (in most cases).

I believe that this concept is increasingly supported by various professionals in typography and web design. MMD focuses on the structure, and allows the appearance to be determined by the structure. There is no built in default way to directly influence the appearance alone.

As to how this applies to annotations, something like ‘’ (I have not looked into whether this is valid XHTML, BTW) assigns structure. CSS/LaTeX can then be used to assign an appearance to this structure. I might not incorporate this directly into MMD, but I agree with this philosophy.

[AmberV]

You would also lose colouring everywhere else, too. There would be nothing for the CSS to style, and the RTF would be black and white. I already thought of the colour via MMD as that would have been obviously the best idea (you could even retain the user’s Annotation colour choices at that level), but alas, there are no colour markings that would also be LaTeX compatible. You could get it to the XHTML point by just putting a raw into the export. But, that would produce a black and white LaTeX since the span tag is not handled by the XSLT.

If you do decide to go the simpler route, I would recommend plan A. Stick with wrapping them in tags. That way we have colour in HTML and RTF. The only difference would be, add the brackets inside, too.

<annotation>[This is a comment.]</annotation>

So, by default it will be black and white in LaTeX, but at least they would be bracketed, and by using tags its opens the possibility for much easier user modification.

[KB]

Okay, this sounds good. The other solution, though, is that Scrivener could just add the LaTeX colour formatting into the plain text stream before it goes through MMD. Given that this is only an issue with the LaTeX export, surely it would actually make more sense for Scrivener to insert:

\color[rgb]{1.0, 0.0, 0.0}{[This is an annotation, stripped of extraneous white-space.]}

right off the bat rather than using:

<annotation>This is an annotation, stripped of extraneous white-space</annotation>

and then asking the XSLT to convert this. After all, if Scrivener inserted this, then the annotations could retain their colour in the LaTeX document. I am assuming that MMD will not strip out this information and that it will all just go through.

So, to recap, I am now thinking that the most obvious solution would be for Scrivener to insert the LaTeX syntex for annotations directly before it even goes through MultiMarkdown.

Thoughts?

Thanks,
Keith

P.S. As for RTF being a “crappy” format, I heartily disagree! It may be so to text and structural text purists, but as a rough standard that allows you to get text that supports the main features of a typewritten text (bold, italics, underline) across platforms, it is decent enough. More importantly, it is accepted by most editors and publishers; most short story magazines would not be happy with a .tex or .pdf file, for instance.

[AmberV]

@fletcher

(I have not looked into whether this is valid XHTML, BTW)

To save you the time: Strictly speaking, it is not. However, the trick works in every browser so far, including Apple’s XHTML->RTF converter. Since Scrivener is essentially using XHTML as an intermediary to LaTeX, and the XSLT will understand non-spec tags, I figured absolute compliance was not so important in this instance.

I am normally quite a stickler for web compliance, but in the case of internal usage and getting maximum use out of something in a closed environment, I think it takes a slightly lower rung than universal standards.

That probably limits its use outside of Mac Scrivener users, unfortunately.

[AmberV]

@KB

So, to recap, I am now thinking that the most obvious solution would be for Scrivener to insert the LaTeX syntex for annotations directly before it even goes through MultiMarkdown.

I already tried that! Unfortunately it doesn’t work. The XSLT automatically escapes anything that is a special LaTeX character. What you would get when you render, is literally that text, because every special character would be escaped.

Which, incidentally fletcher, my first formal feature request (which, I suppose I should be doing on your Wiki – many pardons! I am doing a thousand things at the moment!) to you would be a header option that allows direct LaTeX input into the source MMD file.

[KB]

Ah… Okay, thanks. In which case, would you mind preparing the necessary XSLT file and sending it to me at keith AT literatureandlatte dot com? Or post it or whatever. I really want to get this going tomorrow morning, but I don’t want to rush you. I can always change it after you have it ready.

Shame that we have to lose all the colour information - but then, the colour information is lost in RTF comments, too.

Thanks,
Keith