LaTeX "simple"(?) syntax highlighter?

Hi Keith, Amber and others interested in this question –

Love Scrivener; it’s becoming a central solution to stream-lining my writing process. That said, I have an enduring frustration with trying to combine my use of Scrivener with my need and love of LaTeX…currently, I write in Scrivener, and then revise in any dedicated Tex editor. I appreciate that MMD is the designer’s chosen intermediary between Scrivener and LaTeX, but my use of LaTeX is complicated enough that it’s easier to code it directly than use MMD as a work-around, and I just export plain text.

My central frustration is that any reasonable amount of LaTeX code becomes very distracting if it isn’t syntax parsed w/ highlights. Vim has, for example, a fairly short parsing routine that permits the syntax highlighting of LaTeX. If a similar option existed to highlight LaTeX code in color fonts in the text composition windows of Scrivener, I’d happily pay double fees! Triple fees!

Second place on my list of frustrations may just be ignorance on my part: I would love to be able to “compile” (would it be correct to say: export from Scrivener native RTF to LaTeX?) EXTREMELY simple formatting without going through MMD – eg. for Scriv to export Outline/Section headers as \section etc? The solution I envision would be for people who are already comfortable with LaTeX – in other words, any code is passed raw, backslash escaping LaTeX protected characters is the writer’s responsibility (which is easy with proper highlighting) – only the document structure would be generated into LaTeX by Scrivener, but the produced code would otherwise be free of preambles etc. It’d be slick to have RTF boldface, italics and tabs go to \it \bf and \indent but with a parser/highlighter this wouldn’t be a necessity – and might well be something that expert LaTeX+Scrivener folks would eschew.

Anyway – this second wish is quite minor, and opens up a large can of design worms, in comparison to the extreme utility of having a LaTeX syntax highlighter.

Many thanks for the consideration of this –
Best,
Eric

As you note, MMD is there for those that like the system and can work around its abstractions or don’t mind coding exceptions. Personally, I think Scrivener makes a fantastic document editor for those that prefer LaTeX or need the extra control available to hand typing in the typesetting instructions. That’s your category, and if you do not want an abstraction layer, I see no reason to use MMD since that is its primary function.

The problem with syntax highlighting in Scrivener is in part due to Apple flaws in the text system. It’s just not viable to recolour, in a temporary fashion, bits of text without a performance hit in a rich text file. You can get away with a small amount of that (and indeed it is used in full screen composition mode as an option and inline notation), but something as intense as syntax highlighting would put a strain on this mechanism. On the other hand, actually colouring the text itself according to a rule set would conflict with the program’s rich text background. But on that notion of actually colouring the text, you might try playing with the scriptwriting function. While it is obviously geared towards screenplays and such, at its core it is basically a simple cascading reformatter. It’s not going to give you syntax highlighting, but you could define certain blocks as “code” in a colour, so long as that code fell on its own line.

As for making compile a bit more automatic: yeah, you can do that already. Try searching in the Multimarkdown board for LaTeX tricks. Most revolve around two compile time option panes: Replacements and Formatting. Set your compile mode to plain-text, and then use the Formatting pane to add a suffix and prefix around the title. Since Formatting rules are depth aware, you can make a suffix/prefix pair for a top level folder that emits “\part{” + TITLE + “}”, and a second level folder that emits “\chapter{” + TITLE “}” and so on. Replacements are just like global search and replace, but without destroying the manuscript and only running during compile. Replacements are useful in a coding context as you can create your own macros—personal codes that expand to often-used full LaTeX codes when you compile.

Anyway, the combination of moving structural data out to a procedural generator, and using replacements to compact common code, I would think the syntax highlight desire would diminish.

Hmm, you can’t do it yet, but in the next version you could do footnotes automatically, too. You’d actually have to use inline annotations, but you’ll be able to define the enclosure strings, and so could put “\footnote{” in the first field and “}” in the second field.

Preambles and such—just put them into their own documents in the Binder. Since everything will be combined into a single file, you can totally separate all of this bulk from the important content.

Hi Amber –

Thanks for the reply.

If I understand you correctly, elements in program design for displaying the document source, because it’s stored as RTF, prevent syntax highlighting? There are a lot of highlighting widgets in existence – GNU’s got source-highlight, there’s a gui wrapper by andre simon called highlight, vim’s got it’s own filter, Eclipse, etc etc – so the filtering itself is fairly canonical…and a lot of OSX-only packages do syntax highlighting, albeit they may not be designed around RTF. If I understand you, there’s a problem putting a filter between the source stored as RTF and elements of the OSX display API that prevents one from doing it on the fly? Bummer.

Perhaps an easy solution might be a UTF-8/ASCII only option for the LaTeX-familiar folks, which strips all RTF twaddle when compiling, but meanwhile stores the syntax colors as RTF, and removes user choice in font color. Actually, this could just be a “no-color” option for the latex folks, if one wanted it, allow bold , italics – eg everything but color options could potentially still be fine, except for indents…

I’ll dig into Formatting rules; “Replacements” as a tool sounds effective…thanks for this…

Best,
Eric

Filtering isn’t the problem, and you are right that there are plenty of resources available in Cocoa and other languages for parsing code and colourising text. If that was the only hitch we could probably support hundreds of languages. The problem, as you point out, is that all of these examples are working in plain-text, where colour overlaying is dramatically more efficient as they aren’t using the tool we would have to use for RTF, and in some cases they are not really using Cocoa too heavily for the text drawing at all. I’m not 100% on that, but I’m pretty sure that MacVim uses its own text rendering code, and I’m relatively positive that all of the Cocoa exclusive plain-text editors out there are rolling their own text engine based off of the plain-text engine, or even lower than that in some cases. It would be, as far as I understand, completely possible to do this in an RTF editor—the problem is performance coupled with it being somewhat confusing in edge cases where temporary text colour is used for other things; that’s the lesser issue though. The primary issue is that using excessive temporary text re-colouring turns Scrivener into a massive hog in even a reasonably sized document (say 5,000 words even, let alone 200,000) due to deeper optimisation problems in Apple’s code here. So the choice really feasibly comes to:

  1. Molasses: typing lag used to be a fairly common problem in Scrivener 1.x for a number of users. It relied more heavily on this feature and stripping most of it out (among other optimisations, to be fair) has reduced type lag bugs to extremely rare, and generally only the result of some third-party tool introducing too many CPU cycles between keystrokes
  2. Actual text re-colouring like the Scriptwriting mode does: risky move in an RTF environment where so many different workflows must collide into the same area of the interface

This is all an in-editor problem. Stripping colour out of a compiled document is no problem. I’m assuming you haven’t tried the Plain Text option in the compile interface yet—as producing a UTF8 file is already an available option, and definitely what I would recommend for anyone using Scrivener to compose documents that have been marked up with a plain-text convention. I myself use MMD because I rarely need intimate control of LaTeX, but I pretty much use all of the RTF tools one way or another as a form of “safe markup”. I know that nearly anything I do in the editor, relating to formatting, will vanish when I compile. It’s a pretty nice way of working.

But on the notion of an all-black RTF output: that’s possible too right now, but it’s a bit of a kludge, you have to set up your formatting rules to use Black instead of pass-through in the formatting editors. The next release will have a handy checkbox for this instead. But like I say: you don’t have to worry about that—just use plain text to compile.

Quick tip on Replacements: you can specify a variable position in a replacement, allowing wrapping something in syntax that expands to larger syntax. For instance, in the Replacements field you could put:

Replace: ^$@^ With: \overline{$@}

The [b]$@[/b] token will trap all content in between two caret symbols (^) on the same line, and replace the carets with the LaTeX code, replacing the value of the stored $@ in the appropriate location. So:

Examination of ^A^ indicates… would become: Examination of \overline{A} indicates…

First - I’m committing the cardinal sin of not reading the entire thread before posting. But I figure I can get a little leeway on this one.

Anyone interested in MultiMarkdown, syntax highlighting, and Mac OS X should check out the latest post on my site:

http://fletcherpenney.net/2011/06/ive_seen_the_future

It includes a link to a Cocoa plain text editor with Markdown syntax highlighting. Once I figured things out, it took very little time to get up and running. My goal, of course, is to tweak it to support MultiMarkdown syntax highlighting, but it’s a great proof of concept. It’s built off of a “PEG”, which I suppose one could write/find to provide highlighting for LaTeX and other languages.

And it’s fast.

F-

That reminds me of another approach to all of this: if all you need and want of Scrivener is plain-text (i.e. no footnotes; links; inline notation; etc), there are some good tools for wiring a text editor to nearly any text field on your Mac, and any Cocoa field. QuickCursor is pretty good and works with modern versions of OS X (older methods using InputManagers most likely will not, at least not without a little forcing); you just load up the section you want to edit in Scrivener and then use this tool to open the buffer in MacVim, TextMate, or whatever you prefer. On save and quit, the edits will be returned to the original editor they came from for you. The only real limitation, which is ordinarily not a huge problem but in Scrivener it can be, is that you need to leave the editor alone while something is out for editing. I recommend locking the editor—otherwise you can end up with your edited text going to the wrong spot.

Thanks for bringing that project to my attention, though, good idea to use the same parsing code for the highlighter.