Am I missing something or is MMD/Scrivener worth the effort?

Hi,

I’ve been using Scrivener for about a year or two, but now I’m involved using it for my first intensive (and extensive) project. It’s very mathematical, so I need a few things. One is numbered equations, and the other is good table support. Others include support for citations and bibliographies and for cross-references.

I immediately ran into limitations with MMD. Version 3 does not number equations, and its table support is relatively crude. It seems to convert labels to lowercase automatically, so I can’t use camelcase to designate subtle differences between equations and tables. And my citation workflow is Zotero -> BibDesk -> Scrivener -> LaTeX with natbib.

So the net result is that my writing is perhaps 70% or more escaped with tags and raw LaTeX code comprising most of the rest. This poses several problems. First, Scrivener has no built-in LaTeX editor, so writing is tedious and very error-prone. Second, reading the document to check for mathematical errors, etc. is virtually impossible. Third, because Scrivener’s editor does not error-check the LaTeX code, and given that even a single missing “}” screws up the entire LaTeX compile, with error messages having no relation to the actual error, even proof-reading a single page can take inordinate time.

Scrivener offers lots of attractive features for large, complex writing projects, but given this experience with MMD3, I’m wondering if it’s worth the effort? Or am I missing something?

Do the rest of you run into these problems but have some way to work around them? Or should I be using LyX or TeXmacs, even though they don’t have Scrivener’s other author-friendly features?

Note: updated to refer to modern techniques and tools.

I think part of the problem is the level of expectation over what MMD is designed to do. Given its level of abstraction over precision coding, and its fairly generic output options, I would not consider it an A to Z solution (much like Scrivener itself, especially for science texts). What it is is an accelerator. It can get you from A to—I don’t know, let’s say Q—much faster than doing so in LaTeX with TeXShop or something, and maybe even with LyX, when you consider that Scrivener is a more appealing environment to do the actual writing in.

But the result of that is going to be what you would expect of state “Q” in any workfow. It will be generic, rough in places, but precisely because of how generic it is, ready to easily continue onward with more precise work. And for that, I consider it to be an excellent way to work in Scrivener. For those using word processors, they get a fully stylesheet enabled document with all of the fancy counters and cross-reference wiring intact. It’s not done by any stretch of the imagination, but because it’s all set up to be easily manipulated, it won’t be nearly as much effort to proceed with the final publication polishing steps. Likewise with LaTeX, which is stylesheet/content based from the very core, you can drop your output into a LyX boilerplate and start working on it immediately. You get the best of writing environments for the creative work, and the best of finishing formats for the polish.

So, yeah, if you want to try and make MMD produce “Z” when you compile, then by all means you will probably need all kinds of `raw code`{=latex}, and Replacements in the compiler can help a lot with that, it’s not a bad way to work if your needs are fairly simple. In fact I work that way for the Lit n Lat documentation. My projects are designed to put out a 99% complete .tex file, and that done via a number of tricks in the software and with MMD. I’m content with that because I have a rapid publication cycle and cannot afford to spend eight hours post-producing every user manual revision. Here is an example of what the WIP for the Scapple documentation looks like:

* If any notes are selected, you can hold down the Option or Command keys while double-clicking to create a new note that is connected to the selected notes. Holding down the Option ({{Opt-}}) key will create a new note that is connected by an arrow pointing to the new note; holding down the Command ({{Cmd-}}) key will create a new note that is connected by a dashed line to the selected notes.
* Use {{Notes//New Note}} ({{<$custom:Shortcut>}}). If there are no notes selected, this will create a new note in the middle of the view. If there is a note selected, this will create a new note directly underneath the selected note---in which case the menu item will appear as ["New Stacked Note" in the Notes menu][Stacks].

What I get when I compile to .tex is:

\item If any notes are selected, you can hold down the Option or Command keys while double-clicking to create a new note that is connected to the selected notes. Holding down the Option (\interface{{\optkey}\,}) key will create a new note that is connected by an arrow pointing to the new note; holding down the Command (\interface{{\cmdkey}\,}) key will create a new note that is connected by a dashed line to the selected notes.

\item Use \interface{Notes\,\RIGHTarrow{}\,New Note} (\interface{{\cmdkey}\,Return}). If there are no notes selected, this will create a new note in the middle of the view. If there is a note selected, this will create a new note directly underneath the selected note---in which case the menu item will appear as ``New Stacked Note'' in the Notes menu (\autoref{stacks}).

As you can see, I’ve got custom syntax going on, most of which is done in the Replacements compile option pane. {{$@}} is converted to `\interface`{=latex}$@`}`{=latex} on the fly when I compile. \interface{} is a simple LaTeX command I wrote which styles the text in san-serif blue. Meanwhile “Cmd-” gets converted to a symbol, and // gets converted to the filled triangle arrows between menu names. The result is clean, to my eyes, and easy to type in. I never have to look at the intermediate Markdown version. This is all greatly enhanced through the use of RegEx. The ‘//’ replacement is actually (?<!http:)\/\/ because I don’t want URLs to get mangled!

The keyboard shortcuts are using Scrivener’s placeholder syntax to recall custom meta-data from a remote Binder item, in this case the menu entry for “Notes/New Note”. If a hotkey gets changed, I switch one piece of meta-data in the menu appendix and the entire project updates.

For special environments, I use Scrivener’s styles, which has a prefix and suffix system, and that is how the extended syntax and MMD’s escaping around them get inserted.

The result in the .md file is a bit of a mess if you try to read it—but I never look at that. No human does, it is passed straight to MMD and out to .tex, with the clean result. I get a readable .tex file and my Scrivener project is easy to edit and write with.

So there are things you can do to make your life easier, pushing beyond that hypothetical ‘Q’, without turning your text editor into a bunch of code spaghetti. And you can even take this further, should you choose to use Pandoc instead of MultiMarkdown. The latter is really simple to use, but pretty much only works one way. If you want to get in there and change the specific attributes figures use, there aren’t many good options. Pandoc on the other hand is template-driven, and so you can fine-tune and even completely design from scratch what your .tex will look like.

The question is always going to be how much of this extra effort to take. The big deciding factor, to my mind, is how rapidly you need to produce revisions of the work. For something like a thesis or a paper for a journal, once one is done with Scrivener they’ll probably never come back to it, then spending 16 hours on final production quality is entirely reasonable. For a software manual that gets published dozens of times per year, it makes more sense to maybe spend a week or two building a much more automated system.

To a few questions:

Version 3 does not number equations…

I’m not sure what you mean by that. MMD has never numbered equations. LaTeX handles all of the numbering as far as I know. Baking numeral characters into the file would be antithetical to the philosophy of the system, which encourages pure semantic statements and automation.

It seems to convert labels to lowercase automatically, so I can’t use camelcase to designate subtle differences between equations and tables

I’d suggest prefixes or something to denote like label types, if you prefer calling groups of things identically. eq:something vs. table:something. That seems less error-prone to the human eye than case variations in my humble opinion.

There is so much good stuff in Ioa’s post that I’m going to have to study it very closely (and I’ve got to get my head around RegEx).

The only thing I’ve got to add is that when I have to work with equations I go straight to MathType. That lets me compose the equations visually then paste the Latex code into Scrivener. A compile replacement of the standard latex [ and ] with the MMD \[ and \] means there’s no need to do anything else to the code once it’s pasted in.

Thanks to both AmberV and you, MrGruff. Like you, I’m going to have to study AmberV’s post in more depth too.

But I think my biggest problem with using MMD, once its limitations became apparent, is the fact that I’m writing a paper making extensive use of math and tables. Specifically, I’m developing a dynamic macroeconomic model of monetary flows to study the role of housing in the global financial crisis of 2007. This approach uses tables for double-entry bookkeeping of the monetary flows and fills in the cells of the tables with elements that ultimately comprise a system of differential equations to study through computer simulation. Because of the nature of the project, I don’t just need to typeset mathematics. Instead, I have to see all the previous mathematics in order to figure out what to write next. With Scrivener and MMD, this implies a recompilation for every equation, using two windows for the writing process – one with Scrivener and one with the formatted LaTeX output – and having to write a fair amount of LaTeX code within Scrivener just to get the desired display.

Put differently, one of Scrivener’s great strength’s is that it facilitates writing by focusing attention on the writing process rather than the final, formatted output. In non-mathematical work, the author typically has no trouble seeing what’s already been written and, therefore, connecting new prose to earlier prose. But because the Scrivener - MMD - LaTeX - PDF (SMLP) workflow does not display anything resembling mathematics or, to a lesser degree, tables until they are typeset at the PDF stage, the workflow breaks the ability to concentrate on (a very difficult) train of reasoning while writing.

Here are a few more comments on Amber’s post.

  1. Regarding MMD’s inability to number equations, I meant that MMD3’s math feature does not generate LaTeX code that automatically numbers equations. Hence, in a complex mathematical document with numbered equations and references to them, one must embed “raw” LaTeX code instead of using MMD3. See this post by Fletcher Penney.

  2. I already use prefixes to distinguish the kind of object in a reference. However, the objects themselves need to be distinguished. So, for example, one might have (1) “eq:1SectorProfits” to designate an equation calculating total profits in a one-sector model, (2) “eq:2SectorProfits” for the same equation in a model with 2 sectors, (3) “eq:2SectorProfitsWithTimeDifferences” for a 2-sector model in which each sector has different turnover times, and (4) “eq:2SectorProfitsWithTimeDifferencesAndLand” for a model like that in #3 but including land rents. I realize it’s possible to come up with a different labeling scheme that MMD does not screw up, but I find this approach most natural. (I’ve been using it for file labels, variable names, and other things ever since I worked on my first Unix system in the 1970s.) Once again, the issue is not that there’s no way to work around MMD’s limitation. Instead, the issue is that the workaround substantially distracts attention from the writing process.

  3. AmberV points out how helpful the SMLP workflow is for maintaining large documents that change frequently. As a college professor, my work might resemble this for syllabi. But for academic writing, my needs are at the opposite end of the spectrum. I need a tool that helps me concentrate on what I’m writing, with as few distractions as possible, and allows me to revise easily perhaps a half-dozen times but with a set of relatively basic and commonplace needs that do not justify the investment of time for extensive customization. The SMLP workflow satisfies these needs remarkably well for documents that are largely prose with simple math and tables, but things are quite different for documents with complex mathematical reasoning with complex equations and tables.

  4. Reiterating what I said earlier, I think this all boils down to one thing. AmberV’s post addresses ways of getting some % of the way to having a document typeset in final form. But my issue is different. Because of the extensive math, I need to be able to see the typeset mathematics (or a reasonable approximation) in order to know what to write in the first place. :frowning:

Your very understandable need to be able to see the equations as you write has reminded me of a hint elsewhere on this forum.

If you have LaTeXiT installed ( which should be the case if you’re using TexShop) you should find that if you select a latex equation there will be several items on the services menu (which you can access from a right-click) starting Typset latex maths … By picking the right one of these you can get the equation typeset within Scrivener: it is pretty close to magic. You can choose to typeset inline, display or aligned maths.

Then, when you want to revert to the latex code - presumably around compile time – there’s another services item Revert equations back to LaTeX code which does exactly what it says. (If you don’t do that then the equation gets spat out as a pdf.)

That may help.