Putting LaTeX header into binder

I’m doing a lot of customizations for a compile preset that uses MMD->Latex, and wanted to move the “Latex Options->Header” to the binder so I can snapshot what works as I experiment with global settings.

But I can’t make Scrivener treat the contents of my Metadata document as LaTeX code. Instead, it translates everything like this:

\usepackage{tabulary}

… to:

\textbackslash{}usepackage\{tabulary\}

I’ve done all the following, trying to make to work without adding the \latexcode{=latex} escaping to every line in my Metadata document:

  • Created a document titled “Metadata” and put it as the first entry in my Front Matter folder, so that it appears at the top of my included compile documents
  • Created a custom style called “Latex Code”, applied it to all of the text in the Metadata document, then added it to the compile preset with “Treat as raw markup” checked and “paragraph prefix” set to , and "paragraph suffix" set to {=latex}
  • Created a “metadata” section type, assigned it to the Metadata document, and assiged “As-Is” section format to that section type.

If I add the escape code inside the Metadata document to a line of latex code, then it gets written as raw latex code, so for some reason, my custom style isn’t doing what I’m expecting. Is there a way to make this work without adding the {=latex} escape code to every single line in my Metadata document?

Yes, thankfully there is a block code injection method!

```{=latex}
<Insert 500 lines of preamble>
```

This is a variation on the “fence block” method. Without the bracketed language indictor, you can use three backticks around a large code block without having to mark every line.

You could of course just type the above into the file. If you wanted to do this on a regular basis however, a Section Layout for LaTeX code with a prefix/suffix should do nicely.

Well, that’s definitely better, but now the latex header code is now coming out with backslashes and curly braces escaped…
This:

\usepackage[T1]{fontenc}
\usepackage{textcomp}
\usepackage[utf8]{inputenc}

… becomes this:

\\usepackage\[T1\]\{fontenc\}

\\usepackage\{textcomp\}

\\usepackage\[utf8\]\{inputenc\}

Also, I can’t name the document “Metadata”, or it encodes everything as before… Is there a compile setting that I need to check or uncheck? I’ve tried everything that seems remotely relevant.

Hi Robert, do you have Convert rich text to Multimarkdown enabled in your compile settings? This is a nuclear option that escapes everything and is not compatible with mixed format workflows…

Ah… of course.

Thanks. I don’t know why that didn’t occur to me. :unamused:

Well Convert rich text to Multimarkdown does not include in its name the fact it actually escapes all existing markup (unconvert existing markup basically!), so it is not entirely obvious that this is what the setting will do. One benefit of making this separate in the settings would be making the consequences of this setting more obvious, but there could be a help button or something there too?

I think a short section in the MMD/Pandoc sections of the manual titled to the effect of “About the Convert to MMD option…” that explains the consequences of using it with custom markup formatting would be sufficient. Especially for someone like me, who already has a large body of text writing without MMD formatting in mind. Actually, a section on converting an existing project to use various MMD compile formats would be helpful. Maybe once I’m done, I’ll dust off the cobwebs on my blog for a “lessons learned converting a Scrivener project to MMD/LaTeX compiling” post.

Thanks again for pointing that option out.

The intentions and limits of this checkbox are documented. From Markup Options, in the section where the general options for the compile pane, and this checkbox, are discussed:

(Emphasis added.)

It goes on to discuss briefly how one can affect a hybrid approach when using this option, via Styles.

The concept is also introduced in the Using MultiMarkdown and Pandoc chapter, where the three major philosophies of Markdown usage within Scrivener are described. The checkbox is included in what is described as “incidental” usage of Markdown, in list item 3, pg. 524.

Do you have a specific example in mind here? Do you mean going from a traditional RTF-based document to a pure Markdown-based way of working? If so I think this very checkbox might prove very useful, at least as a way of getting started. Depends a lot on the source of course. A project that had a very intricate and metadata-laced Draft folder, never mind a lot of annotation/comment usage and such, a series of actions done to convert the material in place would be better.

Sorry, Ioa. I should probably pause to read that chapter/section more carefully, as the interaction between Scrivener and Latex isn’t quite as intuitive as the standard methods for writing in Scrivener. It would save me some confusion and frustration, even though it always feels like I’m sacrificing time that I could devote to doing something when I take the time to learn how first. :unamused:

After delving into fixing all of my now very customized MMD->LaTeX compile issues, I don’t think it’s guidance on how to use Scrivener features, so much as advice on when to abandon those features and just convert to MMD plain-text syntax (and how to make that work with a minimum of latex code{=latex} wrappers peppered throughout various compile dialogues.

One thing I discovered is that RTF tables copied from PDFs, Google Docs, and web pages give me errors about 10% of the time when they’re converted to MMD and then to Latex. I think it’s just the sloppy nature of the original source formatting. I eventually gave up and converted everything to MMD style tables, using TABs to align each pipe symbol/cell border. That was a chore made easier by exporting from Scrivener to Docx, and then using pandoc to convert to MMD tables of a style compatible with the MMD implemented by Scrivener, but it was still a tedious process that ultimately paid off after weeks of trying to clean up the RTF tables so they’d compile to MMD properly.

The next thing on my list was controlling how images scaled by just using MMD syntax+Scrivener links with image files in my binder. That worked a lot better than images dropped into my RTF document in Scrivener. BUT! Adjacent (not overlapping, as far as I could tell) RTF bullet lists and enumerated lists didn’t play nicely with the MMD image importing syntax. I visually inspected everything, from invisible characters to paragraph spacing, and could not find anything different from one file where the images came out correctly on my MMD->Latex->PDF compile, and the next, very similar file’s image syntax was mangled during compile. Rather than struggle with whatever was wrong with the RTF → MMD conversion, it was just easier to replace embedded images with reference-style MMD image syntax, and I’d recommend that to anyone converting to MMD with a lot of images in their documents.

Then errors were thrown when I started including documents with bulleted lists, because of the unicode point characters used in the lists. Converting them all to MMD enumerated and unordered list syntax fixed THAT.

If I could have predicted how much time I’d waste trying to make the RTF->MMD conversion functions work with a lot of externally originating RTF data, I would have skipped straight to converting all of my text to MMD, or at least a close approximation to it.

EDIT: That last statement is not a knock on Scrivener; I think it’s down to the nature of RTF being a poorly defined file format combined with a complex set of source data of dubious provenance.

I see what you mean. It’s one thing to have a menu of features to choose from that Scrivener can handle conversion for—which is how that chapter is largely structured—it’s another thing to know when a simple converter might not be enough. That’s a tough one to document I think, it being so contextual. For example there are things that Markdown lists can do that the rich text list generator simply cannot, like this:

* Here is a complex bullet element.

    This is the second paragraph of the list element.

    1. But we want to enumerate some points within this major bullet entry.
    2. So we do, and MMD doesn't care.
    3. All right, we're done.

    And in closing, we summarise the points made in this bullet element.

* Then on to the next bullet point.

There are cases like that which are cut and dry, the list tool just won’t let you even try to do the above, you’ll get a mess. But overall, what makes it tough to document I think is that it is documenting all of the negative space. When does one need to use figure syntax manually? There isn’t one answer because there are many things MMD figures can do that Scrivener’s do not. (And yeah, sometimes using native MMD image markings are the only way to go when it comes to getting them properly integrated with a complex list).

Like you say, that’s not even getting into the gradient where MMD/Pandoc break down as abstraction formats, and one needs to start thinking in terms of code-specific output, be it HTML, LaTeX or what have you. It’s the same problem in a different place—I’m just thankful that Scrivener can help bridge that transition without burdening the source text with it (which is its own learning curve, but a rewarding one).

So I’m not sure how to approach that one (at least the first gradient). Both of them to me feel like the kind of thing that just requires experience. Something to think about though; perhaps to better inform how things are described in that chapter, in definitive terms of what is provided by a specific tool, with a singular piece of advice in introduction that where one needs to, it is perfectly fine to drop to Markdown on a case by case basis. Even with the incidental approach, so long as one takes care to mark the places where they do so for the compiler—same as one would do if they were injecting raw HTML into their ePub3/KF8 workflow, and for much the same reasons: to do things Scrivener/RTF cannot do.

Could be, though I would have expected tables from Google Docs to be a little less potentially chaotic. I would bet stuff from web pages would be the worst, since table structure is often use to achieve a look rather than purely semantically—cells with nothing in them but spacer graphics and that kind of stuff.

I have had good luck using the table converter (I don’t think I ever had to drop to using a syntax table in the user manual—where on the other hand there were even times where I had to use a raw LaTeX table in the v2 manual because even MMD wasn’t good enough), but I do just make my own tables from scratch right in Scrivener, and I do so with an awareness of them being used to generate MMD. Given how finicky Cocoa tables can be though, I sometimes do get the inclination to just go back to syntax sometimes! :slight_smile: Do you use any coding editors with Markdown plug-ins? Most of the popular ones have something that can be used to clean up pipe table spacing with a shortcut. With Sublime Text I use Markdown Table Formatter. It can turn something like the following, which require minimal effort to compose:

|first column|second column|third column  
|--|:---:|---:
|Oranges|N/A|81.75
|Apples|Good|1.25

…into this, which requires minimal effort to read:

| first column | second column | third column |
|:-------------|:-------------:|-------------:|
| Oranges      |      N/A      |        81.75 |
| Apples       |     Good      |         1.25 |

By and large the latter is the approach I take with rich text source. I convert it, if necessary, using Scrivener, Pandoc or whatever is most convenient and best to use. Then I know I have a copy that will be useful to me in taking excerpts from, that I can archive and never have to worry about fonts being installed or what have you.

For composition, I’ll use Scrivener’s RTF features where I feel they benefit the process. For a simple list it can be nice to not have to number them out as I type (and battle OCD issues if I insert a number in the middle, even though I know the numbers don’t mean anything!). For tables, they are certainly easier to edit, again for OCD reasons. I could edit either of the above tables, but with the second one I would feel compelled to spend time moving spaces around after changing “N/A” to “Rotten” (or cut/paste into Sublime/execute tidy/cut/paste into Scrivener, depending upon the scope of the edits).

Of course using the RTF stuff isn’t all rosy either. In having used them extensively in the user manual, I have come to appreciate just how touchy and fragile they can be. I’m not used to having to worry about when I press return, or if I have the cursor in the wrong place if the whole thing is going to blow up and require reformatting. Or Undo is going to just explode the table. Sometimes they get in the way where otherwise they can be a time saver.

And no, none of that is a knock on Scrivener—if anything it’s on Apple’s apparent neglect of the text engine and like you say, RTF’s idiosyncrasies beneath it all. These are limits that define what Scrivener can do effectively—again stuff that is hard to document though, without going into excruciating detail.

I guess that’s part of what I like about the workflow in general though. I don’t have to use them all of the time for every single list. If a list corrupts itself and I lose my mind over it, I can just nuke the formatting and do it the “old fashioned” way. I don’t have to convert the rest of the project just because that one list doesn’t work well or I get fed up with trying to coerce it. I can use a manually typed in figure here, and Scrivener’s linked image + caption styles elsewhere. Even if I only seldom do that, having that option means overall less frustration with the writing process. I know that in the end, I’ll have a way of doing what I need to do that involves no more wizardry than a few punctuation marks.

The more I ponder this, the less I think a change to the manual is appropriate. I started with a bunch of text sourced from many places, so it’s bound to be too messy for most converters. Mostly, I’m thinking I need to compose a blog post at some point, once I’ve figured out what I’m saying, and who my target audience is, because it’s very much a situation specific to someone assembling a document rather than writing one from scratch.

I again think this comes down the tables that were converted to RTF. The GDocs were filled with tables that were formatted prettily, in ways that can’t be accomplished with MMD tables, so the RTF->MMD->LaTeX conversions would fail. Because RTF is so opaque (and I’m not at all willing to delve into the source code behind the format), I found it impossible to debug what was wrong with the tables in that format that cause the MMD to be malformed and thus the LaTeX to be wacky.

One example of the problem with the tables in Google Docs was that they had entire paragraphs of text in cells. As far as I could tell before throwing up my hands and converting them to lists (which used bullets that Latex didn’t like so that I just used asterisks and tab indents instead), there’s no way to have a single cell that is sized appropriately for a page but contains multiple lines of text.

Speaking of tables…

I tried that, but in order for those to work, you have to have good MMD formatting to start with. Also I’m not in love with monospaced fonts, and finally, when tweaking the contents of a table, one must adjust the number of padded spaces, often for an entire column, to make things line up.

I just used a tab in front of every pipe symbol, and adjusted the tab stops when I needed to to make them easy to read in Scrivener. That allows for proportionally spaced fonts and editing the contents of cells with less tedium. Most of my tables had all the same column headers, so the same paragraph settings worked for most of my tables. After converting a few tables, I created a style that also functioned to mark the text as pure markup in the compile settings. (Have I mentioned that I’m a total convert to Styles in all software that implements them now, because of Scrivener? They’re so handy for a variety of things, including their intended purpose… occasionally. :wink: )

One of my lessons learned was that Pandoc doesn’t convert tables from RTF to the style of MMD table that Scrivener uses. If I specified markdown_mmd as the output format, I got HTML tables. If I chose one of the other markdown outputs, I would get a different style of plain text borders that seemed to accommodate multiple lines per cell for the sake of fitting tables into a narrower editor. I finally just exported from Scrivener to DocX and then used Pandoc to convert to markdown_mmd (if I’m recalling correctly), and that mostly worked as a starting point to cleaning up the tables.

Edit: For a hot second, I considered transitioning to compiling with Pandoc for the (potentially?) more sophisticated table formatting of an alternative MMD input format, but then I remembered that I wanted to finish getting this document formatted for LaTeX->PDF sometime this decade. :unamused:

[size=120]Tables[/size]

Large cells, and multi-line cells in particular, can be a bit of a learning process with LaTeX. Something that will help a lot is learning the tabulary column types and tuning your tables in the .tex file after compiling. For example, Table 23.2 in the user manual comes out looking like this with stock MMD:

[size=80]Yikes.[/size]

The tabulary line for that is:

\begin{tabulary}{\textwidth}{@{}lll@{}} \toprule

The ‘l’, or left aligned column type, has no word wrapping function. I can do much better with this simple tweak, changing the third ‘l’ column to p{9cm}. ‘p’ has word wrapping, and I can set a width on it as well:

\begin{tabulary}{\textwidth}{@{}llp@{}} \toprule

[size=80][/size]

As you can see by comparing the final table design in the manual, I’m doing a little more than just throwing a p column into the mix. Have a look at the makecell package, as well as the array package, for more advanced table control. You can create your own column styles with the latter, which is what I use to achieve ragged right formatting and centred vertical alignment. With the former you can use the \makecell command to create a multi-line environment, where you can use the standard “\” to break lines (ordinarily that will break to a new table row in a table environment).

Of course we’re well outside of what Markdown does at this point. :slight_smile: That is precisely why I even had some raw LaTeX tables in the older manual. Another thing I did with the old manual is manually fix my table column designs every time I compiled. I had a checklist marked by table caption and I’d just go down copying and pasting. It was a pain, so I came up with the system I have now for the v3 manual, which is a combination of Scrivener metadata and post-processing. If you grab a copy of the user manual scriv and jump to the “Markdown Compile Types” table item in the binder, open the Inspector and check out the latexTable custom metadata field. The Section Layout used to format tables will insert this line as a comment above the table as a prefix, using a standard format. The post processing script looks for the specific pattern of text used to print these column formatting overrides and stores them in the loop, until it encounters a table caption matching the given override. It then looks for the stock MMD tabulary table and replaces it with the custom one.

Various tricks like these have turned what used to be a 40 minutes post-compile checklist into what I have to day: the only extrascrivener task I need to take is a batch conversion of the high-res PNG output to JPG, for a smaller PDF. Everything else is fully automated and I could even go straight to PDF if I wanted to (I’m not ready for that though, I’m still going through label and badbox warnings).

Fair enough, when I go through phases where I’m using a serif font for writing, I’ll still stick tables into a monospace style to keep them tidy (and yes, thank goodness for styles). But if you prefer tabs, I believe MultiMarkdown Composer uses tabs for table formatting, and if you copy from it using its Copy as RTF command, the tab stops will come along for the ride. But it sounds like you have a good system set up at this point.

Ha! I hear you on that one. I’ve been considering making the Great Switch one of these days as well. For me the big draw is being able to create my own output filters with Lua scripts. Some day.