Ready for a brand new workflow! Where do I start?

Scrivener Markdown & LaTeX
1

I just finished a long and long overdue paper for which I scraped together an insane workflow: Brainstorm, outline, and compose using roamresearch.com ==> Export to Roam’s weird markdown ==> (Make complex changes in Vim if needed) ==> Git commit ==> Bash script ==> Perl script ==> Pandoc (with various options and add-ins) ==> LaTeX (with a bunch of packages) ==> PDF.

Now I need to revise and reformat it for a journal that only accepts Word documents; so I’ll need to drop the glossaries package and some other LaTeX-specific stuff. And after that, I get started on my proposal and dissertation — and now’s my chance to escape my crazy workflow. It looks like elw2u16’s requirements are somewhat similar to mine, but here are mine:

  • content stored as text (so I can use Git meaningfully and Vim or Bash stuff when needed);
  • Scrivener for editing (because Scrivener, like Roam, is good for constantly rearranging outlines, notes, and draft material);
  • Zotero; and
  • the ability to handle cross references, numbering, and maybe glossaries in the markdown so pandoc can convert it decently to PDF and Word (and maybe HTML or LaTeX also).

The last time I used Scrivener was the last time I tried to write a book, about nine years ago. I see AmberV’s Default Preferences to Emulate Plain-Text is still pinned to the top of this forum, but I don’t see the zip file. The approach I should start with is probably the “Purist” in Section 21.4 of the Scrivener Manual, but I don’t see instructions for how to do that. I did make an attempt to use Scrivomatic, which nontroppo has clearly put a ton of time and careful documentation into. I agree with elw2u16 that it seems pretty complicated, and I’m currently stalled by a compilation error.

Can anyone recommend some simple steps to get me started with a new Scrivener-based workflow that fits my requirements?

elw2u16, are you still happy with your workflow? Have you had to continually invest more time into tweaking it? If you are happy with it, might you be willing to document it? I would really appreciate something like “Simple, Text-Based Scrivener Workflows with Pandoc for Lazy Idiots.”

Thanks!

2

I think what’s confusing me is just that I can’t find where to set up post-processing (as mentioned, for instance, here: github.com/iandol/scrivomatic#s … ing-script). But I just found in the manual, Section 24.22, that it’s only available in the direct sale version — which is what I thought I had, but maybe not?

3

I forgot how complicated and confusing I find Scrivener…

My eyes glazed over while reading these Scrivomatic instructions and I guess I figured it would all be obvious in the Scrivener UI, but no… :cry:

4

These two requirements are fundamentally incompatible. Scrivener’s native format is RTF, and there are no supported ways to directly modify a Scrivener project using Git (or anything else other than Scrivener).

Katherine

5

On your Scrivener menu, do you have commands like “Register,” “Deactivate License,” or “Check for Updates?” If so, you have the direct sale version. If not, you have the Apple Store version. You might also check the App Store software’s Purchases list.

Katherine

6

Thank you for the responses, Katherine.

To your first, maybe I misunderstood this point:

I understood that I wouldn’t be able to use RTF features for formatting, but I may have gotten an overblown idea of how Scrivener supports writing in markdown. So, is this passage suggesting that I can edit text in a plain-text editor, but then I’ll have to paste it back to wherever it goes in Scrivener?

To your second response: I’m using a trial version of Scrivener 3, not from the App Store, and under the Scrivener menu I see “Enter License…” and “Check for Updates…” but not the others you listed.

7

My own copy is registered, so I wasn’t sure what commands an unregistered copy would have. But if you have the trial then yes, that’s the direct sale version.

Scrivener’s internal files are always RTF. If you want to use a plain text editor, the best solution would be via Scrivener’s Sync with External Folder feature. See Section 14.3 in the manual for more information.

Katherine

8

That’s great. I set up the sync with external folder thing, copied in some markdown files, and Scrivener imported them fine, and I’ve tested that updates work in both directions.

So now I’m back to my original question about how to deal with markdown files in Scrivener. There are built in formats for markdown, basic pandoc, and markdown to PDF, etc. But I tried those in various ways, but Scrivener is treating my markdown files as plain text.

I was under the impression from reading a few chapters of the manual that I would be able to view files/sections (whatever they’re called) in Scrivener as RTF, without seeing the markdown marks, but I could somehow make sure that they didn’t contain any formatting that couldn’t be saved as plain text markdown; I could use Scrivener styles for fancier stuff; and that I could set up the compile options with pandoc postprocessing to turn the whole thing into Word, LaTeX, PDF, HTML, etc. I mean, apparently all that is possible, but I really don’t get how to do it.

9

The Scrivener editor has no knowledge of Markdown or any other markup. All relevant processing is handled by the Compile command.

Thus, the Scrivener editor has no ability to “hide” or otherwise interpret markup commands for you. What you type is what you will see, just as if you were using a plain text editor.

But yes, your understanding is otherwise accurate, in that Scrivener’s Compile command can use both Scrivener’s own Styles and markup that you supply to create an output document in a variety of formats.

Katherine

10

Hm I understand, without understanding the basic structure of how Scrivener goes from the RTF files structured in the Binder, through the compile transformation, to an output file; it is not entirely intuitive how the next steps in any workflow may occur!

You really should try to understand “Section-Types” and “Section-Layouts” as these are the fundamental underlying mechanism for any compilation in Scrivener. The tutorials and manual should get you most of the way there.

What I was trying to express in that paragraph is this: I want to ensure the front-matter is NOT transformed by any compilation tools. To do that we need to tell Scrivener NOT to do anything to that document, to treat it “AS-IS” (i.e. please don’t change anything). How can we do that? Scrivener’s compiler uses Section-Layouts to flexibly control what happens to any binder document, including making sure we treat a document “AS-IS”. To apply that Section-Layout, we usually need to assign a TYPE to the document, in this case to explicitly “Type” it as “Metadata” so that it should get “As-IS” applied to it. These are Section-Types, like Chapter or Part or Metadata. This is a Project-level setting.

More generally, I think the Purist or Hybrid workflow (§21.4 user manual) can work great depending on your priorities. I personally prefer the Hybrid approach, as the benefits of Scrivener’s styles far outweighs the downsides for me. I do reliably advocate to go with Pandoc, especially if you will have academic references to deal with. A simple Pandoc commandline in post-processing, as long as you use absolute paths will do most of what you want.

So if you go with a hybrid approach: Create a new Scrivener project, Write a single sentence, and create a new style called Strong (make it bold in Scrivener), apply it to a word in that sentence. Then in the compiler, create a new compile format for Multimarkdown output. In the Styles section of the format editor add your “Strong” character style and make sure ** and ** are prefix / suffix. In postprocessing, enable pandoc and add your Pandoc commandline. Compile. That is a simple Hybrid Markdown workflow using Pandoc. Extend it? Add a citation [@authoryear] to your sentence and add the bibtex file you exported from Zotero to the pandoc command-line , voila, output with bibliography. Cross-referencing can use either Pandoc or Scrivener routes (I use Scrivener’s).

Also remember, once compiled, you can GIT version or otherwise revision your markdown file, which can still be invaluable, even if it isn’t exactly the same as the Binder source. When I submit e.g. a grant proposal, the markdown is revisioned, and I know quickly what changed if I repurpose that project later on (I also use named snapshots within Scrivener).

For me, my workflow grew organically. I started using markdown directly, then upgraded to Pandoc, then tried to find a way to organise different sets of Pandoc settings without lots of commandline tinkering (pandocomatic), then when Scrivener 3 was in beta beta I switched to using Styles for almost all markup. The one limitation of the post-proccesing panel in Scrivener 3 is it can’t access the user’s path — so I created a small wrapper to set up the path to reliably find any required tools. I use several Pandoc filters to power-up how metadata gets transformed into author/affiliations, this can be used by anyone, it isn’t specific to my workflow at all. I sometimes regret giving my workflow a catchy name, it makes it seem like a separate tool or something, and it really isn’t, it is just a series of choices of how to go from A to B. By the way, I’ve replied to your issue, it is most probably a Ruby install issue.

11

Thanks so much nontroppo and Katherine!

It’s all vaguely, slowly starting to come back to me. I couldn’t find the post-processing stuff because I thought the format editor was the box that comes up when you select File==>Compile. Doh.

So, can you tell me if I’m getting this right, finally? Scrivener will convert bolded text to ‘bolded text’ and other basic stuff with the MMD format, and you can make new styles to convert any combination RTF formatting features into plain text + custom prefix/suffix. But all of this is a one way process going out. ‘bolded text’ coming into Scrivener will be treated as plain text with some stars around it, whether in comes in by pasting, import, sync, or whatever other methods there may be. So syncing files as plain text totally obliterates any styles or other RTF formatting as it comes back in to Scrivener.

I’m not sure whether it took me this long to get this because Scrivener is complex and the manual is long, or because it’s hard to understand something when you really want something else to be true.

So, do you think the day might ever come when Scrivener supports a two-way conversion process? I.e., allowing styled text in (say) a synced text document to be converted back into a user-defined style?

In the meantime, I might sadly have to chalk my last few days up to wishful thinking and get back to looking for text/markdown-based tools that support a more fluid brainstorm/outline/compose/revise/compile process.

Thanks again for all your help.

12

Well, the Import and Split command uses Markdown headings to split the imported document and can convert Markdown to rich text. But doing so obliterates the markup, so it’s probably not a great option if you want a Markdown-based overall workflow.

Katherine

13

Correct, this is mostly one-way street with a few exceptions. There are some tweaks if you go the MMD/Pandoc route, like you can add a unique ID to each markdown section so you know which file in the binder it came from if e.g. the title changes.

Personally I never really need to round-trip markdown text (what is your use-case?) BUT if I could have my perfect “simplified” Scrivener, I would ditch Apple’s legacy RTF editor component, and have the Scrivener editor style markdown live. The compiler would use Pandoc for all output formatting. I would then have the power of the Binder / outliner (I never use the corkboard), plus the flexibility of plain-text and the superpowers of the compiler. But I am very doubtful that KB will ditch RTF, and it would be really a tough cookie for KB to modify RTF to support such a flexible conversion (he did make a “simple” RTF<->Markdown convertor, but as you can see it is quite limited)…

Have you tried Ulysses? e.g. leolabs.org/blog/thesis-in-markdown

1 Like