Scrivener, Latex and the researcher workflow

Hello,

I have been really enthusiastic the first time I discovered Scrivener as an authoring software for my researches. However, after spending many hours reading the documentation, the forums, watching demo videos (the ones on litteratureandlatte and others from Youtube) and trying to make it work for me, I came to the conclusion that it is simply not evolved enough for me. Writing on Scrivener would be more of a hassle and a loss of time. First let me introduce myself and explain what I was hoping from this terrific software (it is, really! just doesn’t seem to fit my needs).

I am PhD Student in computer science. Apart from coding, I spend most of my time writing about my researches. For this last task, I value a software letting me write in an uncluttered interface and not handling the formatting issues upfront. Once I’m done, what I look forward is a way to publish my writings easily in the fastest way possible. Usually I have to publish different versions of my work (mainly for reviews) while still authoring and modifying my papers. The last step would involve sending the final draft to a journal and, in my field, most journal require the paper to be in Latex.

So I tried to write exclusively on Scrivener, organizing my thoughts on documents and Folders within Scrivener. This worked beautifully for me. Then I tried to publish my work for reviews. The options I had were to publish it in RTF or MMD-Latex format. The RTF option was unusable for me since I had to work on the formatting as many times as I publish a new version. The Latex seemed to be a bit of a beta version were I had, for example, the titles numbered both by Scrivener and by the Latex compiler. So I also had to manually change the Latex output. I also experienced difficulties with the included graphics, having Scrivener generate Latex unfriendly file names.

One may say that, if my objective is to have my writings in a Latex ready format, I could simply forget about Scrivener and write directly in Latex. But, IMHO, writing in Latex is a challenge and one can’t focus on it’s writings if he’s focusing on latex codes and having to strive to get a simple table (that would look as code on the page until you compile the document…).

I also had many struggles getting my figures and tables numbered and captioned. Scrivener simply do not offer this feature (while Latex does). I had to play with the <$n… trick and all I managed to get is numbers and captions not associated with the image or table and numbered in a weird way (this happens if, say, you reference an image while there are other images between the reference point and the image itself).

I am in an early research stage right now, I really hope a new version of Scrivener will make it’s way and let me use it as intended before I start writing my final thesis. For now, I am regretfully falling back to MS Word… All I hope for now is that my review will serve as an insight for future versions. I would gladly be part of a working group if needed!

Thanks and keep the good work guys!

Hi,

Thanks for trying Scrivener, and for your polite and constructive feedback - it’s appreciated. Out of interest, have you tried downloading the Scrivener project used to create the PDF user manual? You can find it on our support page (literatureandlatte.com/support.php). If you go to Help > Scrivener Manual, you will see the PDF file that this file generated - a PDF complete with images, figure references, a table of contents and much more. The whole manual is written in MMD and compiled to LaTeX to generate the PDF, and so taking a look at the Scrivener project used to create the manual may give you a better idea of what is possible - certainly, from what you describe it sounds as though some settings just need tweaking and that you should be able to do everything you want. Ioa (“AmberV”) is the resident MMD expert, but if you have any questions, fire away and I’ll point him in your direction.

All the best,
Keith

Thanks Keith for your response!

I had a look at the manual project and yes this is exactly what I want to achieve! However, after looking at the contents, compile settings, etc…it looks to me that writing the manual has been more of a programming exploit than writing in a straightforward way. Obviously, as I tried to explain in my original post, what I am looking for is an authoring software that could let me write with a minimal configuration effort, but enabling me to organize my thoughts as Scrivener does it in a wonderful way.

If I may narrow it down to a few features, here’s what I am expecting:

  • Be able to attach semantic information to tables and embedded images. One meta-data field could be enough, ideally two: one for caption and one for description.
  • Having a way to reference a numbering sequence without affecting the sequence. Ex.: I might use <$n:table:Blah> to number a table (insert it in the caption metadata of my table as described above) and then reference my table in my text with, say: <$rn:table:Blah>. The reference would only show the number without incrementing it.
  • Having the option to compile to Latex with minor effort: All i need to is to set the latex template, my bibtex file and compile. All the captions would be inserted as intended in the correct Latex tags.
  • Having the numbering of sections and subsections work as intended and not having my section titles both numbered by Scrivener and Latex. I must admit I still can’t figure out how Scrivener handles the numbering and TOCs so it might be my fault but, to say the least, it’s not obvious.
  • Having all the aforementioned features usable with minor effort with minimum special tags in the texts and minimum options to play with at compile time.

These are the minimum feature that would actually make me reconsider using Scrivener for my writings. I can understand that you guys specialize in fiction writing where all my issues do not apply. However you seem open to non-fiction writers as well by offering MMD-Latex compilations. However, if you want to win the scientific writers market, I think some efforts are needed and may having a native Latex compile (with appropriate options) would be better than relying on MMD?

My 2 cents :slight_smile:

HI,

You can already do exactly this:

[Table]
Table <$n:table:foo>

[Table]
Table <$:table:bar>

See Table <$n:table:foo> and Table <$n:table:bar>

Will result in:

[Table]
Table 1

[Table]
Table 2

See Table 1 and Table 2

Ioa is probably better placed to respond to the rest.

I had thought you were looking for some way of avoiding writing in LaTeX? Or do you mean you were hopping to compile rich text directly to LaTeX? If the latter, then that’s definitely out of scope, I’m afraid, at least for the foreseeable future, as that would be a massive undertaking. We do have a number of science and technical writers already using Scrivener, though.

All the best,
Keith

What you describe (bar the bibliography stuff) is what I have been doing in Scrivener for the last four years to produce software manuals and training course materials.

The Scrivener --> MMD --> Latex route does work.

I wonder though if you are trying to do too much of the heavy lifting in Scrivener, rather let MMD work its intermediate magic.

If you are using MMD there is no need to add the section numbering in Scrivener: MMD will do that. And if you structure the Scrivener project right then all the sections and subsections will fall into place, numbered according to their position in the Binder.

Then you can use your Scrivener document names to build your cross-references. E.g. if I’ve a section called Why I love computers, I can cross-reference that in the text, so "see ", turns into see section 4.3.2

That will also work for numbering figures and tables. And that all works through into the TOC as well. (There’ll be others who can tell you more about the bibliography side of things.)

The Scrivener manual might be a bit misleading as a starting point, as it pushes the MMD workflow to a level of customisation well beyond what I understand. But most of us don’t need to set a document up so we can compile different versions for Windows or Mac or web or print. We need to get a PDF we can email or print. Scrivener + MMD + Latex will do that.

If I understood correctly, one of the points the OP raises is that there’s currently no easy way to write in Scriv which is neutral between a) rtf and b) MMD → LaTeX as output formats. You have to choose whether to do your bold, italic etc. formatting the conventional way (for ultimately going to rtf output) or use MMD conventions (for MMD → LaTeX), and the same thing goes for numbering of examples and figures: there’s a Scriv way (described in Keith’s post) and an MMD way (described by MrGruff) and they don’t mix well.

I think that Ioa has said somewhere that one potential solution is that MMD 3 allows production of OpenOffice format (which is similar to, and convertible to, rtf, doc, docx etc.). But then that would mean forgoing a lot of the nice stuff that Scriv supports and using kludgy MMD formatting.
A nicer (but probably very difficult) solution, i.m.o., would be for Scriv to develop into a kind of neutral, semantically aware editor and frontend for the production of rtf, LaTeX or whatever. What I mean is that it would be nice to be able to just mark an element as bold, or blockquote, or image caption, and not have to worry at that stage about which format the final output will be in.

That would also put off a lot of users - all of those who like applying their own formatting rather than using semantic mark-up.

I didn’t mean that, Keith. I’m one of the ones who likes being able to see bold, blockquote etc. in Scrivener, and dislike MMD formatting because it’s hard to read at a glance. I just think that it would be great if the formatting in Scrivener could (sometime in the future) serve both roles.

Hi Keith,

What about this:


As presented in <$n:table:bar>.....

[Table]
Table <$n:table:foo>

[Table]
Table <$:table:bar>

See Table <$n:table:foo> and Table <$n:table:bar>

As a result, I’ll have:


As presented in Table 1.....

[Table]
Table 2

[Table]
Table 1

See Table 2 and Table 1

I’ve seen a post on the forum, someone was arguing that since you reference your tables in the order they are listed, the problem won’t occur. Well, I had exactly what I am describing in my text, not trying whatsoever to play with Scrivener’s limits. Is this really a limit? If so, wouldn’t be easy to have a distinguished code when we reference something? Let’s say: <$n:table:r/foo> with the r/ meaning I’m simply referencing to the table foo and Scrivener should not increment the numbering sequence?

This is how the text would look like:


As presented in <$n:table:r/bar>.....

[Table]
Table <$n:table:foo>

[Table]
Table <$:table:bar>

See Table <$n:table:r/foo> and Table <$n:table:r/bar>

And the compiled result:


As presented in Table 2.....

[Table]
Table 1

[Table]
Table 2

See Table 1 and Table 2

As for the rest, Yes I am trying to avoid writing in Latex because I find it cumbersome, counter intuitive and value having some formatting appear on-screen rather that code (ex.: for bolds, tables, etc. ). However, my final objective is having the final paper in Latex. All I’m saying is that it takes so much efforts and took me so many hours to get to that point and wasn’t satisfied with the result. Definitely, much of that time was used to learn how to do it so may be one easy thing you guys could do is publish a “how to” that explains the workflow for my “type” of customers? May be there is something I’m doing wrong. That “how to” could be the basis of discussions from were you guys could elicit more clearly our requirements…

I seem to remember that a workaround has been suggested for the problem you show (that is, needing the caption use of each placeholder to precede all cross-references to it). Namely, list at the beginning of the document all the placeholder tags you will use in the document, in the order you will use them. If you’ve ever programmed in Pascal, this may seem familiar…

My solution is easier. I just avoid cross-references in Scriv.

I would really appreciate it if someone could pull out a tutorial to do just that. A tutorial that would go from authoring the text in Scrivener, compiling in MMD-Latex (along with all the appropriate compile settings) and the document compilation into PDF from Latex. If citations handling were included (using bibtex) that would be wonderful! I think this tutorial would be very valuable for people like me and sould be located on litteratureandlatte’s website as an official reference. I hope that there’s a way to make it work my way because Word is the worst option in my opinion. However, there is definitely a lack of documentation on this matter.

I found a couple of Youtube videos but they are based on outdated versions of both Scrivener and MMD.

One more quick post on the cross-reference issue:

This is a great idea. Any chance of this finding its way into Scrivener at some point, Keith?

:slight_smile: How is that a solution ?

I am aware of the workaround. However, my point by using Scrivener is to remove any clutter and source of distraction. When I’m writing, I want to focus on my ideas and material, not on “should I pre-declare my cross-references up in the document to have my figures numbered correctly”. Moreover, the compiled version should be as predictable as possible and one shouldn’t have to scrutinize his compiled file each time. Otherwise, we’re back at square one and Ms Word would stand the comparison with Scrivener.

Keith, I hope you understand that, through this thread, I am only trying to expose the problem as I think there is a problem. All I’m expecting is an acknowledgment of that, either being a limitation, a lack of documentation. Your willing to resolve this (I know you guys must have other market pressures) won’t affect my judgement about Scrivener that I still find it to be a wonderful peace of software that worth every penny I’ve spent on!

Can’t agree more with you! However, I think Scrivener is already just that! All the MMD code within the text are more of workaround than the normal process as Scrivener’s designers envisioned it to be. I’ve never played with Scrivener’s file format but I am expecting it to be neutral semantically aware that should be translated into any other format! This is why I was suggesting adding metadata to figures and tables. These won’t be in the text, but in metadata fields on the side. At compile time, these metadata would be included in appropriate tags in latex, or ignored in another format.

Well, that way the example numbers and table captions come out in the right order, at least…

anis797979, I agree, a tutorial on the Scrivener → MMD → Latex workflow would be a Good Thing.

I have thought about developing one before, as I know from experience that getting the process to work can be very frustrating, but time pressure and an acute awareness of my own lack of knowledge on many of the issues involved have held me back.

However, as the proverb goes

From that point of view, a guide from one toad to another (if you’ll excuse the characterisation) which describes the spikes in the S>MMD>L workflow might be useful.

I’m going to consider it now, quite seriously.

Thanks MrGruff!

You can count on me if you need any help! Especially, I’ll be glad to review it and point you towards any missing insights that could be helpful for people in my position!

Anis