Better Endnote integration: format Endnote refs right in S

I do biology/biotech research and write papers for science journals, and am loving Scrivener - all but one problem. This is a major issue that keeps me from recommending it to all my scientist colleagues. The problem is the integration with Endnote. I use Endnote to manage references. To generate the bibliography, I have to export to Word, and then format the list there. This would be bearable, except that, based on what output style I choose, the resulting text is of different lengths (references like "[1]
vs. “(Author et al., 2014)” take up different amounts of space). This means that I then have to edit the Word document to move things around to better fit onto pages (figures etc.). Once I’ve edited the Word document, I now have a choice to make: I can either make this Word document my Master Doc, and only make changes on that, or make changes back in the Scrivener project, but then I know that I will have to recompile and move everything around again. There are other Endnote-related issues as well, which end up as changes in the Word doc. The problem with changes in the Word doc is that they will be lost if I recompile, or I have to keep 2 parallel sets of edits (in Word and in Scrivener). I would love for Scrivener to be the only thing I ever edit (the Master) and for Word to be just a temporary format into which I compile for others (journal editors and such). But Scrivener doesn’t allow me to replace my citations with formatted Endnote references right inside a Scrivener Project. So, my wish list item concerns better integration of Endnote so that it works like it does in Word.

Sorry for the dumb question… why? If the piece is complete enough to compile to Word, why do you need a new “master” version? Once it’s edited and submitted, it’s done. If you’re not going to submit, then it doesn’t matter (and why are you doing all that cosmetic work on something you’re not submitting?).

Asking as a former academic, and long time Scrivener user, who is struggling to see the underlying problem. :blush:

I agree. I find this to be a major drawback with Scrivener. I am in the humanities and use Bookends and Mellel, but the problem is the same. Several colleagues have considered to begin working with Scrivener but changed their mind when they found out that there is no integration with a reference manager.
The thing is that you need to revise a manuscript several times. You submit the manuscript, you get it back, need to revise both the text and the references, send it back, get it back, revise, and so on. Sometimes you even have to submit a manuscript to several journals with different reference styles, before it is accepted.
If you have to finalize the manuscript in another program (Word/Mellel) already at the first step of this sometimes long and tedious process. Then Scrivener loses its potential for the last part of the work. The revisions can be substantial and you cannot know beforehand that your submitted manuscript will be accepted as you have submitted it. In effect, Scriveners lack of integration with a reference manager has made me use other programs for the final stages of work with several academic papers and books. It is a shame, since, besides this issue, I find Scrivener to be the best program for academic writing — by far.

I really don’t think this is something that will be added in the near future or in the mid-term. For a start, there are several different reference managers - our users use Zotero, Bookends, Endnote, Sente and others - all of which would require separate bodies of code that would need maintaining. But then there’s the fact that Scrivener is different from most word processors - it is not a single chunk of text that can be processed, but many different text documents. Where would the bibliography even be placed and how would it be formatted and so on?

We have many academic users, and using temporary citations that get processed after export works for the vast majority of them. I’m not sure I see how, in the case of Scrivener, processing the citations inside Scrivener itself confers any advantages. Scrivener is primarily a drafting tool, so the idea has always been that you create your draft, process it with your reference manager, and then tweak the final draft - or revise it - in a word processor.

I may well revisit in the future, but I’d need to research it thoroughly as I’m not that well-versed in citation managers, and given the number of features and enhancements already on the list, it’s unlikely to be something I’ll be able to look at in the next year or so.

All the best,
Keith

This is an example of where having an Applescript API in Scrivener would make integration much easier…

The only reason Endnote integrates with Word is that the developers of Endnote wrote special code to do it. Not the other way around. It is not clear to me that an outside developer could have done what they could do in-house. So, I think this one is in the reference manager’s wheelhouse.

To be clear, if you examine the issue mlevin is pointing to, you will see that his wish could only be satisfied by having i) the functional equivalent of Endnote’s CWYW plug-in for Word together with (I might add) ii) accurate WYSIWYG page layout in Scrivener.

–Greg

P.S. Would love to see some Applescriptability though!

AppleScript is one of those things I’ve always wanted to add but have just never had the time, and I can’t foresee a time when it will move up the priority list enough to get implemented, I’m afraid. I could spend a year or so trying to expose features to it, there are so many!

Indeed, I took a look at Endnote’s site and see they have an API, but it’s written in C and C++, whereas I am strictly an Objective-C coder (and no doubt one day a Swift coder, sigh).

Where would the bibliography even be placed and how would it be formatted and so on?

I’m certainly not an expert on how these things work, but it seems to me that it could be added in a special Document in the Binder called “References”, and it could be formatted like it is in Word - to whatever style Endnote has chosen for formatting (I think Endnote produces the formatted output).

tations inside Scrivener itself confers any advantages. Scrivener is primarily a drafting tool, so the
idea has always been that you create your draft, process it with your reference manager, and then
tweak the final draft - or revise it - in a word processor.

it's fine, except that sometimes Endnote requires us to make changes that then have to either be made again in the Scrivener project (if we intend on using the Scrivener version as the Master) or, we have to abandon the Scrivener version and just move forward with the Word version, which is unfortunate because I like Scrivener.

I am basically with nom on this one. As an academic and an Endnote user since version 1, I cannot even remember a time I had to format a paper first in one citation style and then another. So, maybe once or twice long back? And the prospect of needing to do that regularly seemed sort of central to mlevin’s issue. So, I can’t really relate. Guess I am glad right now I did not go into bio/tech.

Nonetheless,

I did some thinking on this problem and FWIW I fancy I got pretty far in developing a low-octane, low-coding way of solving the brunt of the problem. But… Endnote could not deliver. There appears to be no way to programmatically ask Endnote for an in-text citation (as opposed to a temporary citation or a bibliographic citation). I guess there are good reasons for that (because in some styles it is context sensitive, so outside the formatting of a whole paper, would not make sense). Maybe Endnote’s API would have some tricks, I don’t know.

Anyway, just to say, but for that hang up, I think I found a route whereby the road to pretty-okay integration was not looking bad at all. Just saying.

–Greg

P.S. I had also worked in some gratuitous Tom Stoppard references.

Some helpful comments here:
https://forum.literatureandlatte.com/t/comparison-to-other-word-processors-and-writing-tools/68/1

And the interweb is littered with blog posts on using Scriv and reference managers, which might be useful to the OP.

Fair enough, I can appreciate that other issues are more pressing.
Of course there are ways to work around this. I write in Scriv and then tweak/revise in Mellel + Bookends. But still integration would make Scrivener more attractive for academic uses. I have had several students/colleagues who were tired with Word and asked me for alternatives. Most of them have given Scriv a try but have ended up with Mellel because of the poor integration of Scriv with a reference manager.

Don’t get hang up on Endnote I used it for a couple of years and found it buggy in comparison with Bookends and other reference managers.
For what it is worth, I couple of thoughts:

  1. Why not put the the bibliographies created when Scriv syncs with a ref manager in a new folder in the binder and let every new sync with a ref manager produce a new text in it?
  2. Bookends has a very good support. They answer questions fast and accurate. Contact them. It integrates with rtf-format. Furthermore, there is a function in Mellel called ”live bibliography” that may be something to have a look at for inspiration (but don’t ask me about the technical details of it).

HA! Yes, you’d have one of the biggest applescript dictionaries around!!! :smiley:

I’ve gone around and around on this issue too. What I currently do when I use Scrivener with an academic manuscript is to use EndNote temporary citations in the draft (i.e., in Scrivener). Once you get used to these, they are no big deal, it’s basically {Firstauth,2011,pages}. I use pages, like 123-134, instead of reference numbers, like #3213, in EndNote because then if I’m working with a co-author, the page numbers will be the same whereas the reference number is specific to the EndNote library. There are modifiers you can add in, like {e.g., \Firstauth,2011,123-134} and so on. It’s in the EndNote documentation. To add a temporary citation to a Scrivener document, either type it in or just drag in the reference(s) from EndNote.

Anyway, I use this format only until it’s ready to submit somewhere, at which point I export it to Word format, add the relevant Word styles and clean things up, format the references, and set up the bibliography. This takes a while, I definitely am burning bridges when I do it. It’s possible to go back to Scrivener after that, but it’s not really advisable. Normally, after converting over to Word, the old Scrivener project just goes into the archive so I can refer back to it if I want to.

I also tend to used scripts to set up tables rather than entering them into the Scrivener project or in Word, because a script can more easily extract or compute data from the analysis files. I create the tables in HTML, which is easy to script, then import the HTML into Word and clean up the layout. Anyway, if the table needs references, you can type or drag references into the script and the resulting temporary citations will get formatted with the rest of them once you’ve moved over to Word. This also works fine for figure notes. If you need references inside the figure itself, I can see where that would be difficult. I’d suggest inserting letters or numbers into the figure and then defining them in the figure notes.

As for issues with changes in spacing due to different reference formats, I just bite the bullet: if you’re changing references formats, especially between a number style and an author-date style, expect to spend some time at it, because spacing is the least of your worries. If you have, for example, "Smith and Jones (1945) conducted the seminal…”, you aren’t going to be able to keep “²¹ conducted the seminal…”. It just won’t fly (although I have in fact seen that kind of rubbish in print).

a side question if you will: can you really avoid using the reference number? How does Endnote then match it up? I thought it did it by reference number, especially because if you change the number (say, by deleting the reference and re-importing exactly the same one from PubMed say, and let it get a different record number), Endnote gets confused and makes you tell it what the correct reference is whenever you try to format the bibliography. It sounds like that problem could be avoided if it’s possible to not use the reference number at all, but I’m not sure how that can be. Could you shed any light on this?

I am guessing that gshenaut’s situation is one where you have multiple author’s who use Endnote and are working on a joint text. If they are all inserting temporary citations from their own installations of Endnote, the ref numbers in the temp cites will be all screwed up. None of them could process the paper correctly in the end. So, in that special situation, gshenaut’s solution is that none of them should leave the ref number in the temp citations they insert. When you run Endnote, it will try to resolve any temp cites without numbers and will prompt you for confirmation. You are right ref numbers would save being so prompted, but in gshenaut’s co-authoring situation this cannot be avoided easily. I think what gshenaut said about including page numbers was just misleading about the situation – there is no way that the imputation of page numbers in the temp cite would help Endnote resolve the reference. (In fact, one could not use it as a alternative to ref numbers even if it did help Endnote, because whether a reference should cite page numbers or not is dictated by the sort of reference one is making.)

best,
Greg

that’s what I was afraid of… Ok thanks!

Nothing to fear really!

If you compiled to rtf, Endnote will process that and just put up a single confirmation dialogue – so long as each of your temporary citations w/o ref num had a unique match in the database, you just Okay that and go. The only possible hassle would be cases where Endnote found multiple references for the same author and year – they you need to tell it which reference is meant.

If you compiled to .docx, Endnote will process the document and you will not even get a confirmation dialog if all your references have a unique match in the database.

–Greg

P.S. I am not sure how gshenaut is working with temp cites with the combination of page numbers but no ref nums, like {Adams, 1975, p. 123}. My Endnote just chokes on these (i.e., cannot find the reference), though it will find {Adams, 1975}, and of course also {Adams, 1975 #2117, p.123}.

If you go to EndNote Preferences > Temporary Citations, you will see an option “Use field instead of record number”. When you check that option, then you can choose among Label, Pages, and Accession Number. Choose Pages.

The only gotcha I’ve encountered here is with references imported from those despicable journals who insist on writing page ranges like 1234-56 instead of properly as 1234-1256. Note that EndNote interprets it correctly either way when formatting references, so your references section will always read 1234-1256 if that is how the current format specifies it. But when looking references in your library, EndNote just uses string comparison, so 1234-56 will not match 1234-1256 in that case.

Also note that the ‘#’ is what makes a record number a record number. So don’t put it in front of page numbers, that won’t work. The format for page numbers is {author,year,page-numbers}.

Interestingly, when you have the option checked, EndNote will also use page numbers when you drag in references as TC (in fact, that might be the only actual thing that is changed by checking the option).

Thanks for the explanation! That is an option setting that I did not know about.

It also resolved another puzzle I had about the strategy for me: the page span being used in the temp cite specification is the full page span of the article, not the pages one might be referring to in the actual citation. That makes sense. And it would be something that Endnote could indeed be using to resolve which among several references with same author and year is meant.

-gr

A little late to the party but, how do you get “Smith and Jones (1945)”? I’m only able to get “(Smith and Jones, 1945)”