Will Scrivener take Citation Seriously?

Scrivener Wish List
, ,
1

It was suggested a lot of times integration with specific Reference Manager and personally I think that the decision to do not integrate with a specific software is the best one (for obvious reasons for any developer), even if the software is very popular (like Zotero) and “free”.

But, this does not mean that Scrivener needs to leave Citations apart. Citation is a critical thing in workflows, especially academics; journalists that need to deal with sources; etc. Thus, I think that it is unanimous that citation is a core and basic features for writing and can not be leave for third-party solutions or crazy workflows that I see here (with all sort of very low quality and suboptimal workarounds).

Actually there are a some softwares that integrate citation capabilities with their environment, for example: Essayist https:// www.essayist.app , Author https:// www.augmentedtext.info/author , Citer https:// www.notewrap .com/citer/.

The point is that I am not asking for Scrivener to be a reference manager, far from it. I am asking for Scrivener to have Citation Processing. I would expect to import a library in Scrivener (.ris or .bib) and pick-up some citation (when I am writing) that Scrivener would process this in the output.

There is a lot of open-source Citation Processors (generally called citeproc), some of them with permissive MIT Licenses (Here) and can be integrated to Scrivener (also, for example, the citeproc-py are already integrated in commercial products). All them support CSL Standards. There is also o lot of parses of .bib and .ris (xml) files around there, for example, this one that convert directly from .bib files for integration with markdown environments.

I know that this will need adaptation and will give some headache for the developers, but, again, it is not a innovative work (a lot of people pavemented the way) and Citation is CRITICAL for any serious writing. Given these two things, besides I like very much the app, I think that this HUGE setback is an instant deal-breaker. I really do not want to have the headache to work more on workarounds and integrations than in writing my manuscripts.

As I did not find a roadmap of the app, I would like to ask if the Citation Processing is in the roadmap and the team will sometime consider it seriously.

PS: I think that any academic/journalist needs this feature as hell. I would not mind to pay an upgrade version Scrivener 4 just to have this feature.

EDIT: I just can put two links, so I edited some content.

1 Like
2

I have previously wished for citeproc integration here:

There is some feedback from @KB on what may be needed to integrate. There are technical details that need to be addressed. All of the citeproc engines are not directly compatible with Obj-C, so a bridge is needed. in 2018 when I wrote that wishlist rs-citeproc was looking very promising but that project died. The most up-to-date citeproc engines are written in Javascript and Haskell. I suspect js-citeproc would be easiest to integrate as Apple does use JS via webkit and JSX already.

Now let me caveat: if you use Pandoc, you have citeproc built-in and the Scrivener+Pandoc workflow offers a fully functional citation workflow. The quickest fix would be to replace MMD with Pandoc and with some minor tweaks to the compiler, citations would be natively supported. The previous argument against replacing MMD with Pandoc was a packinging one, Pandoc is ~40MB where MMD is ~1.5MB

TLDR: I still fully support the request to have a citeproc engine as part of Scrivener, either via switching to Pandoc (with many other benefits), or using js-citeproc. I would’t use it personally as my Scrivener workflow already works perfectly with citations via and external Pandoc, but having it bundled would help less technical users.

3 Likes
3

You can already use Scrivener 3 + Pandoc and get a powerful workflow; all my research papers use this, I get formatted PDFs or DOCX directly without any fussing for journal submissions… I prefer Bookends, but Zotero, JabRef etc. are also working well with Scrivener fine: see the wiki — Using Scrivener with reference software (wiki)

2 Likes
4

The point here is not, let’s say, technical skills. I have a strong opinion on the so-called “Reference Managers”. For me, they almost all are bloated pieces of crap. You see BookEnds, Zotero with tons of “wonderful” features, but lack very basic things, like hierarchical entries (multi-volume (- book), proceedings (- in proceedings), collection (- article), etc. I will not even enter on the topic about “format”, “citeproc”, etc. JabRef and BibDesk are just GUI for .bib files, so, that is why they are already much superior than Bookends and Zotero.

I made my own solution based purely on plain text management, that works in the terminal. But, I cannot use it on iPad.

For sure, I think pandoc works well on macOS, but, again, how could I replicate this in iPad? I want this feature to be hardcoded because I expect it to work properly across all devices. Also, from a structural point of view, I want to the citations to be properly dealt with Scrivener engine, like headings, footnotes, bold, etc. (as an tag in xml file not a plain string).

I did not see you thread before post here, but after read it I fell somewhat disappointed by the developers answers. Maybe they can change their minds after see the recent developments and how this feature is requested a lot here. Also, the confusion about reference manager and citation processor cannot be made. Let’s be clear, citeproc-js IS NOT the core of Zotero, but one feature of it. The core of Zotero is a sql base management (with information about files, annexes, annotations, etc.), while the citeproc-js is only used when Zotero scan a Word document or when you request to copy a citation on style.

I do not know about the internals in Scriviner software, so I cannot guess which citeproc is the easiest to integrate. This is something that just the developers can answer (and I have the feeling that their answer will be no one again).

I really do not know which software Author, Essayist and Citer use as backend, they are tiny programs with citeproc capacities. I suspect that they may be using biber [in Perl] the backend of biblatex and, by far (veryyyyyyyy far) the best citation generator on the present day. The “downside” (?) is that you need to format a biblatex style to use it. BibLaTeX has a lot of prebuilt formats and actually there is an citeproc being developed that aims to be a bridge between the crap csl and latex, here [in Lua].

(?) I doubt that this is a downside, because the csl is beyond crazy. BiblaTex is much more sane option than the crap of csl and all their duplicated and non-sense entries.

5

Well you certainly have strong opinions[1] :grin: I’m not clear why Pandoc can’t work on iPad if it was bundled by Scrivener, it is just a processor of text, already supports Apple silicon, and can sit in Scrivener’s app bundle doing its thing, taking JSON or BIB file alongside in-text citations and giving us a formatted bibliography in many final output formats…

Not sure why you want to attack CSL, it is easyish to implement and hack on than anything in the LaTeX world (as evidence by its uptake and the number of styles availabe), and CSL renders an order of magnitude faster than the hodge-podge of ancient scripts and code that is LaTeX. Suggesting everything else is bloated crap and holding up LaTeX with >4 GBs of install just to output a document[2]: pot, meet kettle – if you want to discourage the Scrivener developer from bothering to try to implement something, claiming the thing you “wished” to implement is actually “beyond crazy crap” is not going to help anyone…

  1. Bookends support link relationships, so Book ⬄ Chapter, Article ⬄ Review, Cited ⬄ Cited-by etc, as well as multi-level smart (live/dynamic) folder hierarchies. Works well to cite while writing on Scrivener for iOS too (my iPad and iPhone love Bookends). JabRef is far more than just a GUI for BibTeX, it offers some wonderful management features etc. Anyway, everyone has a valid opinion on what tool works best for them, but to call tools other people use daily bloated pieces of crap is most likely missing some perspective from your side. ↩︎

  2. Run Typst vs. LaTeX on the same document, Typst (which is just a ~10MB download, yet supports most of what is needed for academic work) generates a PDF >350x faster with less memory consumed… ↩︎

2 Likes
6

Scrivener is writing software. It isn’t an academic tool so I see no reason why citation should be something the developers of this long-form writing tool should be treating as a priority.

I suspect the answer to your question is “we created a space for you to think and write in, beyond that, your specific use cases are for you to figure out”.

3 Likes
7

When I mention the iPad I want to say that I cannot install pandoc in the iPad and use for post-processing. If Scrivener put pandoc in the same package, that would solve a million of problems on this area.
But they will need to do a lot of internal adaptions, I am afraid. And maybe here the things are backing off. Also, pandoc has native math render, so they could write tex for example, and output mathML directly for .docx. Well, the possibilities of pandoc, when used smartly, are very promising.

I do not know your field, my one is history and I work extensively with critical editions and other projects alike. Surely I cannot use Scrivener for my heavy projects (.xml is the way for encoding complex documents). But for papers and small books I never saw a such good writing environment, that is why I trying to push this. With citations, this software will become a true beast for academic work.

The CSL was initially an xslt transformation sheet for xml stored bibliography, that lately became the .ris format. For example, biber use internally .xml. Unfortunately, this crap becomes the industry standard (as the crap JavaScript, as you see, the mythical industry standard almost never means good).

The styles of CSL are just small modifications of each other, I could write dozens of biblatex styles just modifying one option here and other there. When I say that is a crap, is because of the foundations of this protocol. The entries types, the encoding, etc. Again, everything very far from any professional take with a lot of duplicates and other stuff. But this does not interfere on the implementation at all, because, again, Scrivener just need to provide an way to the citeproc read the csl (xslt) and apply it to the xml of the bibliography (and sort it). The quality of the protocol and if it is a crap (for sure!) or not, does not play any role here, since it is the so-called industry standard.

About the LaTeX, you are considered the full installation, with the majority of the size is due to fonts, documentation, etc. I have a custom install of luatex with every few packages that does not excess .100mb installed. LaTeX is full of abandoned packages and other things, but the core of latex, i.e, the core of tex is not crap, if you are a professional, you know very well what you need to do. But the csl, on the contrary, was born and evaluated as a crap.[1]

I do not think that this “discourage the Scrivener developer” because it is not just me that asked this feature. Also, as I said, csl is a protocol, the quality or not is not the concern for any implementation. Everybody use this crap, as the majority of the people use the crap Word from Microsoft (at least we can agree that Word is piece of crap?)

The feature of Citations is the missing thing is Scrivener, if they want to implement it without headache for further people asking for styles and etc. They need the csl protocol. If they priorize quality of output, they surely need to choose biber. For a commercial software, option 1 (csl) is obvious.

The Developers will evaluate the alternatives, but for this, it is better to make a critical review of the thing. In this case, what I want to show is simple: there is some softwares that do this with limited styles Bibliography (just the major ones: APA, Chicago, etc.) but better output (mostly in sorting entires). There are softwares that do this with any style (CSL), but lower quality output.

Well, at least we need to be clear here, reviewing critically the options. Because, for sure, Scrivener will not develop one citation software, but embedded one already done in the software.

  1. Your suggestion of typist is another thing. This software lost a big opportunity due to very wrong decisions on design and implementation. The removing of the concept of glue, the parse for internal use (they struggle in html export, unbelievable). This has consequences, for example, the implementation of line numbers as a query on the height of the characters in the page is a joke (tex implementation, on the contrary, is in pair with the line-break algorithm, as you can see in the lineno documentation, much more refined and professional, ideal for hacks and going around). But typst is not the topic, the unique thing that I can safely say is that they will a lot of hard work to even implement the basic of the basic of the core of tex and they did not bring any innovation (except compilation time). The software is years behind any professional application (dissertations and simple papers are the extreme bottom line of any typesetting system). ↩︎

8

Exactly, it is a writing software and citation, as I said above, is a critical aspect of writing. It is not just academic people that cite sources or other works. That is why I think you cannot ignore it and leave this aspect for third-party softwares.

9

Scrivener is already soooo close to supporting a full Academic workflow natively, if they just swapped the built-in MMD for Pandoc. I strongly contend that the extra ~38.5MB for the installer is totally compensated by gaining so many new features (full templating, indexing, maths support with cross-referencing, citations, multiple new target formats and many many more). Scrivener already supports Pandoc markdown from the editor, the integration would be designing how to control pandoc options from the compiler (a lot less work than integrating a citeproc engine natively). You could use a Binder item (can be symlink to a file elsewhere on the filesystem) for the BIB/JSON file and select it in the compiler like we already do for front/back matter. The binder could also manage Pandoc templates so we could customise how the output looks, and drop-in Lua filters for more control etc. It would utilise the existing mechanisms that Scrivener already provides. For convenience Scrivener could autocomplete citations, but it isn’t strictly necessary as many cool tools already provide great floating citations support.

Personally, when my APA requirement (I’m a Neuroscientist, so Biology+Psychology formatting) was not working the way I wanted it was quick to tweak the CSL, the XML while verbose is easy to understand (I didn’t want disambiguation to use first names in-text, disambiguate-add-givenname="false" is so obvious and easy to understand, also I wanted et al to italicize in the text: <et-al font-style="italic"/>). This is NOT the case for biblatex at all, where the code is like trying to read Klingon:

Each format probably has its advantages and disadvantages for end users, but the reality is that CSL is certainly easier to parse and integrate for implementers.

Anyway, I do support your broader wish and also do hope that Scrivener 4 evaluates how it could broaden its appeal to more academic writers, who may not want to implement the kinds of workflows that users like me have been using day in day out for years. Transitioning to a built-in Pandoc is the most obvious way, followed by citeproc engine integration.

1 Like
10

It’s a critical aspect of certain kinds of writing and arguably not the default use case. I get your perspective and I get that it’s “right” in a sense. I don’t agree, however, that Scrivener/LL should be allocating developer time/resource to facilitating citation. There are external tools for that. Scope creep = bloat from my point of view.

1 Like
11

There are different needs for writers, and Scrivener does aim to appeal to these different needs. A bunch of users love word count targets, that is totally superfluous “bloat” for others. Script writing is arguably also fringe use, is quite complex and certainly fits “bloat” for most of us yet Scrivener caters to it. Writing academic content is a big field, occupying significant portions of knowledge production in our modern world. Legal writing also depends on citation extensively. Many of these users really really could benefit from the totally wonderful Scrivener structured writing workflows, integrating knowledge management etc. arguably as much as or even more than fiction writers.

3 Likes
12

In this sense I strongly agree with you. If they behind the scenes use an intermediate structural file format json or xml (and I believe that they already do so), they simply could put pandoc as a backend with Advanced Options. This would be used just in Advanced Cases. For the trivial use, the mmd is sufficient. I really hope that the reason for this integration is not an extra 40mb of disk space, because this would be unbelievable with the current specs that apple devices are packed today.
Control pandoc output is not that hard.

In this point, I believe that Scrivener can take a rigid approach here. Just allowing to change very specific configurations and leave all the details internally done. If the user really needs all the pandoc things, better to use the program itself. The main thing, again (that pandoc offer) is math support, citation and cross-reference. This combo is enough for 99,9% of the academic writing.

1 Like
13

@nontroppo already properly answered you, but, again, what would we consider the default?

If default is just the most simple possible user case, even Scrivener is not needed. Just open something in TextEdit with .txt extension and write without any format and any other thing. For example: why footnotes? why endnotes? All of them superfluous. Why italics, bold? This is not the default.

As @nontroppo already said, Citation is the default for a lot of writers (inside and outside academy).

14

That is definitely something we should consider in the future. When we first decided to not include it, this was back in the Pandoc 1 days, or maybe early 2, and it was a lot bigger! It has been greatly optimised over time and indeed would provide a lot of internal power for things we can’t currently do, and could likely replace some things that are just sitting there inert for most users like the Java-based Aspose converter for making ODT files.

5 Likes
16

Very good news! This would put Scrivener on another level, years-light above any other writing software. A lot of dedicated word processing still struggle on all these matters! pandoc with the plain text approach makes everything much more sane.

17

Well, please don’t expect soon, for that. For what we currently use Pandoc for, the level of integration we offer, the separate installation route is perfectly adequate for those that want it. If we were to embed it that would imply a need to do so, and really using it, and that takes time. Really we’re talking about Scrivener 4 here, which isn’t even something we’re thinking much about yet.

1 Like
18

You can take it to a silly place if you’d like but the common sense answer is the answer here.

Scrivener is a tool for writing long texts. It’s designed to facilitate the process of writing those texts and managing the complex moving parts of working with the many sub-ideas that make up a larger piece of writing.

Citations may well be a common element in some (even many) forms of writing. They are, however, nothing like italics or bold, which are elements of the expression of the written word itself.

Point me to where I argued for the inclusion of footnotes and endnotes in the Scrivener feature set.

Arguments that go to absurdity don’t tend to have much value.

That said, if the team decide they have the resources to build it and a large enough set of users would benefit from it, more power to them. The fact that the new, simpler app still hasn’t been released indicates, to me, that there are limits on what they can do though and I’d prefer a tighter feature set in Scrivener itself.

2 Likes
19

Then, too, L&L would have to support the new feature. Eek.

20

This is indeed a key consideration, and one advantage of relying on third party citation managers.

2 Likes
21

I agree. It reminds me of how I have vastly extended the capability of Scrivener by working with another third-party app that syncs with Scrivener. That app is no burden whatsoever on L&L, yet increases Scrivener’s value to Scrivener users.