Dear Scriveners (assuming that’s what citizens here call themselves),
I’m a physicist and have been searching for years for a decent lab notebook. I used to use CircusPonies, but when it vanished, I switched to a mix of Omnioutliner, Devonthink, and just Finder. This combination sort of works, but not very well. I just downloaded Scrivener to see if it might be a good research notebook. It’s clearly a tool that its users love, but I don’t see many (any?) people using for my intended purpose… which of course makes me nervous. I don’t want to waste time if it’s the wrong tool.
So, for me, a scientific notebook needs:
Something that can behave as a two-pane outliner. Titles of notes on left, and full entry in the main view. This way, I can quickly scan through notes.
Notes can be arranged in a hierarchy.
Each entry can have lots of plots, tables, etc.
Links pointing between entires are easy to make.
Replicas of entries are possible (this isn’t essential, but would be helpful).
I’ve tried tinderbox, but it is not well suited to working with lots of images.
Any thoughts from experts (which is everyone compared to me) would be greatly appreciated.
Thank you!!
All except #3 are trivially easy in Scrivener. #3 in part depends on integration with your preferred tool for creating these elements: Scrivener has no built-in plotting capabilities, and large data tables are going to be more easily handled by a dedicated spreadsheet.
If you haven’t already, I’d recommend downloading the trial version and taking a look at our Interactive Tutorial, available from the Help menu. That’s the best way to decide if Scrivener’s for you.
Thank you for the quick reply.
From playing around with the demo, it looks like a possibility. And it’s a beautiful enough piece of software, that it makes me wish that I were a novelist.
A couple of questions though.
Replicas: It looks like on can duplicate notes, but not replicate them (i.e. edits appear in both). I see that there are collections, which hold replica-like objects, but the list of collections is just a big flat list. A replica doesn’t look like it can be in the outline structure. Is that correct?
Plots: I can add them to an entry very easily, but unlike some other types of editors, “double clicking” doesn’t open a full size image. Is there a way to do that? Images can’t be gigantic on a page, but one often has to look at the images full-size when reviewing entries.
Correct, duplicates are independent documents: changing one copy does not change the others.
A Collection is just another way of viewing the same documents. It doesn’t create new documents.
For your application, I would recommend using linked images. That would both keep the overall project size down and facilitate editing images in the source application. See Section 15.7.4 in the manual – available from the Help menu – for more information.
Thank you for the reply Katherine. As for including images vs linking them, including is much better than linking. There are two reasons for this:
Scientific notebooks need to be very stable against accidental changes (original image file moves or gets renamed for example)
There is no need to edit the images externally, or at all really. Generally, one is putting in plots/figures that represent results. So once an image gets added to an entry, it doesn’t ever get changed. If there’s a mistake, corrected images go into new log entries.
So, this brings up a question: How many images does it take for Scrivener to get bogged down? I’ve begun using Scrivener as my logbook, and it’s starting to look like it could be the scientific notebook tool I’ve been looking for!! But, I don’t want to get too far into using it, if storing a lot of images (pdf, png, jpg) is going to become a problem. So, very roughly, how big is too big?
Thank you!
I am also a physicist and while Scrivener is fantastic for organization, outliner etc., I think Markdown in Visual Studio Code is the best way to write technical notes with math. Notes taken in MarkDown can be imported into Scrivener. Visual Studio Code displays MarkDown in real time, which is important when writing documents with lots of math. Of course, Visual Studio Code also supports Python, Jupyter notebooks and other programming languages.
If you are doing experimental stuff with data, graphs and diagrams then Scrivener might be a better choice. Markdown can display graphics and tables but doesn’t really cut-and-paste with those objects. Scrivener, of course, has much better organization, which is certainly required for large documents.
I really like Scrivener, but the fact is Markdown is becoming the default method of writing technical documents especially in programming, statistics, data science and computer science. The perfect solution for me would be if Scrivener supported Markdown Math internally so I could see the equations as I type them.
Thanks for idea of VSCode @logaandm. I think where Scrivener can be useful, is not so much in writing every single log entry about every run/test, but in writing summaries. My PhD advisor used to say that the trick to making progress is to “summarize, summarize, summarize.” I have found that that is emphatically true. A bucket of detailed notes is useless without summaries and links into them. Those summaries need to have real organizational structure, and are most effective when they are written like a book. I currently use Omnioutliner for that part, but I think scrivener will be better… as long as it can hold a meaningful number of plots.
For the individual entries, I currently just use textedit and save in rtfd. It’s not bad, but following your suggestion, I’ll also try md using VSCode. The ability to include latex would be a nice perk.
What do you mean by “a problem?” Scrivener will work with truly enormous projects, well into the million word, multi-gigabyte range. But depending on your hardware and how the project is structured, such large projects might have performance delays that are unacceptable to you. (But that might not bother other users.)
Scrivener is designed so that it only loads the files that you are actively editing at any particular moment. So for best performance you would want to keep your individual log entries in separate documents, and possibly keep the associated images in sub-documents, not embedded in the main entry text. If performance starts to be a problem, you might also consider using multiple projects, broken down by month or by year depending on your volume of material.
From a documentation/archiving perspective, Scrivener is not ideal because it is possible to edit “old” entries. There is no way to write-protect a portion of a project. So if you need to be able to prove that a particular entry was made on a particular date – for instance to support a patent claim – you should make regular backups to more secure media. (If this applies to you, your institution probably has guidelines explaining exactly what this entails, which you should follow.)
Yes, slow performance or crashes. I don’t have such a large volume of stuff, I just don’t want to use Scrivener for a purpose outside its intended use. Using software for something it wasn’t designed for often causes all kinds of grief. I know that I can drag images into files, but if it’s intended really as text only, with maybe a couple images here and there, then I don’t want to use it in a very non-standard way
OK, that’s good. It’s easy to make a larger number of small files within the project. And yes, multiple projects is also pretty easy.
Right. In my field of physics, that’s not an issue. I’m not worried about regulatory requirements, I’m worried about my own foolishness. If I accidentally rename a folder, then all the image links will break and I wouldn’t know about it until I happened to look again at that file. And by then, I might have no idea what folders got renamed/reorganized. Similarly, if an image name ever gets trampled by another file, then the image in the note would change, and I might not ever know that the image has been changed. Finally, if images are stored external to the notes they correspond to, then I need to make a system for storing/organizing those images, It’s best to just have Scrivener hold them… if it can.
At this point, I’ll just push ahead and see how it goes. I really appreciate the quick tech-support help.
I’ve got a project with over 3000 small to medium resolution images in it, and have had zero trouble with performance. I don’t include most of those images in RTF files, instead they’re just sitting in folders on their own, so you mileage may vary if you insert images directly into a text document, instead of importing them into the binder.
As for renaming and moving them around within a Scrivener project; no worries there. When you import a file into the binder, Scrivener assigns that file an internal identifier, a kind of serial number. When you link to a file within that project, it’s that identifier that Scrivener is tracking. Binder titles, and where in the folder structure it’s located, are just other bits of data associated with that identifier. In other words, you can modify the file and every bit of Scrivener metadata that you can see in the program, all without other parts of the project losing track of it.
The only exception to the above is if you make use of the <$include:binder title> tag. I believe that only works for text documents anyway, but the “binder title” portion is used during compile to locate the first instance of that title in the binder, and copies that document’s text into the place where the $include tag is located. If you rename the source document, then you have to update all matching $include tags as well. But that doesn’t really apply to images anyway.
Thanks @RDale. That’s a great suggestion. Keeping the images in a separate folder and linking them from within Scrivener would be a very good middle option between linking to external files and embedding them. If Scrivener is linking them, then it’s handling the overhead of cataloging.
It can read Scrivener’s project format and can do live markdown preview as you write.I love VSCode and the markdown enhanced extension too, but Scrivener+Marked2 is a very sweet setup. It supports Pandoc so you can see your citations rendered out and utilise many filters etc.
I use Scrivener as my lab notebook. It organizes my material very well, and that is all that I use it for. I do not store data or analysis files within Scrivener, but I do link to them.
An issue that was brought up is that links can break. That is an issue, but I have knocked it down to something trivial by using a technique from the olden days of traditional lab notebooks: organization by date.
Within MacOS, I organize all of my data and analysis files using Finder, nothing fancy or more complex than that. I have a folder called Experiments, and within that folder are subfolders with sequential date names, like 2021-01-01. All of my experimental data and analysis files, either as actual files or as aliases, are stored in those date folders. Within Scrivener, in a page that discusses the experiment or activity of the day, I create a bookmark to the corresponding date folder. I also enter the date in a date metadata field. If I decide to reorganize my files in Finder, it will break the links, but it’s super easy to find the files again because within Scrivener the date metadata is recorded and it matches my date folder names in Finder.
It would be nice if the bookmark links within Scrivener were more like aliases that can follow the path of linked files or folders, but I’m not technical enough to know if that is a reasonable feature request. If the bookmarks in Scrivener could include followable path data, please consider this a feature request.