A scrivener/scapple-based PKM?

I personally LOVE the way that Scrivener and Scapple work… there’s just something about the way my mind works that is captured by these pieces of software.

I’m wondering whether Keith might consider building Scrivener, integrated with Scapple – into a full PKM? Scapple could provide a beautiful visual aspect to Scrivenings, so that each note could have a place in the binder AND also a place in visual space (or visual spaceS).

I mean it’s almost there already; it CAN be used as a PKM as it is. But I’m sure there are certain adjustments that would make it even better if it were deliberately altered for that purpose.

And L&L’s sense of aesthetics and what is actually useful, not to mention its large user base and the obvious and easy movement of data into writing projects, would instantly propel it into the top tier of PKMs.

Just wondering: what is a PKM? :grin:

This was brought up years ago when Scapple was first launched. At the time, KB ruled it out for technical reasons.

:slight_smile:
Mark

Personal Knowledge Management.

Isn’t that already the case? If you have your Scapple files in Scrivener you can see them there. And I don’t even notice that Scapple is opened when editing.

You can import a Scapple board into Research, and you can click to open it in Scapple to edit it, but on my reading that is not the level of integration that @shivohum is wishing.

Mark

We even have a knowledge base article on this request.

Yeah, good points.

Perhaps the point is not so much integration with Scapple per se but rather to suggest that L&L has a unique design philosophy that would make it particularly expert at creating a PKM, and one that would link right into Scrivener.

But what does L&L gain by entering a space that is already extremely crowded, that is only peripherally related to our core competencies, and in which the biggest players are corporate behemoths with locked-in users?

4 Likes

I’d be curious to know what sort of values you assign to something operating as a PKM? It seems to have a visual component to it that isn’t something I would at first associate with the genre as necessary for something to be included within it. To put where I’m coming from in perspective, my “PKM” (if one would be so inclined to jargonise it) has no software requirements at all, and consists entirely of a formalised network of metadata found in plain-text files, essentially all thrown into the same folder together by the thousands. It’s simple enough that it could operate on a box of index cards, but just tuned enough to keep millions of words from getting lost. There is nothing at all visual about it, but that’s how I organise almost all of my information.

When I think of ideal software for this, Scrivener does in fact comes to mind, and as it stands. Here is a recent discussion, which has some links that branch off into other discussions—one of which in fact goes over Scapple vs Scrivener and how well the former is suited toward a role in this area.

So with that all in mind as background, what do you find missing in what it provides?

2 Likes

Good question. Scrivener is a piece of artisanal software that has a clearly defined target market – writers – and a clear design philosophy.

Knowledge management for this particular target market is in fact not such an extremely crowded space, and is in fact very close to L&L’s core competencies.

I’ll tell you what got me started thinking about this. It’s the way that Scrivener deals with notes. The frictionlessness with which I can create a bunch of notes, one per thought, and then group and reorganize them in various hierarchies instantaneously… is unparalleled in any other software I can think of.

Scapple has the same frictionlessness with visual play of information.

What would personal knowledge management mean for a writer? It would mean keeping, organizing, and searching information that is now or may in the future be relevant to one’s writing. Often the relevance of information may not be clear until far in the future. So you want something that will allow you to group and regroup information and to bring up playful new possibilities.

But Scrivener does a lot of this already. It keeps and organizes information. It can handle all kinds of files. It stores metadata on that information. It allows for intricate search. And Scapple allows for intriguing visual manipulation of information.

It seems to me a natural step to expand this from being specific to a particular project to being for all possible projects.

Something that would allow me to keep all my information in a single place, manipulate it, play with it, and then funnel it seamlessly into writing projects, would be valuable. And it seems like it takes advantage of many capabilities that are already well within L&L’s purview…

Scrivener and Scapple already are knowledge management software; they’re just slightly restricted in focus. If that focus could broaden itself a bit, it could manage the entire world of information for its artisanal audience

Great links, thanks. I suspect much of what is required is already in place. Scrivener has so much that’s already there :slight_smile: .

A few of the things that I’d want to see:

  • some kind of “meta-interface” that goes beyond the project level and allows a view into all projects
  • an amazing web clipper
  • in-line handling of video clips and the like, so they could be thumbnailed and played in-app
  • the ability to have the same Scrivener document placed in multiple places in the binder, rather than just copies of the document, so that editing that one doc edited them all
  • maybe some kind of simple Apple Notes-like interface on iOS so that a note could be taken in seconds and just blindly dumped into a kind of inbox

As far as the visual, I’m thinking of something like Obsidian’s canvas. It would be nice to be able to take the same document and see it hierarchically, but also be able to see it in a visual space where its connections to other documents could be played with…

That is a lot easier to remember than the word banded about on the Obsidian forums and elsewhere of Zettelkasten.

1 Like

I’ll tell you what got me started thinking about this. It’s the way that Scrivener deals with notes. The frictionlessness with which I can create a bunch of notes, one per thought, and then group and reorganize them in various hierarchies instantaneously… is unparalleled in any other software I can think of.

This is an astute observation. A big part of what you are seeing comes from it being an outliner, rather than a document bucket, the latter of which a lot of these kinds of tools orient their metaphors more toward (some to the point of it being literal). If poke around in the heavy-duty outliner space, where we would find tools such as Tinderbox, LEO Editor and Logseq/Roam, you will find a similar lack of nonsense where it comes to the creation of new data containers—to strip back the metaphors—in the system.

That said, Scrivener adheres to the headline+content outlining model[1], and that model can be more mechanically cumbersome than the inline outlining model (Roam, Logseq and even Word’s outline is an example of inline outlining, though outside of the note-taking space). But we have put a lot of effort into it being less mechanically cumbersome. For example you can hit Ctrl+N / ⌘N from the content field (text editor) to seamlessly start writing in a sibling node created after what you were just typing in before[2]. This can also be done from its long-description field (synopsis), and just about any other context in the inspector or main editor. And in the outliner it does follow many of the normal outlining principles: Enter key to make new headlines, shortcuts to indent and shift siblings around, etc.

All of that is going to give you a big leg up over anything that uses a more strict document metaphor in my opinion, and especially if the metaphor isn’t so much of one, and is in fact more literal, a skin over the file system itself—because then you need file names, and file names need to be valid and not duplicated, and so on. Overhead creeps into the “I have a new thought!” moment. Scrivener doesn’t demand a name, and works just as well without one.

One ring to rule them all

Something that would allow me to keep all my information in a single place, manipulate it, play with it, and then funnel it seamlessly into writing projects, would be valuable. And it seems like it takes advantage of many capabilities that are already well within L&L’s purview…

I think the main problem with this notion is a lack of direct plumbing between projects, which could be framed as a downside, but there are reasons for it being that way—and meanwhile projects are not entirely discrete either. External links are fluidly created and used, and the Documents ▸ Copy to Project system can make cloning data between open projects a bit easier in some circumstances.

The are good reasons for segregating data, though. Ulysses used to use this model as well, back in its v1 and v2 days, and when it switched to a universal model more like what you describe, that was for me a pretty big downside—enough so that I stopped using it. Perhaps its a matter of personal scope, but personally I wouldn’t want my Scrivener bug reports mixed up with my personal journal, nor bad novels I wrote in my twenties, blog articles, chat logs from over 25 years, email and never mind everything in between. The ability to create larger and harder walls between exclusive data sets is valuable.

Put simply to an example: being able to link to a bad novel from a bug report is valuable, and I can do that with Scrivener, while at the same time have diverse search scopes hard-limited to bug reports without having to adjust a bunch of settings.

There are ways of doing that in a program designed to intake everything, but those models are going to start to resemble something more like “projects”—to the point that you have to wonder if it’s worth redesigning the concept of projects. Perhaps the most effective model I’ve seen for that is Devonthink, which still uses the file system to create hard walls, but presents them in a UI that makes working with multiple open super-containers more seamless. It does so at the cost of complexity.

That segues into some other issues as well, such as how all of those discrete super-containers are managed. I have hundreds of Scrivener projects, 90% of which are .zip files that rarely get opened any more. They have served their purpose. Would a universal model like you describe allow for that? That’s more complexity. If it still used the file system to store these super-containers, what happens when in ten years I change my archival directory structure and move these hundreds of projects around. Does the “manager” understand where they went? If the file system isn’t used any more and all of this is 100% managed, then that’s just more feature set and more complexity to handle that particular problem—one that already has a solution called the file system.

The other end of that spectrum is something like Ulysses or OneNote that doesn’t allow you to manage your super-containers at all. No selected deletions using file system tools, no easy discrete backups, no easily sharing just this bundle of data in its native format with a colleague, no sending in just this one book to tech support. Complexity is reduced, but at a very high cost (as is often the case) in flexibility.

Obsidian is another system that allows for hard super-containers, with its Vault system, where I would say it’s about as segregated as Scrivener’s project system—perhaps even more so as each vault is in its own sandbox, to the degree that the entirety of its preferences system is individual to each vault (without a bunch of advanced symlinking at the file system level against hidden directories and files).

I don’t think there is one right answer to this problem, and whether one wants a spider web that encompasses everything or lots of little containers that can be interlinked is probably a major factor in which tool they would choose.

Specific features

an amazing web clipper

Yeah, I guess I’m more of the UNIX mindset on this kind of stuff. Why should that be something reinvented dozens of times over, at great effort mind (it’s not just the intake but developing and maintaining multiple browser extensions in systems you aren’t familiar with)? Doesn’t it make more sense to have tools that just do that, that can excel in that purpose, and produce a results that can be easily taken into other programs that dedicate their code base to other tasks?

If you take that concept and allow development to flourish within it as a sub-genre of software, it also allows for variety that no single developer could probably ever reinvent into their software. For example, what you want to web clip may look radically different than what I want. I almost always want a clean Markdown translation of the original HTML. No colours, no fonts, no ads, no navigation sidebars: I just want the data in Markdown format. I have a browser extension that does just that, and since I use Scrivener exclusively to host Markdown content, that data is exactly what I want in the text it takes in, too.

in-line handling of video clips and the like, so they could be thumbnailed and played in-app

That’s already in the software. Not only that you can open a video in one split and pause-resume remotely while typing in another split. This can be set to auto-rewind by a defined number of seconds on each pause-resume. Not only that, but you can easily insert media timestamps as you type, or even set up a transcript workflow (we provide an example in fact in the scriptwriting section) that auto-timestamps as you type.

I think we’ve got the basics covered, for what writers need of video in their writing environment. It’s suitable for subtitling work (to a degree I’d say it competes with expensive special-purpose tools for that).

the ability to have the same Scrivener document placed in multiple places in the binder, rather than just copies of the document, so that editing that one doc edited them all

Scrivener’s Collection feature taps into a lot of what that capability provides in terms of user interface. If one doesn’t get hung up on the binder as being the only tool they use, that is, the notion of expanding your listings to many overlapping reuses of that content is quite powerful. Heavy-use projects of mine often have dozens of collections.

It is worth bearing in mind that a Collection in Scrivener is as a concept very similar to an outline item—it’s just not listed in the outline tree. By that I mean you can navigate to it and load it into the editor as a container, viewing it using any of the editor’s view modes. Freeform corkboard + an inbox collection that is in part fed off of a watched sync folder that intakes .txt files you dump into it from other programs, for example!

When it comes to export on the other hand, the binder is necessary as is a full outline model. That is where the <$include> placeholder comes into play. With that you you can replicate text content into multiple areas of the outline—even more so I would say, since the content can be injected into different contexts. What can be a discrete glossary entry at the end of the book can become a footnote elsewhere. I.e. the outline model can be injected into the text content model, which is not something you’ll find in any of the traditional replicant/clone models used by most outliners.

maybe some kind of simple Apple Notes-like interface on iOS so that a note could be taken in seconds and just blindly dumped into a kind of inbox

That is one aspect that Devonthink does do, as one of its “super-containers” is an embedded inbox that is entirely managed, rather than being a database you create and save somewhere in the file system. You always have an inbox. Scrivener’s closest analogue to that is the Scratch Pad, but it’s hardly even worth comparing the two. It makes more sense in my opinion to have an “inbox project” that is pretty much always open. Less integrated and managed, but not too far away from final effect.

There are other ways of addressing that concept though. Project Bookmarks have a special interface that operates out of a Quick Reference window with a sidebar. Using that you can establish a inbox folder where new notes you create within that window go into it. And I already mentioned the external folder sync method, which not only has an interface for note gathering in the software, but integrates with any external software that create notes as files.

From notes to works

To expand on this notion though, and go beyond the pure mechanism you’ve described: I’ve often said that what makes for a good writing tool makes for a good note-taking and thought gathering tool, or “PKM” to put it to jargon. If we brush aside Scrivener’s primary purpose for a moment, and look at its technology purely for what it is capable of, in pieces, it already has much of what you will find in tools that only target that goal. This much is fairly obvious, and we can argue it isn’t quite as effective here and there in the details, but what I’d like to look at is where it goes beyond that, with its natural tendency toward writing, and how that can benefit this alternative goal. I think it goes back to what you said, and how I started this response.

Something I always found very liberating about using Scrivener in this role is how effortlessly I could transition from “I need to get this thought down before I forget it, dump it in the inbox”, to “okay now I have lots of ideas for this”.

That transitional space is something I’ve often found to be very awkward in traditional note-taking tools, precisely because of their bias toward the document container model over the outliner node container model. And to address the inline outlining model, while one can of course effortlessly go from one heading to 100 child headings—it’s a bit awkward once you start writing in bulk. I wouldn’t want to write a 50k word document in Logseq!

Scrivener strikes a balance between these models. My dashed down note that I threw into an “inbox” collection or folder can effortlessly become the master parent node of a branching 100+ node outline tree, that I can start writing directly into, and have it eventually become a 50k word document. There was no point there where I had to make a decision to stop using one tool and move to another (even within the same program to be generous as most dedicated PKM tools wouldn’t handily address any of what I just described with ease). I never had to set anything up (like a transclusion map3). I didn’t have to switch to a different kind of “document”. I just went straight from a one-liner in a list to 50k, gradually, bit by bit.

My example there was deliberately extreme because few tools can handle all of those extremes at once, but perhaps at detriment to the full gradient of the argument: it’s not just about the one-liners that turn into 50k documents. If the tool you are using can do that without blinking, then it can also do everything in between as well. The one-liner in the inbox can become small outline, let’s say six child items, representing a checklist of steps, with a modest total of 200 words of documentation of a task that required learning a command-line tool to carry out. That can then be transferred somewhere more appropriate (into a super-container/project or another folder in the binder of that one project), tagged up, however one does such things, and archived.

This very post that I am writing could have been developed in outline chunks, bits moved around or discarded, and when done, copied and pasted from Scrivenings into the forum. When I’m done I could archive only the flattened version, but why? My PKM handles the original thought model and can flatten it with ease if I ever need it again.

There is something in the twilight space between what Scrivener can do at extremes (a series of ten fat fantasy novels in one project) and what it can do with a sentence as a heading in an outliner view, that encompasses a lot of what many PKM tools try to do. Scrivener does a lot of it differently, but I don’t think it’s fair to say it doesn’t do them.

I’ll drop a link to this post on using Scrivener for notes as well. I think it may have been linked to indirectly above, but it’s a pretty good rundown on some of the particulars of what Scrivener provides as a note taking and information gathering tool. Stuff that might not be obvious just by looking at it from the perspective of it being a long-form writing tool (and certainly from the poor confused perspective of it being a “novel writing program”!).

Notes

  1. At first blush, Scrivener can easily be confused as falling under the notes-as-documents model, especially with its jargon and pretence at having ‘folders’ etc. This should be quickly dispelled the moment you nest a PDF file beneath another PDF file, or type text into your “folder’s” content node.
  2. Well… save for bugs in the Windows version with keyboard focus getting lost whenever the text editor changes its content.
  3. In case the jargon is thick here, this refers to creating a document in a document-based system that manually refers to other documents in the order they should appear, with the purpose of creating a master document that can be exported containing all of the content. I.e. the outline is completely separate from the content and must be maintained as such. You could do this in Scrivener as well with its <$include> tag, but you’d be insane to.
6 Likes

@reepicheep : That is a lot easier to remember than the word banded about on the Obsidian forums and elsewhere of Zettelkasten.

Yeah, but loosely translated, the concept is quite close to the phrase I used above, “box of index cards” (well, slips of paper, but essentially that’s a good comparison). That doesn’t say everything, but it says a lot about the method’s philosophy, and is easier to remember.

To compare it directly to the phrase “personal knowledge management” would be a bit different. A bit like saying Scrivener is harder to remember than Software. Maybe so, but one is an umbrella concept with an incredibly diverse field of ideas within it, while the other is a fairly specific form of that field. A formal system using a box of cards (digital or physical) is one way of managing one’s personal knowledge. That system itself can be embedded in almost any software, or even none at all.

1 Like

Thank you for the very thoughtful and comprehensive response. You make some really good points, and I’m going to have to give them more thought. I’m certainly going to have to play with Collections a lot more, that seems clear. I totally agree with your beautiful point about the way Scrivener makes it easy to transition from a bunch of notes, step by step, to a large and serious formal text – while keeping the transitional structures at the same time.

One thing that strikes me off the bat, however, is despite all the advantages of projects and their scope-limited search, why not also have some kind of dashboard that allowed search across all projects? I wonder in fact how much of my jonesing for Scrivener-as-PKM is really about that.

Right now my hack is to use DocFetcher and/or Easyfind to search my writing folders… but when Scrivener projects come up, it always feels like it’s a bit of a mess, since you can’t see the search term in its natural note form.

So: searching all projects.

But then – a step further: in a world where we have information scattered across the entire file system, email, perhaps – like you have – hundreds of Scrivener projects, thousands of text messages, maybe some other PKM, and on and on… it feels like there ought to be indeed one ring, if not to rule them all, then at least one to search them all. It shouldn’t take trying, one by one, all the various places you may have put the information, with various searching standards and options, presenting information in messy, hard to comprehend ways. No?

1 Like

@shivohum I don’t know how many projects you have and how big they are. But it seems to me that Scrivener’s ability to handle (very) large projects is rather underestimated. Have you ever considered merging your projects? I’ve been working mainly with one large project (PKM) that keeps growing for years and never had any problems.

Being able to search and find everything in one project makes my life much easier. I have found for me a search method that allows me to search for everything (words, metadata) at once. You don’t have to use this menu where you can limit the search to certain criteria.

With Scrivener, almost anything is possible. It’s just not all in the manual. :slightly_smiling_face:

…why not also have some kind of dashboard that allowed search across all projects? I wonder in fact how much of my jonesing for Scrivener-as-PKM is really about that.

That really goes right back to what I was talking about though, about how Scrivener isn’t aware of any projects on your disk until you open them, and even if it somehow were (that is a complicated problem when you have full file system control over them!) it has no way of searching a closed project, and the only way it could conceivably do so would be quite a bit slower than I think you’d find useful.

For it to become the kind of program that can do that effectively would require radical changes—ones that probably would merit a more unified interface like Devonthink has (which adds operational complexity in my opinion, rather than reducing it). To accomplish that and reduce complexity would require an even more radical change to a singular database format. We just don’t know if we want to go down that path though, because there are a lot of cons to that.

It’s something to think about, for sure, and I have quite a bit written about how we could maybe strike a middle ground between these various options, but it is something we would want to do very carefully, and with as much respect as possible for how it can currently be used. Keep in mind most people use this program to write one book, then stop using that project “forever” and start on the next book, one after the other. In most cases you wouldn’t want a search to go sprawling across everything you’ve ever used the software for. This is what I mean about preserving how it currently works, while maybe trying to dip into different and broader uses of the software where such a sprawl might be useful.

But then – a step further: in a world where we have information scattered across the entire file system, email, perhaps – like you have – hundreds of Scrivener projects, thousands of text messages, maybe some other PKM, and on and on… it feels like there ought to be indeed one ring, if not to rule them all, then at least one to search them all. It shouldn’t take trying, one by one, all the various places you may have put the information, with various searching standards and options, presenting information in messy, hard to comprehend ways. No?

Well that is an interesting problem for sure, and one that has occupied massive resources to try and solve it. Tools like Spotlight, Cortana and Akonadi have been created to try and solve it, with varying success.

My own approach has been to embed this kind of information into the PKM structure itself. I use a “library card” approach for all things, rather than embedding the actual thing into it, and have an infrastructure built for doing so efficiently. For example my Scrivener project itself is not in the system, but there is a “card” describing it, and often containing a final archival quality compile of it (I use the compile option to embed external links so if I need to get to a specific thing within it I can do so, after extracting the project to its original location). I use this approach for everything, so I do have one single place that I can go to search and find things. It doesn’t always mean I can get to a singular sentence in a resource, but I’ve never found that to be super necessary. These “cards” are all mixed in with original material as well.

My approach certainly does have more overhead than a fully automated system like Spotlight, but because I have curated the data that goes into the “index” I find it worth the results. I’ve never found automated tools to be terribly useful to me, particularly as they tend to have simplified front-ends that don’t allow me to, for example, use regular expressions or other combinations of command-line tools to query them.

To couple that notion with what @fto just posted above, there could be a way of doing something like that in a “master” Scrivener project that is intended purely as a global search tool. The kind of project you always have open and can drag and drop important things into as external links. Bearing in mind that a card in Scrivener can have a list of Document Bookmarks, and that those can point to individual binder items in other projects when you drop binder items from other projects into that list, there is a lot of latitude and power there. Myself I use text files as “cards”, but again that’s mainly because I really strongly prefer the power of the command-line interface and the ability to write scripts that speed up the creation of them, and manage them otherwise.

4 Likes

I just wanted to briefly point out, including external documents in Scrivener certainly has advantages. Especially if you have such a smart system as AmberV seems to have. But they are still external, with all the disadvantages that entails. I prefer to have everything in one Scrivener project. One backup for everything.

1 Like

You can have your cake and eat it too, by importing such files as aliases or shortcuts into the binder. The result in the project interface itself is 100% identical to fully copying the resource into the project.

While there is something to be said for what you are describing, I would rather my backups of all this only back up what has changed. I don’t need the same PDF from 1999 backed up every night. My nightly backups to external disk only write what has been modified or added.

I’m not necessarily disagreeing with you, and you are right that a lot of people do underestimate how much you can put in a single project; just offering a different point of view, and also speaking more to the concept of having a master project that helps you find stuff in other projects (as opposed to that ancient PDF), with those external links. It’s not a huge burden to double-click on such a link and have the original project load in its own window, with its own search scope. There can be significant advantages to that.

1 Like