Heading to graph databases

Hello there !
To respect the “be polite” rule, I just want you to know I use Scrivener for at least ten years, and your amazing software has been my daily companion since we first met. I love Scrivener very much, and using it is a great pleasure, every day ! So, a big thank to you sir ! (and to all the L&L team ofc).
The version 3 has been a huge update, and made its use more and more simple AND powerful, which is an amazing thing in the software development.

My wish of the day could be for the v4, if it implies huge core modifications, but I think that would improve Scrivener A LOT : thinking the core of Scrivener, the behind-the-scenes, the documents and the intra-Scriv links, as a graph database. Each document is linked to other documents through objectified relationships.

  • each document seen as an object, adding a new metadata to identify it clearly
  • each document can be linked to any other document thanks to as an objectified relationship, and the relationships can be created in the project settings
  • you can show a “dynamic” view of the whole graph, and as the corkboard, the scrivenings and the outliner, the “graph” view can be shown to selected documents only (may need to choose between “include” or not the documents linked to the selected documents in the graph)
  • templates and objects could be merged (next part of my wish)

And, to avoid making a second post, because I think these two ideas can be related : reforging the templates with two parts :
1st part : a “real” template, with placeholders where to display some variables, mainly some basic tweaks needed because documents are most of the time plain text, sometimes with styles
2nd part : some kind of form, where we note the values of the variables used in the template, so the variables used can be involved in filters and collections

Of course, the template should need a “free text zone” in which the user writes everything (s)he wants, and each instance of the template can also be filled freely, and the placeholders can be modified.
This second whish would allow documents to be objectified, and relationships exposed in the first wish could also be some “special” documents to link two documents.

I’m not sure about the “objectified” word, but in my french brain it’s the best work I found to say “make like an object”, and of course I talk about object in a development semantic field.

Well this 2 part wish could allow even more flexibility in Scrivener, allowing users to really work on some freeform organization. Thanks for reading me, thanks again for Scrivener, and long live this software !

Cheers from France !

Thank you for your suggestion.

I think the application you are describing would no longer be Scrivener, but something else.

Scrivener is intended to help writers complete long works. It is not intended to be a general purpose database.

Katherine

All the things I describe in my post would of course be implemented in the backend, actually there would be quite few changes in the frontend, except an optimized version of the freeform corkboard and better links between documents.
Thanks for you answer anyway !

If the change is mostly invisible, then what benefit to the user justifies essentially rewriting the back end?

Katherine

I’m not arguing to prove my wish of the day is worth the effort, if the dev thinks it’s a good idea maybe he will implement it, if he thinks it’s not he won’t :slight_smile:
I don’t know how Scrivener is coded, maybe the changes I talked about in my wish are not necessary to implement better relationships between documents and better templating.

edit : tu summarize,

  • that would be cool to be able to make different relationships between documents
  • that would be cool to be able to display these relationships in a freeform corkboard-like view
  • the template system is imo underexploited, and it could be interesting to create forms to fill templates, like real templating system

Be aware that up to this point, the person discussing this with you is long-term L&L staff and probably has a much better idea than you and I put together what KB’s thoughts and preferences are when it comes to the future of Scrivener.

I, however, am just a random user! :slight_smile:

One of the things I really like about how Scrivener’s file format and back end are implemented right now is that it relies on commonly available data formats, like RTF, XML, and plain text. Although a Scrivener project is made up of a large number of interrelated files, I have access to free RTF and text editors on both Mac OS and Windows so that if something happened to Scrivener (or God forbid L&L) tomorrow, I can still get my data out of the Scrivener project and transfer it to some other program of choice. Depending on how much I broken down my project into separate documents, it might take some effort – but all of my data is there and still accessible without Scrivener or separate expensive tools. KB (the developer) has stated that this accessibility is one of the design objectives for Scrivener.

If Scrivener moved to a graph database format (or even any database format) for its projects on the backend, what would that recovery scenario then look like? Would this make things more locked in to a proprietary format that required special tools and higher-than-average levels of computer skills, or would the average writing user still be able to recover their data if they had to?

WoW, didn’t say “I’m not arguing WITH YOU”, I just said I won’t enter a debate to prove anything, it’s just a wish on a wish list, and I summarised to get rid of the unnecessary graph and tech thing.
I’m not the dev nor the team, if they don’t want to keep my wish my life will be the same, not a big deal actually oO

Which is on a community forum with an active community. Don’t take it as an attack or an attempt to make you prove something – we discuss each others’ ideas all the time. This isn’t an argument This is a recognition that good ideas can come from anyone and anywhere, and there isn’t a good idea so good that it can’t be improved through feedback and discussion.

@ Arkhaiel,

A)

If I understand correctly you are imagining adding to the free-form corkboard the ability to make connections between the cards. What you are describing is literally what L&L’s Scapple software does! This means the developer also thought this ability would be useful. But they also decided it should be separate from Scrivener. If you search on these Forums you will find others who wished that Scapple could be integrated with Scrivener, and you will also find the developers response and reasons against doing that.

B)
I don’t think I understand clearly what you second wish is. Is the idea something like: project template with placeholder text in various places and then a designated place to go to specify the replacement values for those placeholders for the particular project? (Something like the Replacement area in the third pane of the Compile dialog box, but built in to the project instead.)

Of course, I completely get it.

I understand your point, and I think my wish implies more things than just a visual organization. That could also allow you to make searches, or collections, based on these relationships. But I can’t explain here what does the graph databases allow to do, not in english between two answers. You could see in this video ( https://www.youtube.com/watch?v=bL7wBbk0PIs ) a short presentation. Scrivener actually should work this way imo, since it encourages to split documents, and make connections between them. What’s not in the video is the fact that different types of relationships can be used, so you can easily make very powerful queries.

Damn, the language barrier is sometimes hard to handle ^^
I think the templating feature can be way better. In web dev, for example, the templates systems separate the content from the form. This way, you can modify THE template and it will automatically modify the way ALL your pages look. Considering that, if you could make a solid templating system where some key informations (name, physical description, introduction scene…) are stored in metadata (or in another place), and displayed in a document, you could then be able to modify the “template” to update the way the document is displayed.
No template ? All the “template metadata” are displayed basically in a blank document, one per line.
Switch to template A ? If you defined “$NAME”, “$DESCRIPTION”, “$INTRO SCENE”, into a well designed template A, then the document now is “template A + template metadata”.
Switch to template B ? Juste change the “template” source and the whole document is now changed, keeping the templates metadata into the well designed template B.
Back to template A ? Just change the “template” source.
Want to modify template A ? Juste modify the “Template A” file, and all the document based on this template are now changed.

You can see that as a dynamic templating system, which would refresh each time is needed, and display in realtime the datas “template model + template metadata”. You can also modify the template metadata whenever you want, of course.
And, in the template, you can define “free zones” where you can write whatever you want. Free zones are not related to “template metadata”, it’s just zones where you write some longer text, like a short bio, to keep the characters template example.

I’ve used Scrivener since its first release. I use the Workbook template for almost all my projects because I tend to be research heavy. While a visualized graph is useful, especially for complex projects, I think Scrivener is a top notch platform because they are micro-focused on the needs of writers.

If you are interested in a graph-like view you can accomplish this without a graph database. Someone else mentioned the companion platform Scapple. I’m not sure that’ll work for what you want to accomplish. Using markdown #tags or Scrivener tags, you can link files together. A potential method to get you part way is to add a tag or tags to snips of content you append to new pages. [highlight>Append to… I think].

From there you could export from Scrivener into something like Obsidian, Zen Kit’s Hypernotes or Ingo Straub’s Knowledge Base Builder with the latter having more functionality. Obsidian and Hypernotes just visualize notes you link more like a mind map. You designate the elements to reference [tag] in each document. From this, you can visualize relationships between elements and pages. You add new relationships using tags. These are visualized in a dynamic mind map that is presented in a view that looks like a graph database. The items you tag become your nodes [entities]. KBB is a user friendly graph database and you can add or delete relationships or nodes and add supplementary depth from the stable of Wikipedia projects.

Putting your manuscript or project into a full blown graph database is essentially creating a corpus of your work. That is not a simple thing. It requires some machine learning work for text analysis and a means of identifying what’s important and relationships between entities. Including that level of depth into Scrivener would require two databases when the markdown tags would give you the same visualization capability without the bloat.

Another feature of KBB that I like is you can import a file, or cut and paste text, and use the text analyzer. It identifies entities and concepts and creates a graph. Using expanded view, nodes become cards with your text chunks. I sometimes do this, export my graph to csv and import to Kase or a mind map program depending on what I want to accomplish. This does not require markdown.

Obsidian is open source, KBB is free & Hypernotes is free for 3 projects but paid after that if you want more.

Let me know how it goes if you play with this.