Stable way to reference external files

Scrivener Scrivener for macOS
1

Hi,

I’m looking for a good approach to handle a larger number of external files relevant to a writing project in a stable fashion, since I repeatedly managed to break aliases:

I have different files, which are relevant to a writing projects — both rich and plain text files as well as other file formats, such as mind maps, PDFs, various graphic files, video and audio files, etc. While I can import text files as binder files and have them stored inside the Scrivener projects, for non-text files, I imported them using Import > Research Files as Aliases, since the files are too numerous and too large and they would slow down Scrivener for file saving and backup operations, while significantly increasing files sizes of those projects unnecessarily.

I recently started bought the iOS version on my iPad so I could do some small writing on the go, but after I had set up Dropbox synchronization I noticed that all files imported as aliases were not found by Scrivener anymore, although I keep the relative folder structure intact, before moving files to Dropbox. Since I had imported most of these only recently and haven’t made many annotations, I could just delete and re-import all those files, but I’m looking for a more stable way.

I have been contemplating different ways of managing these files:

I was first thinking of an online repository. For example, https://osf.io/ produces stable URLs and even DOIs for published files. The advantage would be that such resources are available across devices, with the disadvantage being the need to be online to access a web resource. It’s free and the source files are even versioned.

Alternatively, I was considering a local document management system, such as DevonThink. I can copy links to files in a local database, which use a custom protocol handler x-devonthink-item:// followed by some internal identifier. This would probably not work on an iPad or different machine, but I would not need to be online to access any resources. Also, I would need to buy Devon Think (which is acceptable if it works reliably for my scenarios) and I would be limited to the platforms which run Devon Think (Mac and to some level iOS it seems).

(P.S. I also came over a proprietary solution for links called Hook https://hookproductivity.com/, which seems to gain popularity, but I fail to see the advantage or regular OS level links, to be honest).

I’m already quite sure that a writing tool such as Scrivener should not be abused as a document repository, so files which live elsewhere and are linked back to Scrivener (wether linked as aliases or similar links offered by the OS or as URL/URI/DOI) seems the right approach. Theoretically, and online repository seems to outweigh the advantages potentially offered by solutions on a local machine such as DevonThink, but I’m new to either solution myself.

Unfortunately, it seems that Scrivener considers either as URLs, which can only be imported as external (document or project) bookmarks, not files inside the binder, which are handled quite differently. I believe that there is a gap here in the feature sets of Scrivener, so this may be a case for a feature request, but I’m not sure if I’m overlooking some functionality of Scrivener, or if I am not getting the use cases and capacities of Scrivener’s bookmarks right.

Of course, I would be interested, how other people have handled such issues.

2

I have used DEVONthink fairly consistently for over a decade (though I stepped away from it between version 3 and version 3.5 – the last updates have done much to make it more workable for me). It is a solid solution to storing (and finding) large amounts of research material in many different formats. Search is very fast. I had databases containing text files amounting to several million words and could find things faster than I could type the whole search phrase. It has recently been stated by their support staff that the comfortable limit for a database is 250,000 to 300,000 items (https://discourse.devontechnologies.com/t/importing-30gb-300-000-item-mail-archive-possible-tips/57906/2). I have found item links very useful in my work. I sometimes use them in iThoughts mind maps, and they work even in iOS (I can go from a mind map in iOS straight to the file in DEVONthink To Go). In Scrivener, it is possible to add an item link in the text, and clicking on that will take you to the file in DT, both on MacOs and iOS. I use that method in preference to bookmarks in Scrivener.

DEVONthink is complex and can take a while to get used to. It would be worth downloading a trial and testing it with a small database and a test Scrivener project to see if it will do what you want.

1 Like
3

Thanks for the detailed reponse.

I have no doubt about the general use, capability and stability - I have heard many good things. But my need is not about moving things ‘from the attic to the garage’ - so long-term storage or mass storage is not my main interest. I‘m looking for a more stable link which is equally useful to the Alias or the file:// links created, when dropping files to Scrivener. So I want research materials to be accessible to Scrivener/from within Scrivener - ideally without multi-tasking - and without bloating the project size. Sorry if that wasn‘t clear.

So from this I gather that the custom links generated by DevonThink also work on iOS, which makes the cross-plattform use less of an issue.

Do you have a reason for your preference of creating hyperlinks on text rather using bookmarks?

I started testing, but noticed for example that neither macOS Quick Look nor Scrivener recognize a bookmark pointing to a PDF in DevonThink, so that Scrivener does not show a preview of the file in the sidebar unlike if I were to link a PDF directly.

Interestingly, neither macOS Quick Look nor Scrivener recognize a hyperlink to a .pdf as external bookmark either, e.g. of the form [i]somedomain.com, so it seems to be something about which types of links Scrivener can recognize and which not.

4

I should have described my workflow and my rationale more fully, but I was in a bit of a hurry. Let me enlarge.

Like some others who have posted in these forums, I adopt the method of keeping my research/reference material in one place, and my writing in another. Working with more than one application open is just the way I have always done things, and I am very used to swapping around. Command-Tab is probably the shortcut I use more than any other. Because I use Bookends as my bibliographic manager, I am more-or-less obliged to keep it open while writing, and a simple Command-Y takes me straight to it. I also use utilities like Moom to arrange windows if I need to. This makes the split editor in Scrivener less important to me than it might be to others. In short, I tend not to put reference material in Scrivener, because I am likely to have too much of it, and because I like to keep it all in one place. It is a little like having a library, and I know that if I want to consult something, I have to go to the library.

If one reason for my working method is habit, another is that – as you have found – links can be fragile. So I don’t use them unless they are bullet-proof. A link in the text of an rtf or markdown file is very robust if it points to a file in DEVONthink using the database’s item link. I can take a document from a Scrivener project, export it to a plain text file on my phone, and still have a link that connects to a pdf (for example) in DEVONthink. Alternatively, I can find a webpage using Safari on my phone, clip it to DTTG, copy the item link and put it in a text file, and open that text file in just about application I want.

In short, my philosophy would seem to be different from yours, in that being able to see my research materials in the Scrivener window is not important to me. One of the reasons why I don’t do it is that – for me – there are advantages to working in other ways. There may also be costs, but having done a lot of work in Scrivener/Bookends/DEVONthink over the years, I find that large projects are better off with just the writing in Scrivener, and links in the text rather than bookmarks, which don’t always travel well.

I may not have answered your questions well, but I will try to elucidate further if I am not clear.

5

In addition to DevonThink’s item links, you can use DT’s Reveal in Finder command to burrow into the database structure and find the actual file. Once you’ve done that, you can bookmark it in Scrivener just like any other external file, meaning that you’ll get the Quick Look preview and all the rest.

This method is not as robust, though. DevonThink guarantees that their item links are stable, even across databases. You can move the file and the link will still work. Using a direct Finder link explicitly bypasses that mechanism: you’re linking to a location, rather than an item, if that makes sense.

Like Mbbntu, I use DevonThink as my main research repository and rarely bring materials into Scrivener. In my case, that’s because I revisit the same topics repeatedly over multiple years, and across multiple Scrivener projects. Keeping the research and writing separate allows me to organize both in the way that makes the most sense for me.

Katherine

6

Thank you, I’d forgotten to mention that factor. It also applies to me, in that I expect to use the same reference material in more than one piece of written work. I can keep track of where I have used a reference by using tags (e.g. L4_Diploma, Journal02, or something of the sort) which if applied to a pdf will appear in the Finder, Bookends, and DEVONthink, and also in iOS. This might be useful to me, because I might want to find out what I have said previously in relation to a source work. Using tags, I can track down what I have written before.

This is all part of creating a network of links that does not depend on a single program (well – actually it depends on DEVONthink, now I think of it!), and will not break. Scrivener is good, but it is only a component part of what I do, not necessarily the hub of everything. If there is a hub, for me it is more DEVONthink that is at the centre of my work.

7

Thank you for your detailed replies. I didn’t notice them until now, since I didn’t get notified of an answer somehow.

I agree with all of your remarks - Scrivener is not a replacement for a knowledge-management (I used OneNote earlier) or reference-management (I used Citavi earlier) software. But I’m not trying to pull in PDFs of literature and annotate them in Scrivener. It’s more about single .docx or .pdf files which contain brainstormings or summary write-ups on single issues or the other. However, since they are very rough and often quite old they nowadays mostly serve me for purposes of extracting data, rather than my earlier writing about that data. So I would like to keep them separate from Scrivener, and simply extract from them occasionally annotate and assign a status to decide whether I need them again or not.

I understand that you are both rather enthusiastic about DevonThink and I’ve heard from others about its stability, but DevonThink has limitations which may turn it into yet another data-pit I can’t get out of - it runs only on a local machine, and only on macOS, I don’t see any open standard to export all the (meta-)data (of course I can get the files from DevonThink’s databases since they are transparent to the macOS file system, but the whole point of the database is the meta-data) and in trialing it I’ve found it repeatedly to be overloaded from a point of view of the GUI.

I also understand that I can reference the files directly inside DevonThink’s databases, as mentioned by Katherine, rather than via the file handler, but I could just as easily reference external files in the Dropbox, but since Scrivener doesn’t do relative paths, it puts me back to square one of having platform-independent external files or attachments in Scrivener, since these links only get recognized on the platform they are created (I’ve mingled around with aliases in earlier cases across macOS and Windows, but I doubt they would b e supported by Dropbox).

However, it so turns out that I actually may have found a good alternative to your use cases for DevonThink in the meantime, which ticks all of my boxes, namely Notion https://www.notion.so/, which is kind of a cloud-based hybrid between a note taking tool, task management tool, and a database. I’ve managed to create databases inside there to which I can import external research files, which I can then reuse and even embed in a file viewer inside pages, which I can share to the web via non-public links, so it would be very similar to a OneNote page with a file attached, which I can then annotate directly or on that page (as a bonus I get database features - think data properties and Excel formulas).

I’ve further managed to import those URLs as external bookmarks in the document bookmarks session, and unlike either OneNote web copies or file-handler URLs for either OneNote or DevonThink these web shares use a system, which the browser component of Scrivener can read, and I can even manage to log-in for commenting or editing in that preview, and the log-in seems to be retained across sessions (I would assume the cookies get reset after a restart of the macOS or of Scrivener) - unlike OneNote links for example (see my other post on the issues with reusing OneNote pages here: [url]https://forum.literatureandlatte.com/t/how-to-add-password-protected-urls-as-external-links-and-import-links-to-onenote-pages/50955/1]), a data-pit I have issues getting out of today.

So in short this seems to be a great solution, which is fully compatible with Scrivener. But trying to test those links on iOS I then discovered, that Scrivener for iOS does not implement document bookmarks - which I discussed in a separate post just now https://www.literatureandlatte.com/forum/viewtopic.php?f=53&t=63222 - which again, puts me back to square one. So I’m considering giving up on that endeavor of stable cross-plattform links for now, but probably will stick with using Notion of OneNote, since the database aspects of the former clearly come very close to the DevonThink use cases (the only advantage of DevonThink over Notion I see at this point is that DT has powerful OCR technology embedded, so it can index attached files, which Notion does not do at all at this point, but again I have only tested DevonThink twice for a brief time).