Scrivener project with 32,000 files = sluggish?

Hello,

I recently had the need to import about 30,000 small .txt files (2 short paragraphs each at most) into one of my Scrivener project. Although I had organized them into folders with about a thousand files in each, I noticed recurrent sluggishness in navigating the binder or editing in the editor. The dreaded rainbow wheel would appear for a few seconds, as if to comfort me that everything was still going on…Is there a way to deal with this number of files without slowing down Scrivener, for example organizing the files with less files per folders?

Now, I have searched the net to see if Scrivener had a technical limitation as to the maximum number of files and could not find anything except a recent writing, a theoretical hypothesis, on Scrivener’s blog:

A Scrivener project is flexible; there is no theoretical limit to its size, or to the number of folders and files it can contain. While elements you may add to your Research folder, such as graphics, PDFs, or videos, may take up a lot of space, text doesn’t. You could store millions of words in a single Scrivener project.

Because of the way Scrivener works, project size does not affect the app‘s performance. Unless you have million–word text files, which is unlikely, each file you work with in your project does not require a lot of computing resources. So you could store 100 articles, or a dozen books in a single project, if you wish, and the app wouldn’t be slowed down by all those texts.

Of course, I would much prefer this to be the case—and I am way beyond a 100 files—but this is not what I am experiencing for the moment. Now, to add a couple more infos, the total size of this specific Scrivener project is no more than 250MB. I had projects going north of 1.5GB and never noticed any slowing down, except upon opening. Here’s a screenshot of the statistics in case it is useful. Any pointers highly appreciated!

To add to this, looking in the Activity Monitor, Scrivener has plenty of room memory wise and barely make any use of CPU power. So I am still trying to figure out where the bottleneck could be.

That’s an obvious bottleneck already, but the question isn’t how many documents in each folder or how big they are. The question is how many you open at the same time.

If you have a thousand folders, each with 32 documents, and you click on one of those folders in scrivenings mode, you’re opening 32 files. Not a big deal.

If a hundred of those folders are grouped in a higher level folder, and you click on THAT folder in scrivenings mode, you’re opening 3,200 files. That’s a big deal.

Thanks Bob. After experimenting, even if I open a single file located on the highest level folder that contain only one file, there is a slow down, it seems just from having this amount of files in the project. Each document is really not longer than a couple paragraphs.

Yes, makes sense. I am not opening the folders content (each about 1,000 files) in scrivening mode, which of course would result in quite some processing time. In my experiments, even opening a document on the highest level gets some kind of slow down when navigating the binder or editing.

In what mode are you opening them? Do you never click on those folders, or if you do, it’s only in Document mode?

Don’t forget you always have the .scrivx open. It has entries for every file and folder, whether they’re open or not. Here are the entries for 4 or 5 files in my WIP:

I never click on those folders as they would be displayed as scrivenings in the editor’s panel. I am only ‘unfolding’ them with the arrow on the left to select one file in order to edit it.

Thanks that is interesting. Although I am not sure how much really gets loaded in the memory when not open. After some more testing, it seems the slowing down I experience navigating the binder is dependent upon the combination of several parameters, for example:

  • how many files are to be displayed when unfolding a specific folder
  • how deeply nested this folder is.

For example, if I move the “mother of all” folder containing 32 sub-folders with a 1000 files in each at the ‘root level’, then it seems that all slow downs disappear, whether working within those folders or in other folders in the binder.
If this “mother of all” is nested within another folder containing other folders, it seems to make use of much more CPU resources and results in slow downs.

So I guess, where my question truly lies is: what are the best practice tips when dealing with an enormous amount of files in one Scrivener project? It seems that reducing the number of files present in each folder and having those heavily loaded folders the closest to the ‘root folder’ results in significant performance gain compared to not doing that. Would there be anything else?

If you unfold a folder with thousands of files, don’t you think Scrivener has to access them, to get their files names and icons at least?

What do you mean, move it at the root level? Do you mean move it to the root level, as in outside the Draft folder?

My advice is not to do that, or if you must, do it outside the project, in the macOS file system designed for large numbers of files.

Absolutely. What I mean is that the content is probably not pre-loaded. But I don’t know the intricacies of Scrivener’s code, so I can’t really tell.

Yes, exactly. On the same level as the Draft folder.

Thanks for your input. It’s definitely pushing it. Although, for now it seems that moving the loaded folder at the root level worked as I am not encountering slowing downs anymore. Let’s see if that continues to be the case.

Actually, I don’t think Scrivener should have to access the files, just their .scrivx entries.

Maybe the difference is in computation of project statistics, if those are set to ignore word counts for documents outside the Draft.