What is the maximum number of documents a project can have?

The Manuel says: “Scrivener can (…) handle many thousands of individual components (…) Scrivener has been tested against projects with millions of words in them (…) So for ordinary usage, you will never need to worry about limitations.”

What is “ordinary usage”? What does “many thousands of individual components” mean? And what is the result of “has been tested”? I wanted to find out, so I tested it myself.

I created a monster project. 25000 documents (research material)

  • 70% of the docs are texts: Two thirds with less than 4000 words, one third with more, up to 35000 words.
  • 10 % PDF, 1 page up to 900 pages (Scrivener Manual)
  • 10 % images
  • 10 % web pages

Positive: Almost everything works surprisingly well. Writing, scrolling, navigating in the project, searching in a document, all good. Scrivener only crashed once in several hours of testing. When restarted, the monster project was restored perfectly. But the creation of the search index takes a long time.

Negative: Searching in the project is almost no longer possible. Searching with the * to call up all documents takes 8 seconds. Typing in the search field is strongly delayed.

Conclusion: Because the search function cannot be used, it does not make sense to work with a 25000 documents project.

What is the maximum number of documents that the search function can handle? To find out, I have reduced the size of my Monster project in several steps.

To be able to work/search properly, I have to limit my project to 4000 documents. The search with the * to call up all documents now takes 2 seconds. (Almost) normal writing in the search field is possible.

Summary: Too bad. A Scrivener project with 25000 documents would be functional if the search also worked.

Hint: Scrivener offers the (hidden) option of managing two projects in one (so to speak). In the trash, not in the trash. This allows the maximum number of documents to be doubled to 8000. Documents/folders in the trash can be ignored by Scrivener. Or Scrivener can only search in them. So, only half of the documents are “in use”, even though they are all in the same project. If you do it this way, you can also work/search normally with 8000 documents.

This is of course only an approximate number. Perhaps this result would have been different if I had chosen a different doc mix.

2 Likes

The manual is vague about this because there are no limits you could run into (unfathomable 64-bit numbers aside), only what time limits you are willing to tolerate during certain operations. This will mainly be search, backup and crash recovery, and how fast hardware gets over the years will change all of that. What was once less tolerable in 2012, now is perhaps fine. Backup times can be mitigated by setting up your own automation, and rebuilds by not using mobile to edit it (but the size is probably reason enough not to sync it). You can’t avoid the rare crash, but coffee breaks never hurt.

I do agree that we need a short delay timer on the search field though, or an optional commit model, the urge to show results instantly is typically the right one, but not having one does put a gradient on the scalability of the UI that is reached long before usage really would otherwise effectively need to be scaled back.

1 Like

In this forum, people have repeatedly expressed the wish for Scrivener to be able to search in different (open) projects at the same time. Some would even like to be able to search in closed projects. This is of course not possible. Not even DT can do that.

I wouldn’t object to a faster search function, of course :slightly_smiling_face: What seems more realistic to me (at the moment) is to improve what Scrivener can already do. The search in a project could be limited to different parts. Search only in parts A and B but not in parts C and D. If this could be set quickly and easily, it would be wonderful. :slightly_smiling_face:

Edit: I don’t know, maybe it would also help if you could deactivate “live” search. So you type in the search terms and then hit enter.

I may be misunderstanding you, but you can already do this, by using the Project Search ‘Search Binder Selection only’ setting, can’t you?

Yes, that’s right. But I find it inconvenient to work with this menu and to have to change the settings in it. Before searching and after searching.

I do it like this. some of the documents in my project have this icon :diamonds: in the metadata. This divides the project into two parts.

To exclude docs with the icon from the search, I type this.

To search only in docs with the icon, I type the opposite.

The problem is, to exclude documents/folders from the search, Scrivener has to search for them first. :joy:Which is of course a disadvantage because it slows down the search. The trash method has an advantage here, if only it were easier to set.

Can you try this, please?

Create a collection from your diamond metadata. Don’t search for any text or anything else. You just want the docs with metadata = diamond. Save the result as a collection. (This is a one-time step, of course).

Now, whenever you just want to search in the diamond docs, click in the collection, cmd-a to select all the documents, then cmd-opt-r to reveal them in the binder.

Now you can do a ‘Selected Docs only’ search on all / text etc, without complicating the search by included the metadata element as you’ve reduced the target document number with the collection.

I can’t test how much (if any) quicker this would be because I don’t have any billion-document projects lying around, but does it make a difference to the speed?

That’s a clever thought … but a collection is nothing more than a search. Whether I click on a collection or enter the diamond in the search field, Scrivener does exactly the same afterwards.

The trick with the collection is actually very useful for complex searches for which Scrivener would need several steps. I use that too. For example this collection searches for all documents that have the label green, a certain status and the keyword “neu”. The search is not faster, but it saves me typing.

image

But typing is the right keyword. With very large projects, typing in the search field is delayed so much that it is actually no longer possible. Deactivating the “live” function would help against this. Maybe :slightly_smiling_face:

@AmberV , would it be complicated to add an option enable/disable “life” search?

1 Like

Yes, again there are probably some tactics we could take, involving modifications to the interface and such, but these are all ideas we’ve had on the shelf for a while and probably all best addressed in a more comprehensive look at the feature. I don’t feel that putting a hard commit into the current interface would complement it, and would be more a case of trying to accommodate a rare edge case with something that isn’t in harmony with the design.

I see, I was just asking.