Work Organization

I’ve just started using Scrivener a few months ago. I write short prose poetry so I usually have 100’s of single files in Word. I am staring to move ALL of my work into a new Project File called “All My Work”. And now I am moving pieces into folders by subject/date/etc. So, when the migration is finished, I will essentially have all of my writing in one single Project.

As a fairly new user I was hoping for some input as to the soundness of this decision.

I have so much trouble looking at folders on my laptop with 100’s of files, it seems like a perfect solution for me, but am wondering if it is risky.

Thanks!

I don’t think it’s “risky”. It’s all down to how you perceive your work structure. Keep in mind, though, Scrivener designed to create an environment that provides a structured document that is produced by the organisation in the Binder. If you intend to compile to a document that is structured by “subject/date/etc”, then I guess all is well. Or when comes time to publish in a different structure perhaps you can duplicate your project to a new one and restructure (drag and drop) to what it is you want differently. Something like that. Am I explaining myself sufficiently?

Edit: If it were me with my set of writing tools, I’d use DEVONthink to hold all these files, and then export a selection (or all) into Scrivener when time comes to produce a deliverable document. Scrivener a tool for writing/producing. DEVONthink to hold the files. (Even without DEVONthink, maybe just rely on keeping the files in the macOS system and use Finder to organise them, Word to edit them.).

1 Like

What do you actually want to do with the files once you import them? Scrivener is not really designed as a file manager or a database.

With hundreds of files, your future self might thank you if you did some kind of clustering first. If you see yourself assembling a chapbook about cherry blossoms, say, it might make sense to put the nature poems in a different project from the poems about urban landscapes.

2 Likes

You want to combine everything in one project … 100’s of files … so a kind of database… I can only say in general terms that this is possible. But with restrictions. Scrivener has no problem with large projects, navigating in the binder, scrolling, moving files … it all works well.

But the search function reaches its limits at some point. When this point is reached, you can use a few tricks or move some of the material to a new project.

Thanks everyone for the insights. So much to learn! I’m not really using Scrivener for producing a document for publishing, I guess I am using it more for document management. I will take the advise about grouping poems, perhaps by using Collections or labels. At this point I am just trying to get things out of Word and into one place. Scrivener makes it so easy to drag documents around, make folders, make collections…even things like being able to change the icon images is a tremendous help for someone like me that has “attention challenges and difficulty organizing”.

My big fear was having all of my work in one Project…if a crash happened. But I suppose the same would happen if I had everything in one Scrivener Project or if I had hundreds of Word files!

Thanks again for the responses. This is such a supportive and helpful community.

J

The way to guard against data loss is to have a good backup strategy. Ideally at least three: Scrivener’s automatic backups, an external backup that runs automatically (like a Mac Time Machine volume), and some form of offsite backup. (Not the same thing as synchronizing to a cloud service.)

1 Like

Crashing with Scrivener is a lot less scary than single-document programs that lack auto-save.

If you’ve imported hundreds of old documents and done a little organising with index card text, you may already have close to a thousand literal files on the disk. A project that has been heavily outlined and annotated can have tens of thousands, as each entry in the outliner can be represented by up to six different files, and an indefinite number of associated snapshot files (where the text is saved separately and in an immutable fashion for older revisions).

Given how many files can be in a project, Scrivener does not open all of this when you load the project. In most cases it may only load around a dozen files, and most of those will be settings, the file that describes your binder, a search index and maybe some content for the main editor to show. Content files are only loaded when you click on them, and they are periodically flushed from memory if they haven’t been saved or looked at in a while. There are exceptions, but the take-away is that it only loads what is necessary to show what is in front of you in the window.

What this means in practical terms is that your average editing session is likely only going to touch a tiny fraction of total number of files in the project, and of that total that hasn’t saved yet at the moment of the crash, an even more tiny fraction of that fraction. Often a crash or sudden quit results in the loss of a few words that you were typing seconds before.

So we’ve narrowed down what could in theory be corrupted during a crash, from thousands to maybe one or two files, but what about those files? There you’re pretty safe as well because by default Scrivener auto-saves after two idle seconds, which is probably more often than you might think. Just stopping to think about how often that might happen probably took longer than two seconds, and now your stuff is now saved.

That leaves the truly rare event of when the crash or power outage happens in the moment (and we’re talking microseconds here) a file is being saved. And there, thanks to modern file systems there are things happening beneath the software level that help prevent file corruption in such cases.

Real live bona fide data corruption is exceedingly rare thanks to all of the above. You may hear otherwise, but that is because a lot of people use the word ‘corruption’ as a blanket statement to describe any situation where something doesn’t restart the way they left it, essentially (and a good 99.9% of situations like that involve poor use or configuration of cloud sync technology—i.e. nothing to do with Scrivener, but you know how everyone treats the messenger).


On the broader and more preferential matter of how to organise your work, my main answer to that would be: don’t worry about it.

It is a trivial matter to split a project up into several, if it comes to the point where you don’t like working that way. It’s also very easy to combine multiple projects together if you tire of having to switch between them constantly and are finding they could benefit from sharing a lot of text.

You can’t really make many mistakes here, in other words.

I would even categorise performance as preferential. We’re talking a gradual increase in the duration of some operations, and whether that is okay or not okay is a matter of personal tolerance. There are even settings in Scrivener to avoid the worst of these as well. Archive-style projects like this, that rarely change once settled in, can certainly have their backup-on-close default behaviour shut off. Let your normal external backup systems handle them. While creating a redundant copy of the entire project whenever you close it is great for a manuscript you’re in the middle of writing, it’s wasteful and unnecessary for something that rarely changes.

What I would consider of much more immediate impact, than stability or performance, is whether or not it makes sense for all of your writings to share the same pool of metadata. For something like this, an historic archive that is essentially a level up of File Explorer—it’s probably fine.

This is kind of thinking of the future, rather than what you are doing now: once you start getting into using Scrivener to write, from the ground up, and you use those keywords and labels, and start making your own custom metadata fields to track important dates and whatnot—that kind of stuff will tend toward being very integral and unique to each project. So merging such a finished writing project into an archive shared by hundreds of others would either mean an ever growing inscrutable mess of keywords and such, or it would mean losing that kind of information going forward.

In summary: it’s hard to make mistakes, data corruption has no more an elevated risk factor with a million words than ten words, and the software can take many long books without performance drops. All together that means: do what you want.

4 Likes

Not that this is necessary, but I can confirm (almost) everything AmberV writes from my own experience.

Data loss (corruption) is a marginal issue. I have crashed very large projects (and small ones) on purpose. Many times. Result: The data loss was zero. Scrivener does a great job here.

This view is a bit optimistic :slightly_smiling_face:

It’s difficult to argue with concrete numbers, but at some point, when a project grows continuously, it’s no longer possible to work with it. The problem is exclusively the search function. Everything else works well in my experience.

So, when a project has reached a critical size (you can’t miss this point) it is necessary to copy some of the material to other projects. This works very simply. There is nothing wrong with having several projects open at the same time. I have given up my irrational resistance to this. :slightly_smiling_face:

Nevertheless, even a large projects can be optimized in itself so that it remains searchable. As long as it is research material, a project can be “divided” into three independent parts. “Trash”, “Templates” and the rest. (Of course, these folders can also be named differently). In a way you then have three projects in one. These three parts, can be searched independently of each other (see search menu). This way Scrivener can also handle, for example, 15,000 documents (or more) in one project without any problems, because these documents are divided into three units of approximately equal size.

For large projects, it is generally advisable not to search the entire project but only individual folders (search only in Binder selection). Usually you have an idea where a certain document might be.

2 Likes

Yes, I think that is about what I was saying, that as you add more and more to a project, certain functions will slow down and different people are going to have different tolerances for that. Personally my breaking point was never search itself, but rather how long it takes to open a project that didn’t close cleanly, and needed to have its search index rebuilt. That maybe isn’t something everyone would encounter, but for someone that runs a lot of test builds and deliberately tries to break Scrivener all day, it was a factor. For other people it might be sync with iOS, which has its own bottlenecks in processing what has changed out of a large data pool.

I don’t think I’m being “optimistic” though, and that what you’re talking about is in another realm of outlier than what I was referring to. Let’s look at some numbers.

  • I imported an archive of text that consists of 13,000 text files. As for the more important figures, text bulk within them, they constitute 7.6 million words, or 45m characters.

    It took Scrivener, on a baseline Mac Mini M2, about 15 seconds to import the entirety of this. That’s not even worth mentioning, honestly. Even if it took 15 minutes, that’s something you only do once.

  • Under normal conditions this project takes about five seconds to close, and about the same to load. We can make that slower on purpose, like loading 2.5m words into a Scrivenings session and leaving it open when closing, but these are the kinds of things one learns not to do.

  • Now to look at three of the search tools that would work against the whole project:

    • My first test is with Quick Search, or the URL bar looking thing in the toolbar that lets you jump from one document to another swiftly. This is a very important efficiency tool, so we would want it to work quickly. I could perceive no lag or stuttering while typing in enough words to narrow down the title list to something usable. Navigation was instantaneous.

    • The second test used Project Search, which polls against (by default) a much more comprehensive sweep of text. Though for the purposes of this simple test, it’s about the same as the first (titles + text) because I haven’t spent hours adding metadata, and titles and text are all that exist in the project.

      Again, the search results are instantaneous, particularly where I had a good idea of what I was looking for. Where I ran into noticeable slow-downs (1 to 3 seconds) is where I would pause on a very common word, and then it would indeed take a few moments to assemble a 10k list of items or whatever (that’s a lot of UI to build, and not something Quick Search tries to do so it never hits these slowdowns). Most people would consider that acceptable.

    • Project replace: probably not the kind of tool that one would use without scoping it to the point where it doesn’t matter, but that probably is one that would be slow. If I were to replace the word “the” with “yikes”, without any constraints, yeah that would take a while. Again though, who is going to actually do that in an archive. And in the rare once-in-five-years event where you do, who cares if you need to take a tea break!

So, with all of that said, I think it is important to once again return to what you aren’t qualifying, because while I think 7.5m words across 13k entries is a pretty good stress test of what most people consider when saying “all of my writings” (and certainly fits with my statement of “many long books” with a lot of room to spare; perhaps Danielle Steel would need to give pause though), it’s important to consider that this test project I made is all of 125mb.

That’s… nothing, compared to the outlier you are talking about.

For large projects, it is generally advisable not to search the entire project but only individual folders (search only in Binder selection). Usually you have an idea where a certain document might be.

And yes, one does need to get a bit clever if they have gigabytes of text they are searching against. That’s probably true of just about any software outside of systems whose sole purpose is being a search engine. There are a few exceptions, but they are built to be that exception from the ground up. Again, Scrivener mainly claims to work well against book-length works. That it can go well beyond that speaks to it being a robust tool in the grand scheme of things.

3 Likes

It seems to me that we are sitting opposite each other and someone is holding up a book between us. You see the front and I see the back. Although we see the same book, we see something different.

As you say yourself, how long it takes to rebuild the search index is probably irrelevant for most people. The same applies to opening or closing a project.

But when I do tests (with the project search) (with 12,000 documents), I get different results than you. My waiting times are much longer.

Perhaps the result also has something to do with the type of documents (text, PDF, web pages). Whatever.

As for the quick search, it’s probably me, but I confess that I never understood exactly what was being found.

This is not a criticism. I just would like to understand it. :slightly_smiling_face:

Although searches are carried out in the title and in the text, the search terms cannot be mixed. That is why this is not found. Correct?

And words must be complete and in the correct order. Is that why this is not found?

Quick Search is an “exact phrase” search, with results grouped by titles, synopses, and body text. See Section 11.5 in the (Mac) manual for more information.

I really don’t know what else I can add. You insisting that you have found that spot for yourself with your data set is only reinforcing what I have been saying since the start. :person_shrugging: Perhaps there is a forest for the trees thing going on here.

1 Like

I’m just running into that issue! I guess I really need a document management system. I just write and occasionally post to my website by copy and paste.