[RESOLVED] Disaster: 7 weeks work lost

Five hours ago, a disaster occurred. Out of the blue, and falsely, Scrivener suddenly said that MAIN.scriv must be closed because the file was in the Trash.

“No big deal,” I thought. After the forced closing, I went to File > Recent Projects > and opened MAIN. Then the famous dialog came up about “Scrivener will rebuild its search indexes before opening the project” and asked whether to COPY or CONTINUE. Since MAIN was NOT open elsewhere, I figured it was safe to continue.

MAIN opened, and I noticed that seven weeks of documents in the file were gone. It reverted to 7/31.

  1. FIRST, this is a warning to others. If you set Scrivener preferences to backup upon “closing,” your files will NOT be backed up. This setting only triggers backups when you MANUALLY close your file. Quitting Scrivener, incredibly, does NOT trigger a backup even with “Back up on project close” turned on.

  2. SECOND, when the warning dialog says:

“Before continuing, please ensure that this project is not open anywhere else, otherwise data could be lost. If you are not sure, you can choose ‘Make a Copy’ to work from a copy.” \

… beware that Scrivener will ACT like the file is STILL OPEN even though Scrivener just closed it and says that it is not open! It will rebuild the index and all work since the last time that you MANUALLY closed the file will be destroyed.

  1. Then I looked in ~/Library/Application Support/Scrivener/Backups and found the latest backup … and it was 10 days old! (9/13). I guess that was the last time I physically hit CMD-W. All the Quits and Reboots since then never triggered a single backup! Great.

  2. WHAT ABOUT TIME MACHINE!? This is the worst part of all. ALL the Time Machine backups—even the ones made an hour before the "I’m closing this window because X is in the Trash” lie—are also missing 7 weeks worth of work. This means that Scrivener has been pseudo-saving the file all along—that is, not actually touching the file in its Finder location—for seven weeks! All backups of X in Time Machine of X—which I add to EVERY DAY—have 7/31 as their “Modified” date!!!

  3. Yes, I looked in inside ~/Documents/MAIN.scriv Recovered and found a pittance of files—6 in all. The other 100 or so are not included among them.

Surely, the last 7 weeks of work must exist somewhere. Where was Scrivener saving the litter internal RTF documents all this time? Can someone please help? I’ve tried using Finder search using quote-wrapped phrases from the recent RTFs, but nothing shows.

Thanks in advance.

Firstly, have you checked in the Trash folder to see if there is a copy of the project in there? It kind of sounds like your main project got trashed somehow, and when you used the recent menu it loaded an older version (one that, it seems, was not closed cleanly the last time you used it, given the warning message). It would be very strange for Scrivener to think the project was in the trash when it isn’t—as far as I know that shouldn’t be possible.

That could also explain why Time Machine is “confused”. If you’re trying to restore from this older copy of the project, and it hasn’t been changed in a while, then there would be nothing to restore there. But if the actual copy you were working on in this past week is trashed, you’d have to restore it to where it was. Hopefully then you wouldn’t need to do anything further, but if so, Time Machine should at least be seeing recent changes at the location it came from.

The next thing I would try is using the File ▸ Find All Projects in Spotlight menu command. Go through the list and add the modification date column to the search window, looking for anything edited recently.

beware that Scrivener will ACT like the file is STILL OPEN even though Scrivener just closed it and says that it is not open! It will rebuild the index and all work since the last time that you MANUALLY closed the file will be destroyed.

I’m wondering if perhaps Scrivener isn’t actually closing well on your system? Do you ever get any crash report logs when starting up the next day, for example? That might explain why the project isn’t backing up reliably when you quit—if it’s actually crashing when you quit, then it wouldn’t get to that point.

Ordinarily it would of course back up every open project when you use ⌘Q. There is no difference between doing that and press ⌘W for every open project, in terms of how software shuts down on a Mac. Same goes for reboot, when a program gets the message from the system that it going down, it starts going through the close-window checklist on every open window, which in Scrivener will include backing up under default settings. If I shut down with five projects open, they will all back up, and so sometimes it can take a while to shut down.

So, obviously something isn’t happening as designed. It may be a good idea to reinstall the software and reset your preferences to make sure nothing is damaged with either of those things. I would then run a quick test, after doing that, of loading the Interactive Tutorial from the help menu and then quitting. Do you see a tutorial backup file pop up in the correct location?

This means that Scrivener has been pseudo-saving the file all along—that is, not actually touching the file in its Finder location—for seven weeks! All backups of X in Time Machine of X—which I add to EVERY DAY—have 7/31 as their “Modified” date!!!

If you’ve been quitting the software and rebooting regularly, then how would it be loading the information it “pseudo-saved” all this time, that you’ve been working on? Again you seem to be describing a scenario more along the lines of having copied your project at some point and ended up working in the copy you weren’t thinking you were, and then it somehow got trashed. There is otherwise no way to explain how it loaded the stuff you were writing over the past week, after rebooting or whatever.

The other clue here is that often so much as merely loading the project will bump the modification date, which lends credence to the fact that you actually haven’t opened the copy of the project in Documents for a while now.

One last thing to check, a bit of a sanity check on the system, but are other files you’ve worked on in the past week in a good state? If your user folder seems to have been stuck at a certain point, and there aren’t many things listed in Time Machine as having changed, there may be bigger problems afoot.

1 Like

Hello. Thanks for the quick reply.

  1. Firstly, have you checked in the Trash folder

Yes.

  1. The next thing I would try is using the File ▸ Find All Projects in Spotlight menu command. Go through the list and add the modification date column to the search window, looking for anything edited recently.

In the Spotlight search I have a 9/24 that was the 9/13 that I unzipped from the Scrivener backups folder (2MB), a bunch of 9/24s that I restored from Time Machine (all last modified 7/31: 900KB), but none is the 3MB file I was working on 6 hours ago.

All your other helps are very informative. And for now I’d like to find the last few weeks of RTFs. Is there anything like a temp workspace where Scrivener saves? (Like something in /var/folders/…?)

Are you positive that in the past week you have seen work you have done during that week? Sometimes I write forwards only and don’t really go back to what I’ve done in recent days, and may not notice things were not where I thought they should be. If you’ve definitely seen the work you’ve been doing, loading and closing the project as you have, then it must be somewhere, unless it was just a few moments ago thoroughly deleted by some process (you don’t happen to run any automation or system cleaning tools by any chance? If you do, do they have logs you can consult?).

All your other helps are very informative. And for now I’d like to find the last few weeks of RTFs. Is there anything like a temp workspace where Scrivener saves? (Like something in /var/folders/…?)

No, there is nothing as complicated as that going on. For a project to load successfully all of the data must be in the spot you see it, in Finder, and so long as the project is open, all saves go directly back into that spot. That’s under ideal conditions of course, if something has gone tragically haywire that may not happen, but if it doesn’t, there is no secondary place where that stuff would go. And to protect against that kind of failure, there are numerous safeguards that make sure saving is happening correctly. You ran into one of them—which may have been a false positive, but another is if the system reports a file cannot be saved. In that case it will shut down with a similar error and dump anything it hasn’t saved yet to the recovery folders you found. So that would handle cases where the disk you were working on suddenly lost power, or if it was loaded over NAS and the network went down, etc.

For it to actually not write anything that you are typing into the software, and not throw an error, basically Scrivener itself would have to be fooled into thinking everything is okay. That’s why I worried about larger problems, perhaps an overly full disk, or one that is need of disk repair—where things are so wrong with the system itself that it’s not even properly telling software that it can’t save what it just tried to save. There isn’t anything Scrivener can do about that, because it is misinformed at that point, into thinking everything is fine.

All your other helps are very informative. And for now I’d like to find the last few weeks of RTFs. Is there anything like a temp workspace where Scrivener saves? (Like something in /var/folders/…?)

Well one quicker test you could try is searching for phrases you’ve written recently. That may not always work though, given how Spotlight can be a bit touch and go when it comes to scouring package contents. I would run a positive test first, by searching for a phrase you know to be in the older project. If it, or one of its internal RTF files, comes back in Spotlight, then you should be good to go.

Failing that, you could run a “deep search” that bypasses Spotlight (which being an indexed search engine, can sometimes lose track of reality). It will likely take a long time to execute, but if the data is in your user folder somewhere, it should find it.

  1. In Finder use the Go ▸ Utilities command, and double-click the Terminal icon.

  2. Copy and paste the following command into the terminal window and press return:

    find ~ -type f -iname '*bak*zip' > ~/Desktop/potential_scrivener_backups.txt; find ~ -type d -name '*.scriv' > ~/Desktop/project_packages.txt
    

Like I say, you’re probably going to be waiting a while for that to run, as it will be physically hunting through every single folder and file name in your user folder, of which there may be hundreds of thousands. You will probably also see some permission errors, because Apple feels you shouldn’t have access to some areas of your user folder. These can be ignored—if you can’t search in there, you certainly can’t navigate into and save files there.

The commands will create two txt files on the Desktop:

  • potential_scrivener_backups.txt: these are .zip files that contain the letters “bak” somewhere within them. So there likely will be false positives in here, but the good stuff should be obvious given Scrivener’s easy to read naming scheme. This will detect cases where the you may have switched backup folders multiple times, or reset preferences and lost track of the motherlode.
  • project_packages.txt: any directory type item that ends in “.scriv” will be listed. That’s all it will be—project may be damaged or who knows what, but this will find everything under that criteria.

If there are hits in these files, they will contain lines of text where each line is a full system path to a file (or folder in the second case). This path can be copied and pasted into Finder’s Go/Go To Folder... menu command (even if the path is to a file, it will actually select that file for you as well as open the folder).

Now if you want, you could also run this command:

find / -type f -name '*bak*zip' > ~/Desktop/full-potential_scrivener_backups.txt; find / -type d -name '*.scriv' > ~/Desktop/full-project_packages.txt

The only difference here is that instead of searching your user folder, this will search everything, including all connected drives. Obviously that is going to take a really long time, so you might want to set that one up to run before you go out for errands or something. That will find projects absolutely anywhere, even if they somehow ended up getting accidentally saved inside of applications themselves (we’ve seen everything over the years). It will even hunt down places Finder isn’t “allowed” to navigate to without providing a direct path.

I would unplug the Time Machine drive before running that, if you choose to. That will greatly bloat how long it takes to run, but it will return a list of all backups and backed up projects too, so if you want to be more thorough leave it on. At that point I’d probably set this to run over night and make sure your energy saver settings keep the machine running.

2 Likes

Thank you very, very much for this super comprehensive and knowledgeable response. I’m running the full find now as I finally get to bed! I’ll reply with my report when I awake. Thanks again for being so through and insightful!

1 Like

By the way, a preference I would consider changing is in the Backup pane, so that a backup is created every time you manually save. There is rarely ever a reason to manually save, so that is something you can use now and then to make a full backup. I use that setting, and take a few during the day whenever working on an important project. I also bump the number of saved backups to 25, so that I have more than a week’s worth (assuming two or three on average backups a day). But this would be good for you, since the software doesn’t seem to be reliably closing on your system.

Secondly, I saw elsewhere you were keeping this project on Desktop. Double and triple check you aren’t using something like “CleanMyMac”. The symptoms fit what you say precisely, where some tool comes along and thinks your project is clutter, deletes it even while you’re working on it, and you get the warning message in Scrivener. Going forward, avoid using the Desktop for heavy-duty work. It’s an unreliable working area given how Apple uses it, and that’s one reason why some of these cleaner tools try to keep it clean. An overloaded Desktop folder can lead to system instability and slowdowns.

1 Like

UPDATE: I found the file tucked inside a folder in ./Trash/Trash 00-48-20-799!

So Amber’s first hypothesis was correct, and I never thought to look, not in the Trash, but in a subfolder within the Trash.

Many of you probably recognize this “./Trash/Trash HH-MM-SS-µS” format … it’s DevonThink!

Yes, I had indexed (not imported) a scriv file to DevonThink and DT moved the original external file to the Trash even though I specified “Only In Database” option during trashing. That’s a mite concerning, but not as bad as all the paranoid explanatory metaphysics I came up with in the midst of the crisis.

Well, I used to enjoy some convenience from having scriv files indexed inside DT. But no more! Too much trauma. Lesson learned.

Thanks to the smart, caring people in this thread, and especially Amber for her astonishingly thorough replies that considered all the options. They ought to be in a help museum somewhere.

3 Likes

I’d never turn on an automatic cleaner option. Even with supervision (when you have to approve their deletions), they can get you in trouble.

The DevonTechnologies support team is pretty responsive, and I’m sure they’d like to know about this.

The best way, IMHO, to integrate Scrivener and DEVONthink is to use both in adjacent MacOS windows. Nothing really is achieved by importing/indexing Scrivener projects that I know of, other than breaking one or the other. As usual, might be wrong.

Glad you found your work, and that you could follow AmberV’s guidelines! I use DT’s indexing feature quite a lot, and the Only in Database delete option occasionally, so that is a little worrying. If you get a chance, please do let them know.

Agreed.