lost work!

Scrivener Scrivener for macOS
1

Hi all–sorry for what is probably a repeat of other posts, but I don’t have the heart to search through all of the “lost data” posts just now, and I’m wondering if there are any quick tips you might offer for where a week of work might now be. Here’s the story:
I’m using the latest version of Scrivener (1.51) on a macbook pro, with Time Machine with all software updated. I had just finished some work and had made a cosmetic change (or tried to)–changing the background color for full screen mode from black to dark blue. The setting had not “taken,” and so I hit command-Q to quit Scrivener (something that, I had understood, perhaps incorrectly, would also close the current project–a 7 meg all-text book project–correctly before then quitting the program. When I then restarted Scriv, to see my new setting in effect, I got the familiar message:

“Warning This project was either not closed properly or is currently open on another machine. Please not that f the same project is open on two machines concurrently, data could be lost. Before the project can be opened, its search strings need re-synchronising. This process could talk several minutes. Continue?”

I may have also gotten another message about an older version of a file, and updating that file, which I said “yes” to.

But to my immense dismay, when I saw my book project on the screen, it was a version from some weeks ago–not exactly sure when / which, and–further dismaying–it seems to that the Time Machine backup is no different, though Time Machine had already made a couple of backups this morning. Is Time Machine unable to register / note changes to Scrivener projects–and is it therefore irrelevant as a backup for Scriv?

And, even more to the point–does anyone have tips for how I might go about recovering a depressingly large quantity of now-lost work?
thanks for any and all help you can offer,
John

2

Using Time Machine can be tricky with Scrivener, not because it does not register changes, but because it registers changes while you are working. These copies which are made while you are working will be by default “suspect” for the same reason you wouldn’t want to open a project file twice, or move it while it is open. The trick is to find a copy that was made while absolutely certain the project was closed. A good rule of thumb is to find a copy at the end of a sequence of back-ups. However this is not a certainty as you might have closed the project and shut down the computer before it had a chance to make a final back-up. If you turn on the computer and immediately open the project file, it might never have had a chance to make a clean “closed copy” of the file.

Others have experienced the same symptoms you are describing, and to date no cause has been discovered. It is very bizarre, to say the least. Internally, Scrivener uses the same system level file write commands that most other Macintosh programs use, and it doesn’t itself do any retention of old data internally. All of these bugs sound like Time Machine errors, but there have been cases where Time Machine was not in the mix. Some have had work reverted up to six months prior, which makes even less sense than what you are describing.

One thing you can check for is recovered files, which will be found in the Documents folder. Scrivener will dump data here if it is “confused” about where it should go. It will also put data into a recovered area in the Binder of the project itself under different circumstances.

3

thank you, Amber, for your characteristically helpful and detailed response. I’ll check all this out. The lost data phenomenon was not, I believe, connected to Time Machine in any direct way, but you’ve given me some hope that I can discover an at-least-fairly-recent copy in TM in any case. I’ll poke around in the Docs folder too.
On another (and entirely unrelated) front: I notice you’re from Portland; my son is starting as a first-year student at Lewis & Clark this fall. My wife and I are looking forward to getting to know Portland a bit . . .
yours,
John

4

Again–forgive overlaps with existing posts, but this has already taken so much of my time that I don’t have the additional hours I’d need to sift through other posts that might be relevant. The real purpose of this post is to give Keith and any others who’re working on the program more input on what is, clearly, a not-uncommon-enough and very serious / mysterious problem.

On further reflection, it seems to me that, with the upgrade to Scriv. 1.5 / 1.51, there was introduced some kind of version / date / set of problems, such that Scriv. sometimes thinks that there are multiple versions of a project that are open, or that the project is open on two computers or that a project was not closed properly or that the project was never updated from an older project created with an earlier version of Scriv.

(I’m assuming, by the way, that quitting Scriv. with Command Q will automatically close all projects correctly, as that keyboard command does with every other Mac program; if this isn’t the case, that’d be something to address in 2.0).

There also was a message, before I lost my data, concerning updating a project created on an older version of Scriv). Though I’d been working with this project under 1.51 ever since 1.51 came out, I said “yes” to Scriv’s request that it update the project to be compatible with the current version. I find it hard to believe that this would have caused a problem, but my feeling now is that the “older version” message was itself a symptom of whatever the condition was that was just about to cause me to lose so much work.

again, I don’t know if any of these notes help. And I’m afraid that, much as I love working in the program, if it’s become as fickle as my own experience and other “lost data” messages suggest, I’m going to have to stop using it. One of the delightful things about the program is that one can have many many files–all parts of, in my case, a book project, at one’s fingertips for working on. But it’s also one of the most profound liabilities of working in the program in its current state, it seems to me, since to lose a week’s work on a single file is serious, but to lose a week’s work on a whole cluster of files is, well, correspondingly more serious.

(p.s.: about Time Machine and versions; I’ve now looked at about a dozen TM earlier versions of this project and none of them have incorporated the changes I’d made to a good many internal documents in my Scriv. project. I’m of course kicking myself for not doing a “save backup” zip file on my external drive, but, well, I’d thought TM was doing that backing up for me. And I was convinced that the 2 second auto-save in Scriv. would also be keeping things safe. There are no Scriv docs, by the way, in my Docs directory, nor is there a recovered files folder in my binder . . .)

5

From your description, the experience sounds regretfully very similar to the other few cases. You are right, it is uncommon, and it hasn’t even really been verified that it is a Scrivener problem at all. It could even be as deep as HFS+, or the USB cache system (for those users who were using externals), Spotlight database, or something along those lines. It is extremely weird, and fortunately quite rare, and yes the only real solution is to back stuff up, a lot. Should be done anyway, as you noted. Thanks for posting your details to add to the existing reports; sorry I can’t say much more other than: Yeah, that’s been seen before and it is rare, but it looks like you are going to have to rewrite. :frowning:

Incidentally, Cmd-Q should be perfectly safe, even though it seems to have instigated the condition here (though it could have just illuminated an already existing data problem). While that may or may not be tied together with the bug, under ordinary conditions Scrivener cleans up projects and closes them all properly when a Quit command is received.

6

Most if not all the cases mentioned involve Time Machine. I have found TM to be very buggy and have reported many problems to Apple. I no longer use TM except in manual mode. I close all programs and make zip back-ups of all my files that are package-based. Then and only then, do I use TM. I also turn off my airport and plug in my Firewire or USB back-up directly. I have found that with airport activated, TM does not work properly on my system (13" G4 running 10.5). TM and packages do NOT play well together. I am not sure that Scrivener (i.e. Keith) can really do anything about this. I am convinced that this is a TM issue and not a Scrivener issue.

Apollo16

7

Thanks for weighing in, Apollo16. I appreciate it. So–just to clarify for me (and perhaps for others): what you’re saying is that simply by having an external drive hooked up, via a usb port to my computer, running Time Machine, I somehow corrupted Scrivener files? I’m perfectly willing to accept that TM doesn’t back up Scrivener projects–as AmberV explained in her note–but I’m at a loss to see how simply having this program doing its own backups of files on my apple would cause a Scrivener project to lose work (and, as I recall, Amber had said that not all the lost work problems with Scriv. involved TM).
But, again, thanks so much for your help. Even messages attempting to puzzle this all out are helpful in a “we’re all in this together” kind of sense . . . :frowning:

8

I can only talk about my case but it seems to be a TM issue. TM corrupts package-based files on my machine if my machine does ANYTHING but the straight back-up and NOT via airport. I have been in contact with Apple but the errors thrown are different each time. However, the players are the same (i.e. package-based files, airport and TM). I have come up with several wild @$$ theories but have given up worrying about it. I don’t use TM for important back-ups and I never use it via airport. If it is on manual and via a cable with nothing else going on… it works with no corruption. The types of corruption I have seen are missing files, pasting over of one file over all versions of that file on that drive (which according to Apple is not possible… uh huh…), loss or corruption of only some files in a package and renaming of files (which is also NOT supposed to possible). I have a very dogmatic back-up system and thankfully have not really lost anything because I tend to test my back-ups very carefully. I discovered these issues after testing because I read an article warning people not to back up using TM with an airport.

There be gremlins.

Apollo16

9

I seem to recall one individual stating that they do not use Time Machine at all. Frankly I don’t understand how a bug like this could manifest at all without Time Machine. Nothing else on the system is even remotely capable of digging up an ancient copy of something and destroying the current copy (of course, other back-up and retrieval software could do this, but the only backup and retrieval software that every single user has installed is—Time Machine). Time Machine shouldn’t be capable either, but since that sort of action is precisely what it is designed to do (under user guidance), it is certainly the most suspect (especially when you consider that Airport can foul it up—Airport? Apple has some serious cross-over bugs); and anyone that even has a drive plugged in and TM turned on could potentially be impacted by the bug, whether they’ve ever explicitly used it or not. In other words, said outlier individual could very well have not known they were “using” Time Machine.

As Apollo16 points out, the issue has come up in other realms beyond Scrivener, as well. I think we are just noting a Scrivener centric pattern here because many of us are career writers or spend a lot of time with writing as a hobby, so we are going to naturally notice fubar situations with the stuff we are always paying attention to and the stuff we have open many hours per day.

The factors that make it seem most like Scrivener is at fault are the incidentals, or potentially the triggers to deeper Apple bugs. Most often this happens directly after a Scrivener crash, after quitting the application, or updating it to a new version. This certainly seems to point a finger at Scrivener, except that such “archaeology” is just astronomically impossible for an application that doesn’t handle back-ups, versions, and roll-backs at all. Scrivener’s back-up feature is just a glorified “Save as” really.

I suppose the “good news” that can be taken out of all this is that you can trust Scrivener. But because of the bug that exists, wherever it exists, just take appropriate precautions. I generally back-up (using the zip archive function) on the hour, and then save the last copy from every day for about a month. Once you have a back-up location set up and all, making a backup is just Cmd-Shift-S Return. It can be done without even thinking about it.

P.S. It goes without saying, but be sure to check out Powell’s City of Books (you might want to set aside a half-day or more), and if you enjoy great coffee, any of the Stumptown Coffee locations, or Tao of Tea if you prefer the leaf to the bean.

10

Thanks, Amber, and Powell’s is indeed on the top of the list of things to do in Portland.
I’d be more inclined to think it’s something to do with the package-based underpinnings of Scriv, though I defer to the greater knowledge of those on this list of what’s under the hood of Scrivener. But I don’t see how TM, working on an external, removable drive (running via cable, not airport) would write onto my hard drive during the process of performing its backup. I should add, too, that I haven’t run TM for any kind of restore operation in many many months. I certainly see how its backup wouldn’t work, and I’m a big believer in gremlins.
ah well, water under the bridge at this point, though if these “older version” problems, combined with various erroneous Scriv. messages about another copy of the file being open elsewhere, or the current project being created on an earlier version of Scriv–if those problems are coming up with particular frequency in Scriv 1.5, that would bear examining, it seems to me.
thanks again to all for your help. One of the things I’ve enjoyed most about this program is the fine folks on this list . . .

11

Regarding using time machine and what might be happening. Best practices first.

TM is a backup system (duh). As such it will “snapshot” the FS at a point in time. This snap shot is based on disk allocation via inode/block usage and the journal. if you watch TM in action you will see that the first thing is does is “prepare” the backup. I suspect the TM is doing the same things that its brothers do and is FREEZING the journal and copying it to a new location. In theory this is fast and safe (I use very $$ apps that do this all day). There is one condition that makes this “unsafe”. That is when a write intensive (think save) app start building data in the journal before the copy/move is complete. This is compounded by large writes like a big binary (think mov). Basically the journal is frozen, so the write doesn’t really complete. Now your file handle is pointing to a bad location and data is written but the sectors are not marked as associated with the first part of the file. Since the data is still in memory of the process it looks correct UNTIL you stop (quit) the process) and open the file again. Oracle and I spent quite a bit of time working on this with a backup vendor that is NOT apple, but the claim was that the technology was very similar to TM. Hence Oracle recommends full exports instead of “hot” backups†.

Wonderful that we understand how this might happen. How do we prevent it?

We take a note from Oracle best practices and do the following:

  1. DO NOT ENABLE TM OVER WIRELESS.
  2. Run TM on a direct attached drive.
  3. DO NOT ATTACH DRIVE WHILE YOU ARE WORKING ON PROJECTS.
  4. Quit scriv (and all other package based apps) before attaching your TM drive.
  5. Attach TM drive.
  6. Once TM completes reopen project and get back to work.

If you think about your breaks you should be able to do a backup every 2 hours unless your files are huge. My system will complete a 15GB TM backup in less than 15 minutes (over firewire 800). Using this practice, which was very similar to my manual rsync backups, I have never lost any WIP. And yes I have verified my data. Numerous time. By restoring files that I screwed up.

Hope this helps.

† [size=75]Or if you are me Oracle says “Just build a RAC then you don’t need to worry about backups”. Until the RAC blows up and replicates the data corruption and you get to spend several day applying RMAN backups!!! AAARRRRRRRGGGGGHHHHHH!!![/size]

12

Thanks, Jaysen–this is very helpful indeed.

13

Once a week I take “Step 4” from above a little further and completely log out of my account. Then I log back in with the Shift key held down. This suppresses all start-up user level drivers and applications (log-in should be nearly instantaneous). Then run TM manually and unplug when done. I make a manual note of when this is done in a text file so I have a record of known “complete” back-ups. It’s probably over-kill, but I only do it once a week, and follow the above steps several times a day as well. It’s a pity because the notion of hourly fully-automatic backups is very nice—but in practice I’ve just run into too many situations where TM has backed up partial or “corrupted” data because the resources were in use during the backup operation. Jaysen’s theory about journal freezing and addresses getting all fouled up is the most credible theory I’ve heard for what is causing (at least half of) this; as I do think that this bug has often been coupled with the complaint that Time Machine is not capturing modern data effectively. One other step you may or may not want to do: Unplug the external drive from the wall as well when it is not in use, especially if you live in an area that has frequent electrical storms.

All that aside, you are right, TM should absolutely not be writing data to the host disk, save for two circumstances: User engages restore function by clicking appropriate button in the TM interface, or user initiates a full TM restoration using a Leopard install disk. That’s it. But that said, I cannot myself think of anything else that could possibly dig up old copies of data like that; nothing that is common to all of the machines that have replicated the bug, anyway.

When Scrivener throws the error it does, it is likely just reacting to the fouled up disk state that the deeper bug has caused. I doubt the error itself has anything to do with the formation of the bug. It’s just a sanity check that fails, and rightly so, when the messed up project is opened.

Time Machine is really nice, it just isn’t quite as “fully automatic” as Apple likes to say under certain circumstances. I once had a laptop drive fatally crash; hardware level. After putting in a new drive myself, I was up and running in less than an hour with the entire system the way I had left it. Best restoration experience I’ve ever had. So even though it has its problems, I still use it, just not the way Apple markets it.

Oh, and I believe I recall Keith mentioning he has all of the reports pertaining to this issue in a file, though as of yet there as been no way to reproduce it or pattern within the file. So, it is being investigated, as much as it can, considering that it is highly unlikely that Scrivener itself is doing anything wrong.

14

I don’t think TM is writing to disk. i think this is an issue similar to truncated Oracle writes (which is why I know this). Basically as long as the OS is able to rebuild the file from mem it will appear to be correct. This can be OS level cache, HW cache, or RAM cache in the software. While I haven’t checked all the threads, I bet that typical usage for each user was to keep scriv or the mac open and just sleep between session. At some point a full shutdown of either scriv or power cycle of mac occurs. The user now sees data loss.

Just to reiterate, I don’t think TM is writing to disk as much as the data is never really saved in the first place. Or worse, it IS saved, but the FS loses the pointer to the next block and returns EOF.

Maybe KB can comment on how scriv manages file handles/mem between sessions. Ex. If I click red x, does scriv deallocate the file handle which would purge cache (would it hit the OS?) or is it possible that the FH is retained and recycled if the file is reopened.

Another thought might be the 2 sec write interval is really screwing up TM on journal copy.

Just some thoughts.

15

And for the record, I am, in fact, a complete nerd. Always have been.

Snort knew this when she said “I do”.

16

That would make a lot more sense to me. I do believe there was one user with this problem who claimed to shut down the computer every night, but again some people say “shut down”, but could very well just mean closing the lid or putting the computer to sleep. Given that most of these cases are within the half-week to two week range (that sixth month case was no doubt exceptional for a number of reasons), it becomes much more plausible that stuff just isn’t getting written when/where it should and nobody knows until Scrivener closes/crashes/automatic updates/et cetera.

17

Are you using any other backup or synchronisation software as well as Time Machine? (e.g. Forklift?)
Thanks,
Keith

18

Well…it does help when we are aware of our own faults and can admit to them. Hope for yyet, youngun. :wink:
Vic

19

Hi all–I almost-promise that this will be my last post on this subject, since I think we’ve about gotten all that there is to learn from the events, but–in case Keith’s keeping a folder for further investigation–I want to try and articulate the sequence, with what Jaysen’s taught me now worked into the sequence of things, and with a general question about Scriv. and backup programs.

If I understand the gist of Jaysen’s post, a large, packet-based project like a Scriv. project, if it was “frozen” by TM in order to do a quick write to the external disk, would be much more susceptible to the problems generated by TM (and other backup programs,I gather) than would, say, a simple collection of smaller, single files on another program, since each of those files would be less likely to be suspended in a complex and discrepant set of versions of material in ram, on disk, etc. etc. Right? In this sense, Scrivener is a program that would not play well with any number of backup-in-the-background programs, right?

And, Keith–about other backup programs. I’d made copies of various projects to University storage space using Cyberduck. I’d put some copies of projects on the external drive (not using TM but simply using “copy.”) I am actually, believe it or not, a pretty careful and pretty experienced computer user (never lost this much data before; been doing computer-based writing since the mid-1980s).

here’s the sequence

  1. I was happily working along on a Scrivner project.

  2. I decided to change the background color for the Full Screen view. :blush:

  3. I did that, using the “preferences” pane.

  4. There was no visible effect when I went back to Full Screen, so I thought: I’ll quit Scrivener and restart it, to see if the change takes effect.

AND at this point, as I was about to quit, or was quitting or had quit Scriv (conjugations of errors), TM rears its ugly head and does its dirty work–creating the confusion for Scriv as it will start up again and not know where all its body parts are . . .

  1. I quit Scrivener–without, alas, closing the project first; though I’d done this before and had no problems opening up the project again. Perhaps that was only because, in the Russian Roulette that I now realize is TM, the timing of quitting and backing-up and restarting had never been so unfortunately mis-timed.

  2. I reopened Scriv and was confronted with those error / warning messages, both about the project being open on another computer, and about this project being created with an earlier version of Scriv.

  3. I said “yes” to the synchronize option, and “yes” to the update to current version of Scriv. option.

  4. And at that point, what I had, was a several weeks’ earlier version of the project.

  5. Then, of course, TM didn’t come to any rescue at all, for reasons Amber and Jaysen have both spelled out in useful detail . . .

thanks again, for all your ideas and ponderings . . .

20

Background, automated backup software of any kind will have issues with complex packages like scriveners, yes. Files like .doc and .pages are single files (or zipped packages) that reduce the issues with memory and disk writes.

I am still puzzled by the “toss back in time” phenomena. TM uses a flaky method of symlinks under the hood, but I am not clear on how it is going back to a time that precedes TM current run. I will need to beat my head on that for a while to come up with something provable.

Just a thought, do you use snap shots?