Help Please! Missing Work with Upgrade

I am desperately hoping that someone can help. In the midst of working about a week ago I accepted the invitation to upgrade to Scrivener 1.5. (I have been a committed Scrivener user since the earliest pre-release Betas and have had such a perfect experience with the program that I stupidly did not even hesitate, didn’t do an extra backup etc.) The upgrade seemed to work fine. But I have just returned to files within Scrivener that I was working on about a week ago and they are blank. The titles are listed in the binder but the body of the document is blank.

This is a disaster for me. I am a historian and this is a 20 page talk that I have to give in a week that has completely disappeared.

There are a handful of other documents from the same time period that are also missing.
When I open Scrivener, this Binder icons for these documents contain “text” indicating that there is content to the file. But when I click on the binder the icon goes blank indicating that there is no text (which is true as the document is empty).

I tried to recover these files by looking through the package contents and I discovered that none of the files that I created between 3/11 and 3/27 were saved or exist in the package contents. Everything before and after these dates exist. Another odd detail: when i look at the “General” section of the Info. Panel on my most important document, which is now blank: it says: Created 3/17 and Modified 3/22, so clearly the program still “knows” about this document that I worked on during precisely these days. But there is no text, it’s completely blank and no saved version in the package.

Do I have any hope of recovering these lost documents? Does anyone know what might have happened or how this can be avoided in the past? I was an early adopter of Scrivener and have used it exclusively every day for years now. I have written talks, articles, and now a nearly complete book manuscript. I am mystified and unnerved by these sudden blank documents. Kicking myself for not backing up the whole file during that one week, but there it is.

Any help would be very greatly appreciated.

One more point I forgot to add:
When I do a Spotlight search on specific phrases from the missing document the scrivener file does indeed pop up in the search results. But when I click on it I am returned to the correct title listed int the binder and the (chillingly) empty screen/document.

Sorry for the triple post, but here is a last detail:
If I open up the project in outline view, the document in question is first listed as having 4,105 words. When I click on the title in the binder the document, as I mentioned is empty and if I click back on the folder for outline view it then says 0 words.

Could you give as much information as possible, no matter how irrelevant it may seem, about other programs running, what backup system you have in place, any synchronisation software you use, where the files are saved (local drive or elsewhere), anything. Something similar seems to have happened to a handful of users which is indeed alarming but I still haven’t got to the bottom of what is happening. There is nothing in Scrivener itself that could wipe this work (as it seems that files have gone missing from within file packages) and nothing in the update process either. But it seems that there is some external factor that, in conjunction with the package format and the update, all combines to do something really, really bad. Finding out what that external factor is has eluded me so far, so anything you can think of would be very helpful.

I’m not sure if it’s possible to recover what has been lost, as it does seem that some underlying files from within the package itself has gone missing. To check this, ctrl-click on the .scriv file in the Finder and select “Show Package Contents”. Then browse through the RTFD files to see what is there and what is missing. Also check your Trash folder.

From the descriptions I’ve had so far, it is almost as if some Time Machine-like software is reverting Scrivener files to some arbitrary earlier date…

Many thanks,
Keith

Keith,

Reading this thread made me wonder. I am one of those, I hope thousands, who have had no trouble with Scrivener upgrades.

First, let me say, I prefer to use the Sparkle system only for very minor upgrades; for an upgrade like 1.2b to 1.5, I will always do an old-fashioned download and re-install. In fact, in general, I prefer to download and re-install, not because of fear of the Sparkle system, but so that I have a copy of the latest installer so I can use that on both the MBP and the MBA where appropriate, without having to go through the download on each machine.

Secondly, a big difference between a Sparkle upgrade and an old-fashioned Download-and-Re-install upgrade, is that generally Scrivener will have a project open when the user is invited to click the “Download and Install” button. For the old-fashioned route, you have to have the app closed in order to do the re-install.

Thirdly, to me, especially before doing any serious upgrade, and I think this should be particularly important if about to use the Sparkle route, I do a back-up to a different location of the current state of affairs before the upgrade process … just as you should do a complete system back-up to another disk before a system upgrade. For me, this isn’t arduous as my whole system is backed up every night, and typically upgrades are announced on first opening whatever app it is.

So my thought was this: presumably, those who have lost work, or apparently lost work — some seem to have been able to retrieve all or part later — have upgraded through the Sparkle system with their WIP project open in Scrivener, which gives me two thoughts:

  1. for you, that you should look if there is anything at all in the Sparkle code itself that might change creation dates or any of the files internal to Scrivener … but you’re probably doing that anyway;

  2. put a message, if you can, on the update alert to tell users to do a “back up to” and/or close all open projects before clicking the “Download and Install” button. If there isn’t a project open, the chances of one being damaged are much reduced. In that way, the chances of this happening again will be further reduced.

Just some random thoughts.

Mark

Is it possible that the undo buffer was applied to the project as a whole? If a user were to only sleep the mac and never exit scrivener it MIGHT be possible that the undo buffer would go back several days.

KB, your thoughts?

Thank you for your quick and concerned reply Keith. I appreciate the other posts as well. Unfortunately, I don’t have too much detailed information because I was just working not paying much attention to the program (largely because I never had to because I’ve never had problems). From what I can piece together, I was asked to upgrade at some point, did it, continued to work without a problem, but after shutting down and rebooting my computer a few days later, and going back to several files that I had been working on during or around the upgrade, discovered that there was no content/text. As I mentioned, the file titles were listed in the binder with “text” in the icon indicating that there was content, the outline listed the proper word count for each, and the info. panel had the correct created and modified date, but all the text is in fact missing, the content of the document missing, and there is no trace of the file in the package contents. I was not using any external backup system (like Time Machine) and no other unusual programs were running, same typical stuff I have used for years, and nothing that would interact with Scrivener. Only one week of files has been affected this way, everything before and after that is fine. I love the idea of using the Undo to fix this – I have a real feeling, for some reason, that maybe it could have been that simple, but because I shut down the program and the computer that’s not an option. Most strange is that Spotlight searches still point me to this empty/ghost document. Of course I should have backed up the whole Scrivener project/file to another location AND exported the important file i was working on into a separate rtf file as I almost always do when working on something important. But I didn’t and I lost it.

Did you say that spotlight finds text that is in the file, but it isn’t there when you open the file?

Check to see if the text has changed color (white text on white background?)
Did you check to see is a “snap shot” exists that might contain your text?

Just a thought or 2.

The description is consistent with the underlying files (inside the .scriv file) being lost. The search strings file (binderStrings.xml) stores all the text in plain text for the Spotlight and internal search, so that every time you run a search Scrivener doesn’t have to open all of the constituent RTFD files. This search strings file is also used by the binder to determine whether a file has text or not and thus whether or not it should be drawn as a blank icon or an icon with text - that is, until you click on the file to load it up. At that point, the underlying text is used to determine whether there is text or not.

So, if there was an underlying RTFD file with text that got lost, then Spotlight would still find the project when you searched on something in that text (as would Scrivener’s internal search), and the icon in the binder would show that it had text until you clicked on it.

Clearly, some underlying files have been obliterated. The question is, how? The only point at which Scrivener itself ever deletes files from the disk is when you have used Empty Trash (did you use Empty Trash at any point? Could it be as simple as a bug in the Empty Trash code?). Other than that, ti doesn’t call on the file system to delete anything.

Also, I can’t see how Sparkle could be the culprit simply because the only thing it overwrites is the Scrivener application itself. Moreover, from gmw’s description, it seems that the data loss didn’t happen until a few days later.

In fact, it seems that the work was lost after shutting down and rebooting, if I understand correctly.

(Incidentally, the undo stack would have had nothing to do with this, as it doesn’t work that way and can’t remember files, only actions, and those only for a brief span.)

gmw, could you please do me a favour? Could you please send me the full system profile of your Mac? This way at least I can see all of the software you have on your Mac and so on. While it may not lead me to an answer straight away, if I ask the same of others who have experienced the same situation, I may slowly be able to piece together the larger story or external factors.

To send me your Mac’s profile, go to the Apple icon in the top-right of the menu bar and select “About This Mac”. In the panel that appears, click on “More Info…”. Then, when System Profiler opens, go to File > Save and save your Mac’s profile (it saves as XML by default - that is fine). Then send me the resulting file (to support AT literatureandlatte DOT com).

Thanks!
Keith

The shutdown and restart trigger is one similarity between another case where the files all reverted to a state from some time last year. I believe there was one other issue where content disappeared like this, save but for the first line of text. Again, I think a shutdown and restart trigger was present.

It’s almost as if Scrivener is triggering some file system bug where writes are not getting properly committed to the disk (for some period of time), or getting written in a scratch area and not properly linked to the file data, potentially not triggering the journalling log. I’ve never heard of such a problem with a high-level application, though.

Then for some bizarre reason 1.5 loses track of this misplaced data resulting in empty, malformed, or historic projects, as if the prior version (1.11 or 1.12, by the way?) was storing the internal project data in different innodes (I know, HFS+ only emulates those).

None of this makes any sense to me, less sense than Scrivener mysteriously triggering Time Machine or blowing through eight months of undo stacks. None of that can or should be possible.

It sounds like my files were indeed obliterated. I have been frantically trying to rewrite what I lost. Thank you to those who tried to think through this mystery. Nothing like this has happened in years of using this program. I will of course be more vigilant about backing up, and exporting, Scrivener work daily. I will also send Keith my system profile - maybe I’ll be lucky and he can figure out what happened.

I had the same problem happened to my work in scrivener, about a week ego.
I tried to restore the file from my .mac backup, but even when i got the file that created before the date when this problem happened, when I tried to open the project, it was empty.

So i deleted the program from my app folder, downloaded it again from the web site, and then tried to open the project again.

believe it or not, most of my text files was there. But not in the place they should be, they was in a NEW folder called:
_Recovered Files.

It took me some time to organize but its not a big deal.
The problem happened for 2 different projects at the same day.

I’m using scrivener for just about a week end i use the 30 day try version.

Maybe it’s a good idea to add automatic backup to the program.

Meni Philip

I was just telling my husband about this issue, in case he had any ideas, and I thought I’d mention the result in case it prompted further thoughts in the minds of people who know what they’re talking about! He is a Windows techie (but a Mac user at home). When I described the missing file content problem, he immediately said that it rang bells, and that he remembered reading something on MacInTouch about something to do with zero-length files being saved by the system.

He looked it up, and there is a problem description on the following page (scroll down to 7 Dec 07, or search on “zero” to find the specific entry):
http://www.macintouch.com/readerreports/leopard/topic4694.html
Basically, the writer reports that if a file is opened and then saved without making any changes, it is saved as a zero-length file.

This page links to a thread on a Cocoa developer website, dating from Nov 07, which discusses a bug which affects (or which once affected) document-based Core Data applications:
http://www.cocoabuilder.com/archive/message/cocoa/2007/11/2/192065

One post summarises the problem as: “This bug affects subclasses of NSPersistentDocument using a store other than SQLite when saving “no changes” on 10.5.” And another suggests a workround:
http://www.cocoabuilder.com/archive/message/cocoa/2007/11/2/192085

Obviously, Nov/Dec 2007 is quite a while ago in software terms, so this problem may well have been fixed. But the Core Data bug does seem to produce a similar effect to whatever is causing the “empty file” issue, so I thought I’d mention it. I apologise profusely if this is a total red herring.

Just got back from Norway and am desperately trying to catch up, so apologies for short answers…

gmw - thanks for your e-mail with your system profile, will take a look at it later.

Meni - the _Recovered Files folder gets created if Scrivener finds text files inside the .scriv package taht have no representation in the binder. It sounds as though a synchronisation (or just the restore?) may have restored the binder to an earlier state (without the new text files in it) but the underlying text files were still there, so Scrivener found them and placed them in the _Recovered Files folder. Do you use any sync software? Or was this just following a restore?

Siren - I doubt that applies to what is happening in Scrivener, simply because Scrivener doesn’t use Core Data or NSPersistentDocument.

All the best,
Keith

I wonder if there isn’t something going wrong with Time Machine:
forums.circusponies.com/categories/3/posts/11479

Hi
I’ve the same problem occasionaly. The file name appears in the binder but the content is empty. Up to now it has always been pdf ‘research’ files that I can simply re-load into Scrivener. I use synch software (Synchronize Pro X) almost daily and I suspect that that’s the culprit. I’m diligent with shutting down Scrivener before synchronising. Last evening I repaired permissions and the problem reappeared with at least one file, so I dunno is that part of the problem.

Donncha

I use Synchronize! Pro X, to back up every night automatically, and I’ve never had a problem with Scrivener as a result. I wouldn’t have thought it lay there. I’ve found with Sync Pro X, that if there is any problem with a file or folder, the back up won’t run, or asks me to confirm that I want to skip that file.

Mark

Edit: By the way, no file that has caused a non-running back-up has ever been a Scrivener file of any kind, whether project or one which is part of the app.

I use Synchronize Pro to synchronize my home and office computers. I’ve previously found an issue that seems to relate to Synchronize Pro in that the ‘created date’ of the scrivener folder becomes 1 January 1980 and apparently other files also get similarly corrupted.

Donncha