Data loss during transition

As it seems, there’s a substantial bug that can lead to data loss when converting projects from Scrivener 1 to Scrivener 3 (I made the transition last month, so this applies to September’s beta).

On all the pages that had page content AND the synopsis filled out, only the synopsis survived, while the actual page content was discarded.

Please let me know if you can confirm this (should be easy to test). If not, then my setup is more specific and I need to do some more testing.

Did you retain the original backup copy that would have been created when updating the project format? This backup will be an untouched duplicate in v1 format. Thus any reproducible upgrade bug should be easy to discover from it, if you’re willing to send us a copy. Otherwise any old backup you can dig up that demonstrates it should do.

I have all backups, so the bug can be reproduced and technically I shouldn’t have suffered any actuall loss (the missing pages are in the backup, but I have to identify them).

Where should I send it?

Unfortunately, things seem to be more complicated. Here’s what I have so far:

In September, I converted two projects: One has issues, it appears that the other one hasn’t.

All pages that were lost had at least one image, and all had a synopsis.

When I reload the project, the missing pages show a page icon in the binder, suggesting there’s a page and actual content. That icon turns into an index card icon once I click on it.

There are also pages with images and synopsis that were not lost, so my original hypothises (that all pages with a synopsis were discarded) is wrong.

The most consistent theory I have is that it has to do with size: When I sort the rtf docs by size, the top 7 (down to a size of 311kb) didn’t make it. Smaller ones (204kb and less don’t seem to have issues).

In the other project, there was also one fairly large page (600kb) with images, but without synopsis. It was converted just fine.

I tried to reproduce the problem with a copy of the old projects and the newest beta, but could NOT - everything came out fine.

I’m a bit clueless, to be honest. While I can’t see this problem on the bug list, is it possible it was “accidentially” fixed with the latest update? To make sure, maybe you can send me a link to September’s Beta so I can test?

I can only confirm that there were no project conversion changes or fixes during the last two months.

Anyone?

I am afraid, September’s builds have already expired and cannot be used.

Sorry, I’ve been busy helping to get an urgent Mac compatibility fix out.

You can send test data to beta.testing AT literatureandlatte DOT com!

As long as I can’t reproduce the problem under the circumstances it occurred, I don’t think it makes any sense to send it for testing. But I’d appreciate if you could try to think this through one more time. Disappearing files is a threat to everyone, I’m a very experienced computer guy and I can rule out that I deleted the data myself. There is a reason for everything, and if the reason is unknown then because finding it hasn’t been tried hard enough.

I was hoping that something from the very detailed list of circumstances I described rings a bell with you guys. What sticks out to me is that the largest files didn’t make it. Unfortunately I cannot speculate any further because I have no idea about how the conversion process from Scrivener 1 to Scrivener 3 works. That is something only you guys can do.

The only other thing that comes up is that, during conversion I was told that there’s not enough disk space, so I had to free some. Again, that shouldn’t cause a problem, but since we’re looking for a bug, maybe that’s what made the pages disappear.

@threeblind, given you say you have all backups through to Scrivener 1 versions, and given what Tiho confirms about no changes to the conversion code, I would think the thing to do is to try the problem conversions again, this time with adequate space to do them in in.

And not on USB sticks, if I remember that’s something yo use bringing work between computers. Very unreliable, not to say slow, but not trying to get the discussion started about your use in syncing work, if I seem to remember it. That’s of course anothe r way content might have gone missing, though, one would suppose…

You wrote upon conversion: " I was told that there’s not enough disk space, so I had to free some. Again, that shouldn’t cause a problem,…"

This is definitely something to cause a problem, as the conversion has not completed and most likely left your project half converted, i.e. this is a great explanation why some files were missing at the end. This is another BIG reason against using memory sticks for every day work. Using a full memory stick is even slower, but I also do not want to open a discussion here about your preferred work habits. It has been already widely discussed.

Guys, I think it’s enough now. This is a discussion between me and the makers of Scrivener. There is no need for anyone to waltz in here with a quasi-religious agenda and tell me how to I’m supposed to live my life or write my books. This is ridiculous.

The official view on this is that removable media are not recommended, as they can fail more often. Fair enough. I’ve tried to explain in a different thread why it’s hard for me to adapt to that - not because I’m a stubborn geek, but because of the very real circumstances of my life and work. The Scrivener team was so nice as to make suggestions what I could do to to make their recommendations compatible with my irrevocable writing needs. I will look into that, and in the end I will make an informed decision as a grown-up. End of story.

This report here is a totally different matter. We are NOT talking about disk failure, and assuming that files simply go missing all the time just because they are written to USB sticks or SD cards is ridiculous to a point that the claim is either plain stupid (read: total lack of basic technical understanding) or malicious (read: you know better).

Making false assumptions just to place the blame on someone who is (1) seeking help and (2) trying to keep others out of harms way isn’t any better. As you don’t seem to know (but the Scrivener programmers do), the message I referred to didn’t come from the operating system but was relayed by Scrivener itself: The program has a routine to deal with such a problem, by telling the user there’s not enough space, and supposedly deleting the conversion folder and returning back to the original state. Since this is all straight-forward, it is rather unlikely that it could lead to data loss - but IF it does, then it would be a serious bug in Scrivener definitely in need of fixing.

Not that you seem to care, but there are quite a number of reasons to run out of disk space - converting a huge project (anything taking up more than 50% of the available space would qualify for this error), writing to a small partition, or using a file container (which is what I did). Nobody ever said I used a full USB stick - not in this thread, and not anywhere else. And your derogatory remarks about my “preferred work habits” which you are kindly not willing to discuss are totally out of line. What is even the purpose of your post - trying to be helpful? Seriously?!

If all you have is a hammer, everything looks like a nail. Please kindly go and look for nails elsewhere. Thank you.

Just in case you’re unaware, tiho_d is the developer of Scrivener for Windows. I read The second half of his post as referring to the discussion in the other thread as a point of fact, not as a derogatory reference.

It’s not clear who you are responding to, but AmberV is the L&L head of support, and tiho_d is one of the Windows Scrivener developers. Their descriptions of Scrivener’s behavior should be seen as definitive, whether you think it should behave that way or not.

Katherine

How should I know? There’s no signature or other pointer to convey that information.

I do remember my posts in the other thread. I can live with the fact that I wasn’t able to get across the necessities of my daily work, but then you will also have to live with the fact that anyone concluding that working with a full USB stick is part of my “preferred” work habits comes across as polemic, derogatory or however you would like to dress it.

I will be concentrating on this thread now:

And this is why I said what I did.

I apologize for mentioning the memory-stick-as-method issue – I actually did understand your statement of situation, and sure, certainly you have to do what works in your situation. I’ve done so in many of own situations, fully understood.

I think what Tiho mentions also could help with what you need to do: be sure to not fill the sticks up. I.e., maybe put a limit at 50% full for safety, and follow it.

There are a lot of things that happen or can, when you get near capacity, tied up with the internal aids all memory sticks operate with, to minimize both their speed and cell-going-badissues (which just coime with the technology, and thus also the minimizers are engineered for them).

Leaving room lets these operate. In theory, maybe 20% is enough free space, but again, given your constant use and apparent issues, I’d leave more, the 50% mentioned above. And reading the percentage, off what listing the contents at the time tells you about used and free space (Windows ‘properties’ of the stick).

Ok, I had some other thoughts, and ended up spending too much time researching them. There are innumerable bad alternatives for things like sd-card to hard drive readers, etc. as photo persons try to use, unless possibly at large price.

But then, an easier idea. Why not a portable SSD hard drive, for its reliability, ease, etc.? Well, prebuilt solutions look to have the same bad-box syndrome. But you could make one very easily. Here are the two components, models I think I can recommend:

  • amazon.com/gp/product/B00OJ3UJ2S – Sabrent enclosure; I have one, and it seems fine.
  • amazon.com/Samsung-Inch-SAT … B078DPCY3T – I have the previous model (850 Evo), and it is simply a great drive, as every professional review points out. I use it heavily every day, as the internal on my laptop. I think I’d stick to 1GB, not lower than 500MB, anyway, as reliability and performance goes up with size.

If you buy the Evo (not QVO) drive from Amazon, be very careful to order only where it’s Samsung and Amazon providing the drive and shipping it. There are many scams. I went to the local Best Buy, where they price match, and it’s in their interest to have the real thing, if you live in a country that has those or another trustworthy like it.

Anyway, I wish this was easier, but it might be worth the trouble, which is actually not much at all (the drive just pops into the case) and the resulting portable drive is small and light. The size and reliability should get you away from cards, and you could still use them in a pinch, then copy the hoped-for results onto the SSD from your computer when you get to home base.

You’ll need the USB cable to connect the drive to your computer/s, but as I remember and as the brochure page says, the Sabrent enclosure comes with that.

I wasn’t trying to be disrespectful, just helpful, as with our interaction on other threads. Sorry it didn’t work out that way, but good luck figuring out the issue that started this thread!

Blaming sloppy software for not being able to handle your full memory stick is definitely the way to go. At least we know what caused your data loss, and the sloppy software DID WARN you about it, but you have ignored it. The reason that you do not follow what everyone suggests as a good practice, has nothing to do with your experience.

When your SD card is full, Scrivener will not store your last changes and WILL WARN YOU about that. It would be great to magically add more space to your full SD card, but it can’t.

With all my respect, please do regular backups of your SD cards to avoid future problems. I mean it with only good feelings.

Unfortunately, this topic lacks the necessary differentiation, which makes it difficult to keep the discussion at a useful level (and which is why I have tried to at least reduce it all to one thread). Basically I agree to everything your have written, except that this thread isn’t about everyday use, but about the conversion script. During the upgrade to Scrivener 3, the script will create a full backup of the original project in the same folder (there is no option to specify a different drive or folder). Which means your data will be instantly doubled. Take Katherine’s example from the other thread - the 13 GB project. Imagine it residing on a 25GB SSD. Doesn’t seem like it’s anything worry about, but when you upgrade, you will hit the same error I have, and if you somehow lose data, you will recieve the blame on top of the injury (=it’s all your fault).

I appreciate your effort. The Evo is great, in fact it’s what I use as an internal drive in all of my laptops.

Maybe one day I should post pictures of me typing away on my Eepc 901 in the most unlikely of places, in clubs and bars, out in the sticks, on planes and public transport (or in the bathtub for the matter). Probably there would be no questions after that. :mrgreen: I imagine the computer on my lap, and the external drive would some dangle from my leg, only held by the cable. Then the cable would eventually come out, and then I’d had to justify myself what I was thinking to unplug my writing media in the middle of work (the dangers!). Nope, that just wouldn’t work. :laughing: