Yes, maybe it is best to divide the backups in two different tasks. Thanks again.
The “known issues with cloud services” are at times a bit over-baked, and subject to some really at times almost comedic levels of game of telephone—such that we have people believing only Dropbox works reliably, when the fact of the matter is almost everything works fine, and Dropbox is one of the more problematic ones on modern Macs (and doesn’t even use its own sync engine any more)! I digress though, it has a lot less to do with the physical process (which ChronoSync has similarities to, but lesser one if you are using primarily as a one-way backup) and a lot more to do with people not being careful about the fact that they are essentially using “Merge” mode by editing the project in two different places and leaving sync to create a mutated middle copy that Scrivener has to recover from somehow.
That, and more recently, dumb default settings (Dropbox for example) that deletes your local data and only downloads parts of the project, but that’s another matter that again doesn’t pertain to ChronoSync in a one-way use model.
At any rate, by backing up your whole folder and using exclusion mainly only to avoid truly unnecessary data you do not want restored anyway, you at least have both. If for some reason the backup doesn’t record the project properly, you’ve still got the .zips to fall back on, which is how it should be rather than those being your only long-term saved copies.
1 Like
Thank you so much. I appreciate the time you invested in this.
2 Likes
Footnote: Seems to me you need not even worry about how the live project backs up. So long as you have Scrivener’s zipped back ups getting backed up to the external then these (can) serve as your real bedrock backups.
4 Likes
I wouldn’t put it that way. Again, a Scrivener project is a bunch of files in a folder that gets auto-saved down the second as you work on them. There seems to be an awful lot of talking about this that skirts over that very basic truth. Maybe it is best to put it very simply:
- You have a folder full of word processing files.
- You have a thing that sometimes zips that folder up somewhere else.
- You edit the files regularly, and what you use to edit them saves them constantly.
- You back up both the original folder and the zips.
Which is more up to date? The thing that might have zipped them up at some point, or the actual files that are used to make the zip file? Under what circumstances is is correct to say that one should not worry about the actual files one edits and to instead just make sure the zip files that are sometimes made from it are better?
We really aren’t talking about anything much more complicated than that. Again, there is a lot of mystification and misunderstanding of the “complexity” of the Scrivener file format that is downright false. When you look at what it is on the disk itself, which is what ChronoSync or any other backup system cares about: it is a folder full of word processing files (and yes, some .txt files and other similarly plain-text files). Clean and simple. If your backup system cannot handle a folder full of RTF files then you need another backup system because that’s not the problem. Excluding a folder full of RTF and TXT files is not the solution.
The supposition that it is risky to back up a folder full of text files is baffling to me. The notion that one should trust a periodic zip copy of those that may or may not be as up to date as the originals is also baffling to me.
3 Likes
Maybe stemming from the fact that sync services have trouble keeping all of it in sync — but an OS doesn’t have that problem (unless the disk is failing), and neither do backup programs like ChronoSync. But I have seen it mentioned on Literature & Latte pages that a zip file is safer than an uncompressed folder, when it’s in transit from place to place.
1 Like
I wouldn’t agree with that statement in a general way. With very few exceptions (Google Drive for instance) most sync services have no trouble at all keeping a folder full of RTF files in sync. That is a misnomer. The very vast majority of people that report problems are reporting issues with their own working practices, as stated, not with the technology itself.
That aside, I do agree with what what you are driving at (I think) that whatever factors are involved, 'net based syncing isn’t terribly relevant to hardware level copying. And that’s all we’re talking about here, so I don’t know why cloud syncing keeps getting brought up.
But I have seen it mentioned on Literature & Latte pages that a zip file is safer than an uncompressed folder, when it’s in transit from place to place.
To wit, this is again not relevant to copying data between disks, and furthermore it is a rule of thumb not an objective statement. In theory it is safer to copy one file than 1,000, but 99.9999999% of the time all 1,000 files will copy correctly. We are talking about statistics at this point.
1 Like
My idea was simply that the backedup zip backups always have your back – even if not up to the second. So, one didn’t need to specially fret about the backup of the live project. If something was off kilter with that for any reason (because, say, you were working on the project during the time its disparate parts were being third-party backedup), the latest zipped backup would still hold you in good stead. Undue fretting about the matter thus seemed unwarranted. No special steps needed. For example, it would be pointless (just as you said) to go to steps to make sure the live project was /not/ backed up.
3 Likes
Thank you for explaining the structure of Scrivener projects. Can the ‘Save and Rebuild Search Indexes’ resolve any issues if something goes wrong? I’m also wondering if there’s a description of how Scrivener projects are structured internally? I am curious if it would be possible to update Scrivener in the future to read and update compressed zip files directly or link the projects with Dropbox like the iOS version. Mac users can create and update files on OneDrive directly through Microsoft 365, without having to install OneDrive or store files on their hard drive.
Perhaps over-working this? Just make sure you are backing up those standard Scrivener Backup Zips (all of them), and start, if not already, your routine TimeMachine backups.
1 Like
Yes, maybe this discussion is already going beyond the practical level. And yes, I also have Time Machine backups that run every hour. I’ve just had issues with Time Machine in the past, so I don’t trust it as a long-term backup solution. My Time Machine backups got corrupted twice after a macOS update. The stored old Snapshots were inaccessible via Time Machine. The only way to find the backups was through the Finder. I had to restart Time Machine backups from scratch the last time this happened after the Ventura 13.5 update. Time Machine can be a valuable tool for restoring past file versions, but it’s not trustworthy for archiving backups.
It isn’t the backing up process that is risky but the restoration of material that is.
1 Like
With how I recall ChronoSync working, if something goes wrong it will tell you of its failure, so that you can go back and try restoring again. If a bulk file copy procedure fails, the result usually isn’t something software can fix since not all of the data copied, and isn’t worth keeping.
All that command does is go through every text file in the project that stores your data and copies it into another file. So it is mainly of use if the search index file no longer represents the data for some reason. There are failsafes in the project itself to help detect if that happens, and automatically rebuild it.
I’m also wondering if there’s a description of how Scrivener projects are structured internally?
I don’t believe there is a comprehensive explanation online in one place, but we always make the full specification available to anyone that asks. What sort of information are you looking for? I should add that most all of it is designed to be human-friendly. If you are curious you may find that internally it describes itself, when you open some of its core description files in a text editor (like the .scrivx). Using that file you would find the key to knowing which numbered folder to go into, in Files/Data, and within that folder you will find a content.rtf file that can be opened in any word processor, that contains the text for one binder item.
You should see, as you poke around (I’d advise doing this in something disposable like a copy of the interactive tutorial), that there is nothing complicated in here. The two files you won’t be able to obviously open in the Files folder are .scrivx file backups, which are themselves just zip files with different extensions. The other file you might not be able to read is a layout settings file, ui.plist. Everything else, other than any binary data you import, is something you can load in a word processor or plain-text coding editor. And that’s all it is, that’s why any good tool can copy this safely.
There is technologically and fundamentally no difference between backing up a Scrivener project and backing up your Documents folder. If anything the latter is more complicated and “risky”, yet here all of us are, no doubt backing that folder up at least daily without a second thought. So then why all of this festering over copying a much simpler folder? Nobody is talking about zipping up your Documents folder only backing that up. Why?
Again, most of it goes back to misinformation, rumours and misunderstandings. A lot of it comes from Mac users too, because there is this great air of mystery about what a “package” is, but for the purposes of copying data, it’s just a folder with files in it, full stop. Anyone that has copied their project to a Windows computer will see that a package is nothing but a giant ruse put on by the Mac front end. Far fewer Windows users question whether it is “safe” to copy a project because it’s obviously safe when you’re looking at it and it looks just like your Documents folder.
I am curious if it would be possible to update Scrivener… link the projects with Dropbox like the iOS version.
That would be antithetical to the entire principle of what these services provide. These systems were invented specifically to make that concept obsolete. iOS is a special case because it is a deliberately crippled operating system, where third-party tools cannot read or modify the file system as a whole, which is how cloud sync works.
The example with MS Office is different, as Microsoft has a vested interest in pushing you into using their service. It makes them money. I.e. that’s a sales tactic, not something that is actually good for you, or an example of a good technological implementation.
Besides, if you think about the logistics of what you are suggesting at a large scale, you’ll see it has some huge problems. Which is better:
- One company creates a client that keeps your disk up to date seamlessly. It is their speciality, and they have experts in the field of doing this properly at all levels.
- Every software company by the thousands, large or small, that may or may not have any expertise in how to create synchronisation systems, has to build their own version of the client into their software, with varying levels of completion, stability and data security/privacy. As each developer does this, wasting months of effort on something provided by the first option naturally, they spend less time doing what they actually want to do with the core software.
Case in point, the iOS version of Scrivener probably could have had more features, and had better parity with the Mac version if we didn’t have to waste so much time reinventing a Dropbox client into it.
I’ve just had issues with Time Machine in the past, so I don’t trust it as a long-term backup solution.
Oh yeah, I agree that Time Machine should be considered a nice tool on the side of real backups, rather than the real backup. The main problem with thinking of Time Machine as a real backup is that it is constantly plugged into your computer, subjecting it to the same physical risks as the device you are backing up.
And yes, I too had some reliability issues with it very early on (circa Leopard which was a really buggy OS in general). I think when Apple first released it, it was buggy. I haven’t used it much lately though, so I don’t know what state it is in, but for many years (once they got the bugs worked out) I had no issues with it. I still would only consider it more like a system-wide Undo button though, for the above reasons.
Please explain to us how copying data from Disk A to Disk B is more risky than copying data from Disk B to Disk A, in a way that would make such a flat declaration as you’ve made, universally true.
Secondly, define “risk”, as you are using it. If I copy data to an empty folder and the copy fails for some reason (maybe the USB cable is bad), there is no risk involved because I can wipe the bad copy and try again. So you must be using this word to mean something different.
1 Like
Fair enough, though an open project being backed up would not present a major risk, it would also have a much higher likelihood of being more up to date than any of the .zips, too. An open project on the disk is, with a few minor exceptions, pretty much the same as a closed project. The main differences are the presence of a lock file as a safety for warning you about opening it twice from another computer, and secondly a few non-critical files will not be written yet, like the search index. It’s an expensive file to write since it contains every shred of text, so it is excluded from the auto-save routine. Such a condition would likely be detected by the software when opening a restored project that was left open and rebuild it automatically.
Or to put it another way, you wouldn’t believe how many open projects we get sent to us in tech support. People just right-click zip from their file manager with it open, when we request a copy to see what is going on. I have never seen such a copy fail to open or be in a condition that requires repair.
That all said it does raise a very good point: I don’t recommend backing up your user folder while you are working! That’s a bad idea in general. At the very least close what you can and step away from the computer while it is running, but best case scenario is to not even be logged in while it is running. It’s not something to worry too much about, but Scrivener projects are going to be in the more tolerant spectrum of backing up while stuff is open, than some other tools that are database oriented, like Notes, Mail, etc.
1 Like
Why not run all back ups overnight when you’re not live editing files?
2 Likes
It means a lot that you put so much time and effort into writing such a detailed response. The details of Scrivener’s case that I sometimes wondered about were well cleared up! Thank you once again!
1 Like
The key phrase there is “in transit,” which itself can mean any of a variety of transfer mechanisms.
Email will not successfully transfer an unzipped project, for example. Cloud services behave inconsistently when trying to transfer a folder via their web interfaces: sometimes they automatically zip everything up, sometimes they simply transfer the top-level files like email does. Cloud services have also recently developed the annoying habit of not actually downloading an entire project unless you tell them to. All of these are reasons why using a ZIP file is less likely to result in problems.
But direct disk-to-disk transfer? Dedicated backup services? If they can’t handle a folder structure you’re using the wrong backup service.
There are of course also a variety of user errors that can occur, for instance if the person doesn’t realize that they need the whole .scriv folder and just transfers the .scrivx file. Using ZIP files avoids these as well, but user errors aren’t the fault (usually) of the service being used.
1 Like
How much are you willing to risk loosing? Overnight is fine as long as you are ok with loosing a days worth of work. If/when your hard drive goes you will loose everything since your latest backup.
<Anecdote>
Back in the old days (when mainframes ruled the world) I was the Systems Manager for a large multi-user computer system that was available 24/7 (or as close to that as we reasonably could). We would run hourly incremental backups, nightly consolidated backups and weekly full backups (which were stored in a bank vault several miles away). That way the most we would loose was an hour of work - which met the business requirements. (We also never needed to test our backups because we were constantly retrieving data from the backups for our users when they would accidentally delete or otherwise muck up a file; and unfortunately our hard drives were not as reliable so we had to restore drives several times a year).
<Anecdote/>
1 Like
Time Machine is the default utility that enables you to backup your Mac to an external hard drive for free. Configuration is minimal, but you must accept what Time Machine provides. ChronoSync is a completely distinct entity. For scheduled backups, creating bootable drive copies, and syncing files and folders, it’s the go-to option. You don’t need separate backup, drive clone, and folder sync utilities anymore because this application can do it all.
The downside is that you need to have proper knowledge and expertise to configure ChronoSync on your own. Those who work with multiple photo and video libraries can benefit from the default ‘merge’ option in ChronoSync for package folders. In this discussion, AmberV has proven that ‘dissect’ is the best selection for Scrivener project files.
The fact that L&L can provide support and information for such detailed technicalities is impressive. Thank you!
2 Likes
Yes, I also remember. No mainframe machines, but UNIX workstations. As a youngster, I managed a data network used by over 200 professionals, engineers, and designers. I set the network up with replicated servers, a multitude of RAID disk stations, and automated tape drives with robotics. The media was transferred to a fireproof vault and archived under prescribed procedures. I had the backup tapes from the last day of every month saved forever.
Even though I started using a Mac later in life, I still have a habit of organizing and storing backups of everything in my home office. Someday, I might let go of this obsession… 
2 Likes