It’s not so much that iCloud can’t sync packages so much as it is, if I understand correctly, that it does not sync package contents in such a way as to guarantee that the updates to the files within the package are done in such a way as to not corrupt the integrity of the package.
This is truly a mystery to me. A package, such as a Scrivener project, is just a directory. Finder will show you its contents with the Show Package Contents option in the action menu, and at a shell prompt/terminal window, you can just “cd” into it like any other directory.
I think the reason Finder treats it as a single file is because Scrivener must have a plist file that identifies .scriv folders as document packages.
At a command prompt, if you rename a Scrivener project so it no longer ends in .scriv, Finder will just see it as a folder. Rename it back to whatever.scriv and poof - it’s a package again, and as long as you didn’t alter any of the files inside, it’s a happy Scrivener project.
I don’t want to swim upstream, but I’m not sure why iCloud would have any trouble syncing a folder. The user might have trouble telling when iCloud is finished syncing, I can certainly see that, and a partially synced Scrivener project would be a disaster.
For a view of a similar sort of thing, drop to a command prompt and use the unzip command against a .docx “file”. It’s a file, sure enough, but it’s a file that might as well end in .zip instead of .docx. You can unzip a docx file and look at the parts and pieces if you want. Docx files are zip archives.
Mind your backups when playing with internal organs in living documents.
1 Like
Technically, this is correct. What you are missing is that the component files in a Scrivener project have metadata tying them to each other and to specific locations in the Binder. If the project.scrivx file – the master index used to build the Binder – gets disconnected from the actual documents in the project, all kinds of chaos can ensue. iOS Scrivener in particular puts a great deal of effort into minimizing the risk of that happening, and to do so it depends on the Dropbox API. CloudKit, the iCloud equivalent, does not make similar services available.
A .DOCX file is not equivalent, precisely because it is a ZIP file. All of its components are compressed in a single unit and travel together. It’s not possible to email part of a DOCX file; it is trivially easy to email part of a Scrivener project.
Katherine
To expand on what Katherine said, even though a Word file is actually a ZIP archive of an entire folder/file structure, it does not live on disk as a folder and file structure (the sole exception being in the user’s configured TEMP directory, for transient periods of time). Every time you save it, all of the files that make up that Word document get zipped back into a single archive file that is a faithful point-in-time representation of the document as it was when it was saved, even though the internal components are pieces with a defined schema and links and metadata. That single file is then written to the hard drive and if you sync it with iCloud, Google, OneDrive, or any other sync engine, it is a single file that is completely independent and self-consistent, because Word is working with the copy in memory and the TEMP directory, not with the pristine saved copy it just wrote to disk. It only uses the .DOCX file when you open the file or when you save the file (or otherwise write to it).
A file package, however, from what I understand is not the same way. To your sync engine client looking at the file system, it’s a folder with a bunch of files and sub-folders, linked in the very specific ways Katherine mentions. Scrivener and the Finder treat it like a single file – albeit with the caveats you mention about being able to force your way into the folder structure – but it is at no time a single file that gets unpackaged or packaged up. And the sync engine clients scouring the file system do not see it as a single package – they see it as a folder full of tasty bits and bobs.
And the problem comes in with how they track and sync changes to those tasty bits and bobs.
Say that the sync engine is trying to save bandwidth and be clever. If you’ve just updated multiple files multiple times since the last sync, your sync engine has a choice on how it handles those changes:
-
Faithfully preserve the sequence of writes/changes across the entire portion of the file system the engine is monitoring, and sync those changes in the order they happened on your file system. This is slow and possibly transmits more data than necessary – what if some of those changes were you writing new data into a document, saving it, then reverting those changes and saving back? However, it most faithfully preserves the sequence of changes and is the most “safe” option. It assumes the writes happened the way they did for a reason and doesn’t make any assumptions about file formats or relationships. (This seems to be the way Dropbox works, BTW, from my limited testing and research.)
-
Faithfully preserve the sequence of writes/changes per file, but optimize those so all pending changes to a file happen at the same time before it moves on to the next file. This method will guarantee that your changes to each specific file are preserved, but for a (usually very small) period of time, any files that have relationships to each other (such as those inside a Scrivener package) may not be consistent with each other.
-
Try to optimize, in some fashion, the sequence of writes/changes by removing those that don’t seem to be necessary. If I’ve saved to the same file three times, skip the first two saves and just go to the last version, right? I save time and bandwidth – everybody wins! Unless, of course, those intermediate states track to other changes in other files that now I lose because I assume my files are independent of each other.
Say, for instance, those three saves were renaming the file (because I changed the title)[1], creating a new file with the old name, and adding content into it. By throwing those first two saves away, I’ve introduced corruption. And yes, some sync engines do try to cut corners like this.
The problem gets compounded when you add in multiple clients all trying to push/pull changes at the same time, or the fact that on a Windows system you don’t HAVE a package, you really are dealing with a folder with lots of files and sub-folders, and now you have multiple copies of the Scrivener document open at the same time…all of these levels of complexity add more uncertainty to the mix and are more likely to expose and exercise the flaws in 2) and 3) to fatally corrupt your project.
[1] Yes, I know my example with renaming is no longer how Scrivener does it – UUIDs are good – but the point is there – mess with writes and write order and you can introduce unintended inconsistencies in a non-trivial file system.
To be technically accurate, it is possible, you just have to unzip the .DOCX. However, it takes deliberate non-standard action, so is not trivially possible, so we take your point.
Very, very quickly (because I’m snowed):
-
iCloud has no problems syncing a folder or a package file or anything else. That’s not exactly the problem.
-
There are generally two types of app: shoebox apps such as Photos and iTunes (and Ulysses, for that matter) which have a single window that manages a single database of files, and document-based apps, which use a different window for each file, and whose files may be anywhere on your computer.
-
Shoebox apps provide a window on a large amount of data and just save, for instance, a main index file and the single file affected if you make any changes, without touching the other gigabytes of data in the database.
-
Document-based apps save the entire document every time there is a change. Document-based apps can use flat files or file packages. Either way, the way the technology is designed by Apple, when you make a change, the entire file or package is re-saved (although the OS uses some clever atomic saving to mitigate against data loss).
-
iCloud is, for the most part, built around these paradigms, and integrated into the frameworks based on these common uses. (Which isn’t to say it can’t be used in other ways, but whereas almost any programmer can get iCloud working for these use cases, you would need some serious experience in sync technologies on board to bend it to other purposes and be prepared for a major project.)
-
Scrivener, however, is a bit of a hybrid between these two app types. It uses macOS document-based technologies in order to present each project in its own window and manage a single .scriv file for each project. However, it overrides these technologies at quite a low level in order to support huge projects, since each project is akin go to a mini-shoebox app. It uses a file package (a folder that looks like a single file on any Mac Scrivener is installed), but it does not–unlike standard document-based apps–re-save the entire package every time you make a change to a project. It would be pretty awful, of course, if you have a gigabyte of PDF files in a project and have to wait ten minutes for all of that to be re-saved every time you make a change to a bit of text. Instead, Scrivener overrides all of the standard mechanisms so that it only saves files that have been changed inside the package and leaves everything else alone. Unlike other package/document-based apps, it also overrides all of the reading methods so that it only ever reads into memory files required right now. (The standard macOS mechanism is to read the entire file package into memory–but again, you don’t want Scrivener reading a gigabyte project into memory every time it opens.) This ensures that Scrivener can run fast with any size of project, and ensures stability. However, it means that it bypasses all of the standard mechanisms for iCloud support.
-
Shoebox apps can generally work fine with iCloud because if some files are synced and others are not, they can just show this and notify the user that such-and-such a file isn’t downloaded yet if they try to click on it.
-
Document-based apps generally work because the whole file (package or flat) is saved in one go, so the whole file (or package/folder) arrives as a single entity on other machines. The app can then simply reload the whole thing. (This is also made much simpler by usually only having a single editor showing a single piece of data such as a text document–unlike Scrivener with the many, many UI elements showing different pieces of data, such as the outline, current text, maybe another text, synopsis and so on.)
-
Scrivener agains falls between these, though. A project is a single thing, and a project cannot work reliably if not all of its data is synced. For instance, suppose you make changes to text A on computer one, then move to computer two. But the internet is down and you don’t notice that your changes haven’t synced. So you make changes to text A there, too. At this point, because iCloud (unlike Dropbox) doesn’t make conflicted copies, all the changes you make to text A on computer one are lost forever. And that’s a minor example. You could end up losing entire outlining structures.
-
And because Scrivener saves and loads piecemeal, that’s the way the data would arrive, and Scrivener would have no idea of when all the project data is available to run safely.
This didn’t turn out so quick. 
Anyway, that said:
-
It is perfectly safe to use iCloud Drive to share your projects between computers just as you use Dropbox. As with Dropbox, you just need to make sure that your projects are fully synced and closed on one computer before moving to another computer, and then make sure they are fully synced before opening them again. This is not as smooth or simple as the automatic iCloud sync that is possible with more standard document-based or shoebox-based apps, but it is the payoff for everything else that Scrivener can do.
-
I do have plans to make it so that our iOS version can open and desktop sync projects using iCloud Drive too, and I think I have a solution for this. However, it will take some care on the part of the user, and there is a lot of work involved on my end to get this working (I’ve been going back and forth with Apple over a solution for this for months, picking apart the technical details–the Apple engineer also points out how iCloud and the general frameworks aren’t designed around the sort of thing Scrivener does). So, this won’t see the light of day until sometime later on in 2018. It’s being investigated, though.
I hope that clarifies some of the thorny issues involved.
All the best,
Keith
On top of all that you have this little checkbox under iCloud in System Settings in MacOS:
If it is ticked like this, it means that some things you add on another Mac or on an iPad, won’t show up on the other Mac because OS X thinks that you don’t have space enough and probably don’t need it. I used a photo scanning application on my iPhone and the folder created by the app was visible on all my iDevices but not on any of my Macs, until I unchecked this box.
So iCloud is sometimes a bit too smart in its attempt to help us.
Keith, you have a most wonderful philosophy of customer support. My only regret is popping off and consuming so much of your time!
Dropbox works great, and I guess I need to man up and install it on my systems. I have shied away because I can’t get away from iCloud and didn’t want to install a second sync service. No reason not to, and I already use Dropbox’s web interface for sharing some business data with a partner.
I had forgotten about that option to let Apple decide what to stop syncing. Evil. If I want something synced, I want it synced.
Thanks for your detailed explanation, and kudos to you and your team.
I was excited to learn there was a new upgrade, but was soon disappointed when I found out there was no improvement to the seamless iCloud intergration that so many have been begging for years. Responses like “Scrivener probably isn’t for you. Live with it.” and/or “Your upset that iCloud sync is not available is understandable. ”; are the same standard one liners uttered before, and a poor response, IMHO - pity !.
Note that the response came from a forum member, not a Literature & Latte staff member. You may find the response from ‘KB’ (the developer) much more informative.
Katherine
Hey there! Jumping on…
I am a non-power user. I.E,., I really only have a couple of simple projects under development, and ONLY work on one computer. However I do have my projects stored in my iCloud folder hierarchy because, well, that’s where I store most things. I have never had a problem, but lately have been served up the dreaded “CHOOSE WHICH VERSION” window. The past 2 months I’ve gotten a few of these with TWO competing versions, but I’ve recently been confronted with THREE competing versions.
Is this an Apple problem? My iCloud has been a tad funky lately w/ Photos, music, etc. Which is kinda scary.
I take a lot of care to SAVE and CLOSE each time, too. So…should I just get my projects out of iCloud and off my desktop?
I’m thinking “probably” 
In my experience, iCloud can be very slow in updating, and unless you’re looking at a finder window, there’s no indication of how far it has got with the process. So, while it should be usable, it will take even more care than Dropbox or Sync—both of which have a menu bar icon indicating that they’re still in the process of sync’ing—to make sure the process has completed.
Perhaps you’re closing properly (you shouldn’t need to save as Scrivener should be doing that automatically on project close, but check the preferences on that), but not always allowing iCloud enough time to upload fully before you put your computer to sleep or close it down. If that’s the case, you’ll have different versions on your disk and in your iCloud account.
Mark
For another half-turn of the dragon’s tail, it looks to me like Scrivener V3 projects inside DevonThink databases work fine. I even made a DevonThink template out of an empty Scrivener project, so I don’t even have to leave DevonThink to start a new Scrivener file/project/Polynesian moai.
Of course, the question of why in Bartleby’s name one would want to do that is left as an exercise for the enterprising scrivener. For my part, I feel life is often too safe and assured. In this case, I don’t think it’s that risky since everything in a DevonThink database is stored in good old MacOS folders. They are inside a package, for a hint of mystery, but otherwise, everything is stored in regular OS subdirectories.
I would go into further detail, but stalwart Rocinante stamps his impatience and giants are at the gates!
Trust your muse, but keep current backups. Not necessarily in that order.
Still, the ability of syncing using iCloud is very very important.
As a Chinese user, when I am in China, there is no way to access Dropbox smoothly but there is no problem for connecting iCloud.
If you are only wanting to sync projects between Macs, between Windows PCs, or between platforms, have a look at Sync, https://www.sync.com. It is accessible in China, works just like Dropbox, though I have a suspicion it is very slightly slower than Dropbox, but it’s much quicker and better with Scrivener than iCloud. Just like Dropbox, you need to check that sync’ing is complete before shutting down/putting to sleep your computer. I use it with my collaborators in China.
If you’re talking about Mac <—> iOS, then for the moment you’ll need Dropbox and a VPN.
Mark
Thanks to this thread understand why Scrivener only ‘syncs’ properly with Dropbox. I just wanted to say that I hope this issue gets resolved some time. I share the OP’s concern about DropBox over the longer term. IAWriter stopped working with it recently. It’s pretty invasive with the access it requires to your system and what it wants to modify. Scrivener is the ONLY reason I have Dropbox running on all my machines. It’s another cloud service, on top of Google Drive, iCloud and OneDrive that I have running, but the one I have no use for apart from Scrivener.
I’m in the same situation, I only recently decided to return to Scrivener for a major project and was hugely disappointed to discover I had to re-subscribe to Dropbox (LONG ago abandoned) just to use the iOS version of Scrivener. I have my fingers crossed that an iCloud solution might appear in 2018.
This is not accurate. You don’t need Dropbox to use iOS Scrivener, just to synchronize with other platforms. We have customers who use iOS Scrivener as a standalone application, and others who use iTunes file sharing to transfer data back and forth.
Katherine
Another vote for an alternative to Dropbox syncing, preferably iCloud. As venerable tech publication Macworld says in their May 29, 2018 roundup of cloud storage services:
There you have it: Dropbox is no longer regarded by mainstream tech publications as a best-of-breed solution in 2018.
Scrivener is literally the only app I use that still has a dependency on Dropbox for sync. Every time I’m stuck waiting on a Dropbox sync while opening or closing a Scrivener project, I start thinking about migrating to other writing tools. And as someone who’s bouncing back and forth between an iMac and an iPad all day, that adds up to a lot of staring at the Dropbox syncing progress meter.
That doesn’t really matter from a developmental point of view, if the majority of users are using that over other synchronization services. Dropbox has half a billion registered users. If even 25% of those accounts are active, it is an enormous number of users.