Clarification of iCloud concerns

Scrivener Scrivener for macOS
1

New user here.

I’ve read about the problems of using Scrivener with iCloud but I’m still not clear–many (all?) of the articles are concerned with iPad use.

On my Macs I have the Desktop set to be stored in iCloud (and thus available directly on all my machines). Is this a safe way to use Scrivener–if not does that mean I have to turn off this otherwise very useful feature of the OS…or?

Just need to make 100% sure I’m not heading for a disaster before I start to use the program more intensively… :slight_smile:

–Darin

2

As best I understand from following the forums for the past few years, if you store\sync live Scrivener projects on iCloud, you are heading for disaster. DropBox is the only such service cloud storage/sync service that L&L formally recommends, for technical reasons.

For background, do a search of the forums (upper right hand corner) using “iCloud”
literatureandlatte.com/forum/sea … rds=icloud
and also see these two knowledgebase articles
scrivener.tenderapp.com/help/kb … c-services
scrivener.tenderapp.com/help/kb … g-with-ios

Be aware that even with DropBox, one must exercise some care (discussed in first of above knowledgebase articles).

On the other hand, compressed (.zip) versions of projects should be safe on most all services. But of course one must first decompress (unzip) such off such services to local drive before being able to work on them in Scrivener, and later recompress (zip) them from local drive back to such services.

Hope that is more help than hurt.

3

Ouch. That hurts. :slight_smile:

Since every document on my laptop is auto-synced with iCloud, turning it off the satisfy the requirement for one program is probably not workable.

My god, don’t tell me I’m forced to go with a subscription-model product–I detest the subscription model concept!

Any clues if version 3 will have a solution for this issue?

–Darin

4

It will not. This is fundamentally an Apple issue, not a Scrivener issue. While Scrivener 3 includes some changes to the project format, the basic structure – lots of small files of multiple different file types – remains the same, and Apple’s difficulty in managing it is the root cause.

Katherine

5

Ouch, again. Sounds like Scrivener is inherently unusable on my system. Now what do I do? :frowning:

–Darin

6

iCloud Drive is a reasonably safe location for Scrivener projects. (Subject to the caveats that apply to any synchronization service.) That’s what I’d do. (Well, actually, I use Dropbox. But if you refuse to use it for whatever reason.)

Katherine

7

Oh no, I’m confused again. I’m not entirely sure of the distinction between “syncing” and “iCloud Drive.” All my Desktop files are automatically shares across my computers. There is a folder on my iCloud Drive labeled "Desktop and the files are all in there. I don’t manually do anything–files are automatically put there (and updated there).

is this the dangerous thing or the reasonable safe thing?

–Darin

8

Apple’s naming conventions can be somewhat confusing…

iCloud is a collection of Apple services for sharing data between iCloud-enabled applications. These services are not supported by Scrivener, and therefore iCloud syncing will not work for sharing projects between, say, Mac OS and iOS.

iCloud Drive is a virtual hard drive that works similarly to Dropbox. Therefore, in theory, it is safe for Scrivener projects, provided that you only use it for backups or for sharing between Mac OS systems. (Subject to the caveats that apply to any synchronization system.)

HOWEVER, the Documents and Desktop feature of Mac OS is extremely risky if the amount of data you want to store is larger than your iCloud subscription allows. Which it will be for many people, as the “free” allocation is only 5 GB. There are all sorts of reports of alarming behavior if you either overrun your space allocation or try to turn the feature off. I do not use this feature for my personal data and do not recommend it for naive users.

Katherine

9

I stored Scrivener projects in iCloud Drive without any problems, but decided I would sync a USB drive to a normal folder based on the warnings.

But I never understood why iCloud Drive wouldn’t work, and I didn’t see any troubles. it syncs files. Lots of them, if need be.

I’d like to see clarification on this, too. If iCloud is safe, assuming you wait for sync to complete on power up and shutdown of any device. What’s the downside?

10

Note that this entire conversation only applies to Mac OS systems. iOS Scrivener has to deal with additional complexity because of the way files are managed on iOS devices, which is why we’re only able to support Dropbox synchronization. Mac OS has a real file system with a library of standard tools for file operations.

In theory, all of these services should work: Dropbox, iCloud Drive, Google Drive, and several others all do pretty much the same thing. They all create a special area on your disk that is synchronized with their server.

In practice, it’s important that (1) the software give a clear indication that your changes have in fact been synchronized, (2) the software upload all changes promptly in the order in which they were made, and (3) the software NOT get cute with ‘cleanup’ features like duplicate removal.

(1) Is a matter of user interface design. Should be easy, but somehow seems to elude some services. iCloud Drive is not clear about this, which is one reason why we don’t trust it enough to use internally.

(2) Seems to be the underlying cause of Google Drive’s poor performance. Scrivener saves very frequently, and it can cause enormous problems if the 9:15 PM version of a project has only partially uploaded when the 9:20 PM save occurs. (This is also why it’s a good idea to pause synchronization if you know your internet connection is poor.)

(3) Some services try to save you disk space by removing duplicate files. That’s a terrible idea if, say, you really did want to have three different copies of your author bio, one for each installment of a trilogy. Or if you have five or six backup copies in which most (but not all!) of the files are the same. This behavior seems to be more common in tools that started out as backup services and morphed into synchronization services as the market changed.

Katherine

11

Darin,

It’s not really clear to me either. BUT in any case there is a very easy solution/workaround which should avoid any problems arising. [1]

  1. Create a new Folder (let’s call it Scrivener Documents) which isn’t in your iCloud Drive. The best place to do this is in your Home folder (in Finder Go > Home or cmd-shift-H). Drag the new folder into the side bar for easy access.

  2. Copy your existing Scrivener projects into Scrivener Documents, and make sure you create all new Scrivener projects there.

  3. In Scrivener Preferences go to Backup and make sure you are creating regular automatic backups INSIDE the iCloud Drive and that they are Zipped / Compressed. (Best is to up the number to 25 as well…)

This means you’ll be able to work on the current files safely outside iCloud’s automatic syncing and that the backups will be synced into iCloud for safety (all flavours of iCloud work fine with zipped files). The distinction between various types of iCloud won’t matter then.

It also means that the current files are only available on the one Mac — if you want to do want to share them easily on more than one Mac, then most people use Dropbox. My Current Work folder is on Dropbox on both my laptop and desktop and I work on Scrivener projects there – all I need to do is remember to close the project on one machine before I open it on the other. Backups are zipped and are in iCloud Drive. (Make sure your Dropbox folder is also outside the iCloud Drive and directly in Home – I think that’s the default anyway.)

[1] Note to those more knowledgable than I – if there’s anything wrong with the above please let Darin know!

12

Hi,

Katherine has given lots of good answers here, so I’m probably not adding much, but:

Basically, it’s perfectly safe to store your files on iCloud Drive, nearly as much as it is on Dropbox. However, with any cloud service, you just have to be careful to make sure that files are fully synced between machines before you use them. So, if you edit a project on computer A, you need to be sure that the changes have uploaded to the cloud service from machine A and also downloaded to computer B before you use the project on computer B. And you need to have the project closed on the computer on which you are not using it to avoid conflicts.

As long as you follow these rules, you will be safe on iCloud Drive or Dropbox.

The downside of iCloud is that it is not as transparent about when it syncs files as Dropbox. Dropbox has an indicator right in the menu bar showing when it is syncing. As soon as you make a change to a file in Dropbox, it starts uploading. And you can see the sync indicator on your other computer to check that things are downloading. For iCloud, you have to find the file in the Finder, because the upload/download progress is shown there, not in an easy-to-see place such as the binder.

Also, Dropbox is much better at making conflicted file copies than iCloud Drive. If you do accidentally work on a project that hasn’t finished syncing, Dropbox will make file conflict copies, and Scrivener can retrieve them. You’ll then get a message in Scrivener telling you that unexpected files were found and you’ll have to decide how to handle them - an annoyance, to be sure, but at least you can recover your work. On iCloud, conflicts are just lost, so you could lose work if you aren’t careful about following the rules of ensuring that everything is synced before you open and work on a project.

In most apps, there are no real complications, because most apps have a file format that consists of a single file per document/project. But a Scrivener project is really a folder containing many files - thus allowing the user to bring in research files, and allowing Scrivener to run fast even for massive projects, because it only has to load into memory the files it needs for the current session (whereas single-file apps have to load the entire file into memory). So Scrivener is essentially a hybrid between a shoebox app (a UI for a repository of many files, such as Photos) and a document-based app (an app in which each window operates on a single file, such as Pages). iCloud wasn’t built with this sort of app in mind, because most apps are one or the other rather than a hybrid. This is why you have to be a little more careful in Scrivener.

Ultimately, though, I would just say: if things are working for you on iCloud Drive, just carry on. Just be sure to have Scrivener’s automatic backups feature turned on, so that if anything does ever go wrong, you can easily restore from a backup.

All the best,
Keith

13

Keith, thanks for that great information regarding iCloud Drive issues.

There are a couple of ways to see iCloud sync. In the Finder, a little clock face will show sync progress in the sidebar (view -> show side bar, or command-option-s) if you have a shortcut to iCloud there. The clock face will appear beside the iCloud shortcut.

Individual things waiting for upload or download will appear in the lists of files with a little cloud symbol.

For more detail, turn on the status bar with view -> show status bar (or command-/). That will show you the available space on iCloud if it’s synced, and if it’s in the process of syncing, you’ll see how much is waiting to transfer.

I think I’ll move back to iCloud for my writing - but I’ll keep backups turned on in Scrivener to a local, non-shared directory and I’ll make manual backups. Time Machine, too, with my rotation of three backup drives.

Belts, suspenders, and hard liquor. No cutting corners with disaster recovery and backups.

14

Thanks for that extra info, LuckyJack!

15

This has been a very helpful discussion. Thanks to everyone.

Regarding iCloud Drive, it seems to summary is:

  1. iCloud is fine as long as you have enough disk space (not a problem for me, I have the 2 TB plan).
  2. iCloud is fine as long as you don’t quickly shut the computer before it has time to upload/update files. You can monitor this progress (which seems very quick with a good connection).
  3. iCloud is fine as long as you don’t keep a copy of the Binder open on one machine and then start using another.

I’d like to explore that third option a bit and then wrap this up by learning what the worst case scenarios are.

It is, of course, common to leave apps open, often for months on end, on current computers. For large writing projects I can easily imagine the document (Binder in this case) might also stay open semi-permanently. What exactly will happen if I leave the app open on my laptop, close the laptop (after iCloud Drive update) and then open that file on my Desktop? And then make changes there (letting it update to iCloud Drive) before going back to the already open Binder on my laptop? In short, must I close the Binder after every use (a habit very much contrary to my habits with other programs)?

Also, what about situations without good (or any) internet connections on one side or the other in that scenario, say when traveling?

For all of these situations (assuming I have auto-backup turned on), what is the worst case scenario? Do I lose actual text that I’ve written or the research files I’ve gathered? Or “just” the Binder and all the work I put into organizing the writing project?

I don’t want to over-emphasize the worst case scenario aspect. After all, back in the typewriter days writers ran a huge risk in losing original drafts on paper. Even in recent times, before Dropbox and iCloud, it was all too easy to lose files, accidentally deleting them or having a hard drive fail. I suspect that whatever worst-case scenarios are outlined will, by comparison, present a fraction of the risk we routinely endured just a short time ago.

But, that said, what are the worst-case scenarios with iCloud Drive and how might one attempt to mitigate them?

–Darin

16

There are two potential bad things that can happen, and I should emphasize that they can happen with any synchronization system, including Dropbox.

(1) The file used to build the Binder doesn’t match the actual contents of the project. This will cause “blank” items, as the file that Scrivener opens (or tries to open) doesn’t exist but the Binder entry remains.

(2) Copies of the project on different computers have different contents. If you make changes to a document on one computer, then edit an unsynchronized version of the same document on another computer, you have just potentially created two conflicting copies. You could end up with Version A on Computer A, and Version B on Computer B. Depending on what you do to resolve the conflict and how good your backups are, it IS possible to lose work in this scenario. Note that any part of the project can be exposed to this kind of conflict, including the master Binder file.

We do not recommend leaving a project open indefinitely. Even if you aren’t synchronizing, doing this keeps Scrivener’s automatic backups from running. If you are synchronizing, it’s a terrible idea.

If your connection is bad or non-existent, the best solution would be to treat the project you’re actually working on as canonical, and make a local backup of it before you attempt to re-synchronize.

In any data loss scenario, you are most likely to lose whatever you were actively working on, whether that’s new writing, research files, or the Binder structure. Older data is more likely to (a) not have changed between two potentially conflicting copies of the project and (b) be captured by a backup file anyway.

A full discussion of best practices to avoid synchronization errors can be found here:
scrivener.tenderapp.com/help/kb … c-services

Katherine

17

Thanks for the info. Have you moved back? If you have, has it worked well?

Anyone else, please feel free to chime in.

18

I’m not out of hard liquor yet, but I’m giving it high priority. In the meantime, I’m still using a USB thumb drive and a two-way sync utility, editing on both my iMac and my Macbook Air.

I have some notes in Devonthink, and have discovered a Scrivener project can be stored inside a Devonthink database.

That’s pretty cool.

19

Stored only or viewed also?

20

Stored, viewed, any usage in my experience.

Anything you put in Devonthink is stored in a regular filesystem folder. You can select something and choose “view in finder” and it’s in a regular subdirectory. If you note the path, you can “cd” to there in a terminal window.

I haven’t tested making changes in Scrivener and then searching in Devonthink for the changes. I believe that will work, because if I open a Nisus document or a PDF in Preview from inside of Devonthink, Devonthink updates when the application (Nisus, Preview, whatever) closes.