Seeing Dropbox right...

Scrivener Scrivener for Windows
1

Which is a sort of tricky title: maybe I spent too much time once, in Old Blighty…though with some of the nicest of memories, truly, so I think I get to call it that, in common with sentimental others.

I want to preface by saying that I really considered just passing this article to Lee and Keith. However, given the workload they have, it seems a better way could be to let the community handle as much as possible. We’ve got some very knowledgeable folk here, especially in this Beta area, along with a full group of persons who may in various ways wish to use Dropbox with Scrivener for their purposes.

What I want to talk about is first in recognition of how many persons find Dropbox to be the right kind of magic. Indeed, with some background in replication systems, one can say it is one of the best of its kind, and that it can make many things much easier in a world of multiple computers, hand-held devices, and collaborators.

The central point we have to practically deal with is this: Dropbox is by necessity rather simple-minded. It’s meant to work with one file at a time. And that is truly it: one single file which has your content.

If you collaborate with someone or share files to two of your own computers, and both ends make changes to their copies of a single file, Dropbox will simply keep both copies present and showing, adding a tag to the name of one of them — what they call a ‘conflict name’ — based on who saved first. There is no more sophisticated method — no automatic merging of mismatched content — because in a general -purpose replicator, there really can’t be.

At least, there can’t until we reach the level of computer intelligence (malevolently) shown by SkyNet, if you enjoy watching Arnold in the movies. That’s because merging, as you’ll realize from having done it, is a pretty much truly human intelligence-requiring task.

To cut to the chase, using Dropbox directly isn’t going to dependably work for a Scrivener project, because it’s a file-filled folder, rather than a single file. I’ll explain more completely below, but the proposed safe way to employ Dropbox will be to share a zip file of the Scrivener project, rather than the project itself. That’s so we can be working with a single file that Dropbox can safely handle .

Now, Scrivener does us a lot of favors by keeping its items, ‘scrivenings’, notes, and etc., as separate files, structured within the project folder. We are most of all safe from the bane of special purpose software which has unusual file formats: we can always get at the text we don’t want to lose — and will be able to do so as long as anyone ever builds tools that can work with a most prevalent of portable layout formats, RTF. The deal is done there.

The problem comes when we try putting that Scrivener project directory itself into Dropbox. Now there are many small files which could be changed. If your collaborating partner makes an update on the same Scrivener binder item that you also change, then one or both of these changes is going to have its name altered, per Dropbox conflict-file protections. Thus the changed version won’t show up in your Scrivener binder at all. You won’t know it’s there, and one of the updates will seem to be lost.

Thus with any mutually conflicting changes, you would need to discover that one or more extra small files even existed within the Scrivener folders, so that you could merge them. Worse, there might be conflicting differences made in the Scrivener document structure itself, which you could not merge, because they aren’t things a word processor can handle. Even if you could handle it all, you’d have to check for safety each time, before using a Dropboxed Scrivener project.

This is not as a practical matter something you want to have to worry about. And if you think you haven’t seen the problem, it’s very likely that you will. It can happen, as in the details of Keith’s present Mac Scrivener Dropbox recommendations, for a wide span of reasons even if you use Dropbox only by yourself. It’s much less likely that way, which is why you may not yet have seen it so far, but it is always waiting to happen.

To solve the dilemma, since Dropbox is a very useful tool, I’m going to suggest that the useful answer will be to Zip Scrivener projects, and then put the updated zip into the Dropbox, rather than the project.

With zip files you are safe from issues of timing and other Dropbox points or Dropbox users, because zips are a single file.

Any time Dropbox sees a conflict, and I believe the most recent version of Dropbox is felt to be very sound in its abilities for this, it will create a copy of the zip file with a conflict name as well as the original title. If five persons modify a Scrivener project in collaborative conflict, you can still easily know about and discover the individual writings of each one, then decide how you want to merge them.

This merging is what you would always have to do, if you think about it. The difference with zips is that you will be safe in this, and that you will know about the need for merging.

What would procedures be?

  1. Before creating or reloading a project from the Dropbox zip, Keith’s instruction is entirely proper. You must fully close Scrivener, to assure all its files are fully written.

  2. To create the zip, you can use the familiar method which works best for you. Any of the zip utilities makes it the effort of a small moment to zip a folder or directory, which a Scrivener project is, and lets you automatically keep the project name on the zip. You then put the created zip into Dropbox. It replaces any same-named project zip already there.

  3. To use the updated project from the other end of Dropbox, and again having fully closed Scrivener, you will rename your present Scrivener project folder - just an added letter will do. This will avoid any possibility of over-write errors, and it will also assure you get a blocking warning if you forgot to close Scrivener first.

Then with later versions of Windows, as well as most of those utilities, you can unzip simply by double-clicking the zip file, and then dragging the Scrivener project out to the place you’d like it to be.

  1. If you have conflict versions, then you’ll re-name each unzipped Scrivener Project before dragging out the next. This way you can open the separate projects, as you would always have to, Dropbox or no Dropbox, to merge the changes between them.

To complete this round-up, I’ve stopped short of recommending using automatically zipped Scrivener Backups as the zip files, though you could possibly do so as soon as that ability is back in place. The reason is that you would then have a different name on each zip except for the lucky moment when someone else backed up the project at the same date and time. You could choose this method, but for most persons I suspect the better answer is to let Dropbox do its labeling, because the conflict name tells you who wrote the conflicted file; and if it’s not conflicted, you want the zip file name to remain the same.

The one other thought to bring up is that if what you need is really to collaborate, there is now a free tool which should be exceptionally capable for it. That’s Google Docs, where you can actually even see another person typing on the document. I think I might be tempted to use that myself, where strong collaboration were needed, by Compiling a Scrivener project to an RTF including the optional horizontal lines to mark the components. That way, and with the anticipated Docs markup, it would be easy to paste back in just the final changes.

I think that’s all I want to say about this, and I already see that there’s likely to be discussion about it.

Let’s see how reasonable this plan may sound. I think to be pretty assured something like it is necessary, but perhaps someone has an even nicer answer. As Ioa/AmberV has noted elsewhere, Scrivener and Dropbox are the very best, each in their areas of solution, and we should enjoy all that we can in using them together.

Regards to each,
Clive

References:
a. dropbox.com/help/36 - conflict handling in Dropbox
b. https://forum.literatureandlatte.com/t/using-scrivener-with-dropbox/11295/1 - Keith’s instructions
c. lifehacker.com.au/2011/04/ho … work-tool/ (gets very complicated, and recognizes conflict. Comments below it recommend the new Google Docs instead.

2

Well, the only posting I got which received less response vs. its readership was the one where I mentioned music. Sometimes we live and learn.

What I wanted was to get Scrivener and its practice gently enough on a road towards being able to feel Dropbox won’t cause reliability problems with it.

A thought which later arrived is this:

Thus as soon as the zip functionality is clean again, a five-minute job would add this Export ability. Given it remembers where you last used it, then Exporting Zip to a Dropbox become a one-moment task, making things very smooth.

Hoping this can make it into reality.

Regards, and a nice week-end to everyone,
Clive

3

Okay, I’ll bite.

If I was collaborating, I wouldn’t use Dropbox at all. If the project was not too large, I would probably recommend Google Docs.

But if things were large or complex in some way and you wanted to remain independent from your collaborators for the most part (this is the “distributed development model”, as software engineers call it), then I would suggest trying what some of them use: git or Mercurial. There are free sites that host repositories for your project using either source code control system – github for git and bitbucket for Mercurial. It shouldn’t be too hard to pick one and figure out how to use it. I spent about 10 minutes googling around and discovered TortoiseHg (for Windows), a GUI interface to the Mercurial source control system.

Being a software engineer by trade and already having experience in it, I use svn, Allway Sync, and Dropbox. This is not a problem because I am not collaborating. Thus no conflict mangled names. Right now, I’m using Dropbox just as a remote offsite backup tool. I have non-Scrivener files stored in svn and Scrivener projects in non-svn sandboxes. I have two Allway Sync jobs. Ones backs up my directory tree of writing and the other backs up my svn repository. If I don’t check anything into svn, I use the directory tree backup job that copies that to Dropbox. If I check something into svn, I use the Allway Sync svn backup job.

When I transition to using Scrivener for everything – after Scrivener for Windows settles down – I expect to check my Scrivener projects into svn and only use the svn backup job.

I do not use Dropbox directly. I have a Dropbox directory that I copy to using Allway Sync jobs. I do this so that a network outage does not eliminate my ability to write. Works like a champ.

But again, if I was going to do serious collaboration with others, I might learn Mercurial and TortoiseHg. Doing so would allow all of us to use the existing web-based tools to keep track of our changes, albeit without the Scrivener UI online. Scrivener would remain a strictly local tool for each individual writer. But updating and merging should be relatively straightforward given that Scrivener projects are just a directory tree of text files.

Or just fallback to Google Docs.

4

Just thought of an issue with using source code management systems like svn, git, or Mercurial.

Somewhere in a Scrivener project there is at least one file, perhaps more, that contains the metadata for the project, e.g., the structure of the project, the names of the folders and text documents within it, data about snapshots, etc. I haven’t looked for it (or them) but I know it must exist.

I’ll assume that data is also text so, in theory, it shouldn’t be impossible to merge. It would have to be merged if two or more people added or deleted any metadata at the same time. I expect that collaborating authors would be adding and deleting such metadata all the time.

That data is going to be difficult to merge. One would have to know the format of the metadata file(s) in order to successfully merge changes to a Scrivener project. That format is possible to discover (or simply ask Keith), but I would expect it’s not trivial. And having to abide by some rules about how to merge changes to a Scrivener project’s metadata is going to be inherently fragile. If one author does not merge the metadata correctly, the project is corrupt.

Perhaps Google Docs is the way to go for folks who are not software engineers. It all depends on the format of the metadata and how difficult merging changes to it would be.

5

Hey, jravan, excellent responses both, thanks. A bit technical for the general audience, but understood well here.

  • I really hadn’t considered source management tools, and that was a good thought. I had hewn close to the DropBox line because I know that as with certain audio editing tools in radio broadcasting, DropBox is a tool many have gotten savvy with and will want to use.

  • The Tortoise tools are excellent; use them for some web projects myself, and they are as simple anything could be, to work with source management systems. Thus could be appropriate for a writing project, sometimes gets used so.

  • I had to think twice about whether source control would succeed in doing merges in Scrivener projects. The problem actually I think is as much in the documents as in the structuring file, as they’re in RTF rather than open text, so you really couldn’t see what you were doing to select and control what got merged. Similar problem then with the structure file, as text XML

  • So I think we’re back to making things behave well with DropBox, and using Scrivener itself to make any merges.

  • I think Keith has a definitive set of rules for doing this, and where it can be applied.

    == From my own point of view, and given use of zip files, it’ll work fine for one person operating in multiple locations – they just pick up where they left off each place.

    == With two or more people, even with corruption possibilities out of the picture by using zips, it’s much more tricky. As he and Ioa say, a lot of clear communication is necessary, though less with the zip scheme. Merging then takes opening two or more Scrivener projects. and manually discovering and editing changes between them, right in Scrivener. Can be done, definitely; and especially if persons make sure they are working in different sections of the document, gets much easier.

Ok, then. Jravan, looks like by your effort we’ve covered another distance of the water. Let’s hope a zip scheme keeps us alway on the surface of that water.

And mostly, that all this is in aid of Lee & co.

Best,
Clive

6

Just wanted to say; I completely agree with this for strong same-document-area collaboration. Google Docs is really powerful for this, and won’t make mistakes.

Here’s what I mentioned about this idea towards the end of the original post – probably gets lost there.

In the end, Dropbox with Scrivener likely remains a hit for many Scrivener collaboration scenarios, particularly where it’s just to automatically update several worksites for a single user; or where in following Keith’s rules, excellent communications keep disparate writers working in their own parts of the project, so that individual scrivenings are updated only by one person.

When it’s full-on collaborative writing, I’m with you that Google Docs is the way to go, and proposed a way of merging such a session back with Scrivener easily.

Best, and thanks again for your careful thoughts,
Clive

7

Perhaps for sharing or collaborating on a project, the zip method would work well, but another possibility might be to tweak Scrivener, or release a dedicated “collaborative” version with a check in / check out function for individual files so that two users don’t edit the same file at the same time. Dreamweaver offers functionality akin to this, obviously without having to negotiate with a Dropbox account, but I think the concept is broadly the same.

If a file is “checked out”, there could be a lock on the file preventing it from being opened in edit mode and a warning could inform the user trying to open it that it is already checked out by [user], and offer to open a read only copy.

Once a file has been “checked in”, it will synchronise between the various users’ dropboxes and then become available for editing again.

8

Ed, those are all good ideas. Actually, I’d thought over a range of them Sunday after posting.

The problem as I keep seeing it (again, having gotten involved with replication professionally once) is that you really don’t have any kind of arrangement from Dropbox that’s going to keep track of a check-in/check-out properly. Scrivener would have to pro-actively check in with such a system if it did exist, also.

As I see it, without an atomicity/journalling arrangement, you might be able to get something like this to appear to work much of the time, but it would in the end not be reliable.

I don’t know; maybe I’ve got that professional outlook too much on the brain. A mark-busy/checkout system would increase security, that’s true, and that might be valuable. It’s just that the most likely situation for it to break down would be during a hot-and-heavy mutual editing from two sites for a document, say to meet a deadline.

That’s also where failure and ensuing loss of text would be most disliked, I think.

Good on you for keeping the alternatives discussion alive.

Regards,
Clivfe

9

Can we sticky this one? Seems there’s an issue at least twice a week with dropbox, and the information in here would be quite useful.

10

Good idea; and done.

11

Real-time collaborative editing is, from what I understand, really difficult to get right. I suspect this is one of those areas where the focused resources of a large company are really going to be tough to match. I have a lot of complaints about Google Docs, but versioning is one thing that they absolutely nailed.

Katherine

12

Adding to my earlier post… I thought the question was interesting enough to raise with my husband. He’s a software engineer whose experience includes, among other things, writing synchronization code for a corporate calendar system.

His take on it:

  • Dropbox is completely unsuitable for real-time collaboration. It’s file-based, and you need a system that’s transaction-based. (And as a side note, trying to do real-time collaboration when the files are hosted on a server that the application can’t access directly is “insane.”)
  • Synchronization is a hard problem all by itself, especially for an application not designed around it from the very beginning.
  • So is network-based sharing. (His current job involves maintaining an extremely large distributed file system for a major internet company.)

Probably not something our Fearless Leader is going to want to think about, absent a good third-party collaboration framework.

Katherine

13

So… what would it take for an open-source, homegrown, collaboration site that would be built-to-match Scrivener’s needs, giving hooks to Scriv source code to use for real-time display/coordination/etc.?

There is nothing proprietary or secretive in Scriv’s handling of files, is there? Just complicated?

14

Perhaps all it would take is a Scrivener-aware merge tool for either svn or Mercurial. I think the collaboration sites themselves already exist. I’d suggest Mercurial because the bitbucket free hosting site for Mercurial projects is quite active. xp-dev.com seems to offer free svn hosting, so that’s also a possibility. Both Mercurial and svn allow external merge tools.

I don’t currently have any experience in Mercurial. My existing background is svn and before that ClearCase (and before that, things even older and clunkier).

So it’s not impossible. Someone simply needs to learn (or already know) the Scrivener metadata formats and learn enough about (or already have enough experience with) RTF.

Sounds possible. But as always, the devil is in the details. If there are intractable problems that others already know, either in diffing Scrivener metadata or RTF, please speak up. I can already say, based on my experience with XML, that doing a text diff of two XML files is not the way to understand XML changes.

And a merge tool could find use outside collaboration. Even in single author projects, such a tool would be useful. For example, it would be possible to diff and merge changes from previous backups, and Mac-to-PC or PC-to-Mac changes.

15

Yikes – you guys are heading this into a technical discussion that I was scrupulously trying to avoid, as I think the real issues are about practices and possibly simple additions to Scrivener like the export-to-zip. And had wanted to pitch talk about it so that we got something done those places.

I think this way because I actually have background which includes something like Katherine’s husband. Just not wanting to talk that way here (or perhaps ever again :wink: )

  • first of all, he’s quite right about the superb difficulty of doing any kind of really reliable compound (multi-part document) replication. The magic word I used in posting above is atomicity, and it ain’t easy across distance and unreliable connections. There’s a reason for what you pay, for example, for a database which can actually accomplish this.

  • then, if you had that, then you still have a fine problem in an editor for merging. You are trying to reconcile a structured document, the nested scrivenings, on top of the RTF merges for its parts. If this compound merge were to be done, there’s no finer framework to start with than Scrivener itself. But before you run off thinking about that, consider how Microsoft Word’s document compare/merge works. Complicatedly indeed. And it is just merging two simple Word or RTF files.

You’d have all that, plus keeping track of actual nested scrivening addition/destruction/moves, in a competent full edit/merge design. Word would give you some idea of the user interface challenges just for the RTF individual scrivenings, and then you do it all again for the nested structure, differently.

My personal take is that this is beyond the remit for Scrivener, at least in the present lifetime. I guess it could come, as it finally did over the 20 years development of Word.

  • Sorry to have to think that none of the variations over source control systems are going to do this for you. They can only ‘save the texts’, Mercurial back to SVN or before. You need the compound merge editor to make any sense of the results, just outlined above. The problem is quite as you say, jravan, about the meanings that go beyond text, in working with an XML file, and expands from there.

My suggestion is that speculation like this makes a fine off-sides topic, without doubt, and for another posting stream, but that we should leave a main line open here for working out the simplest possible means to help the Dropbox pattern work best as it can, for people who are helped by that.

Sound good?

Best to each, and hope you’ll make allowances that it’s late here,
Clive

16

Treading softly, but it did occur to me on waking this morning that maybe a good solution for several issues would be to make a zipped project the actual default Scrivener format.

Now ducking, but here are some thoughts behind this.

  • it would solve the Dropbox problem as well as it can be solved for now.

  • it would remove an issue I haven’t talked about that comes in Windows 7, that if you are using Libraries as is the default, and viewing by date, your Documents library view (think My Documents) becomes filled with small bits of recent Scrivening.

  • It’s actually reasonably common for file formats to be containers (mov, wmv, etc.), and often enough using zip as the enclosing format (lost for example, but know it’s so).

  • new-format Scrivener projects would get their own tag; for example, MyProject.scrv

  • Long term safety of texts would not be affected. You would always be able to open the scrv file with a zip utility, or rename it to MyProj.zip for convenience in this. The RTFs, notes, etc. would be right there.

Well, there it is. Shoot away, please.

Hoping not to have been rude last night, also, in a moment’s reaction about directions of the discussion.

Am sure the right answer is rather that we talk about anything, and then a new Sticky posting is made later by Scrivener HQ when they decide what further they want to do, if anything, towards official Dropbox use or recommendations.

Guess we are all still learning, about the ‘new communcations’.

Regards,
Clive

17

To allow DropBox to be a viable tool to Scrivener users who aren’t collaborating, the zip container idea would have to be optional. The reason being is that a Scrivener project can get quite large, and even the tiniest of changes to your project would mean a sync of every byte of every file in the entire project. The sync would take a long time for even a modestly sized project with just a few pictures and webarchives.

As it stands, individual files that change in a project require only a few seconds to sync up if they are small files, which makes DropBox a very handy and unobtrusive utility.

18

Hmm. Very good point indeed.

I like the idea of a configurable option – best to please the most people and their ways of working, and makes clear the older ways are always preserved.

I had meant to suggest anyway that Scrivener would still open unzipped projects; just prefer by default the new arrangement.

Also, though, Zip archives can be written uncompressed, used just as a container, which is enough to solve the problem we want to solve here about avoiding inconsistencies.

Then, as far as I know, Dropbox does incremental syncing – transfers just the portions of the file that’ve changed. If the Zip archive isn’t compressed, that should mean only the changed areas transfer, preserving the quick response you’d like.

Incremental syncing is also a quite long-known tactic, and I imagine any alternatives to Dropbox would also work that way.

Good thoughts to spark…

C.

19

I had been under the mistaken impression that the whole file would sync with even the slightest change in any part of it. Turns out I’m wrong.* So moving to a zip archive would actually be a viable option, if it can be done for both the Windows and Mac version. I’m not seeing a lot of down-side to this, but I would think that Keith and the gang would have considered this as an option and discarded it for some good reason; the longer I’m here, the more I notice that hardly any idea presented by a user is a new one on Keith.

  • I am taking the DB website at it’s word… dropbox.com/features : “Dropbox transfers just the parts of a file that change (not the whole thing).”
20

Good man.

Just to say, you were righter than you were wronger originally, as ordinary default zip files would indeed be completely sent, as the compression would make their entire content different even for the smallest changes.

It’s the choice to use uncompressed, available both in zip utilities and programming libraries that solves the puzzle – and can be easily tried out.

Agree the proprieters are dependably pretty complete and cogent in their thinking – just that angles do turn up, and make life interesting :wink:.

Cheers, robertdguthrie, and cheers to any ‘gruff and quirky’,
Clive