Scrivener Collaborative Writing and Version Control

Scrivener
1

This usage scenario describes how I do group writing with Scrivener.

I now use Subversion (an open source version control system) and svk (an alternate set of client tools to the normal svn tools for subversion) to manage my Scrivener documents. All this trouble gains me two things:

  1. Collaboration: I can now collaboratively edit large Scrivener documents with multiple collaborating authors. As the changes come in, I run something like Word’s “Merge Documents” to resolve the additions, deletions, etc.

  2. Version Control: I now have a true version-control history, and can selectively roll back, change, or restore individual parts in a more fine-grained way than snapshots currently allow.

Background

At first, the idea of sticking Scrivener documents in a source control system like CVS or Subversion seemed obvious and I assumed it would be easy. However there is a major headache - the way that Scrivener stores text automatically deletes the version control information (hidden .cvs or .svn folders) from the package every time a document is changed. I experimented with several solutions (search the forum Technical Support threads for “subversion”) but they were awkward, labor intensive, and you had to be careful or your document would become “broken.”

One solution it turns out is to use svk - an alternate set of client tools for checking out working copies that hide their version information in a safe place (~/.svk, rather than ~/Documents/MyNovel.scriv/.svn). There are some drawbacks to svk, the biggest being that there are currently no pretty OS X GUI tools to use it - you have to type on the Terminal command line. However, it works great.

If anyone is interested in replicating this setup, I’ll be happy to follow up with more detailed instructions. Just understand that following this workflow process requires that you have no fear of the command line: you install the open source subversion and svk client tools into your system, add a Scrivener document into a subversion repository (either locally or on a remote server) and then use the command line to check out a ‘working copy’ of the document using svk. Many people can make edits to this document, and they can be checked back in, or ‘committed’ using svk, merging changes at that time.

Example: Editing by Email

Each collaborators can install svk on their own system if they want to quickly sync the latest changes - or they can simply follow a slower schedule of emailing their revised Scrivener documents back and forth to the primary author, who manages the merging and emailing out new merged documents. This is an especially nice option for working with editors etc. Say I have a document:

~/novel.scriv

I’ve just finished revising it, and I save my working copy using svk. It is revision 10 (like a snapshot). I decide I want send some copies out for comments, so I use svk to ‘check out’ a couple versions of the same document for my co-author Frank and our editor Shirley.

~/frank/novel.scriv
~/shirl/novel.scriv

I then email those off and get back to writing. A few days later, I’m already at revision 15, I get an edited document back from Shirley in an email - she’s edited a dozen documents and added several with comments. I copy her .scriv file into the /shirl/ directory and then use svk to merge her changes. As it tries to figure out how to combine them, svk knows that Shirley was working from revision 10, and it knows that I’m now on 15. When done merging, svk creates revision 16, including all of Shirley’s suggestions. I go ahead and use svk to remove /shirl/ as she isn’t working on it anymore, and get back to work. Two weeks later, when Frank’s additions arrive by email, I’m at revision 19 - but svk remembers that Frank was responding to version 10, and it can help me merge our changes into revision 20. After seeing it, Frank has some questions about Shirley’s input - no problem, I can export a copy of revision 16 to show him exactly how it looked Shirley’s feedback was added into the document, etc…

More Info

SVK homepage: svk.bestpractical.com/view/HomePage
The SVK book: svkbook.elixus.org/
SVK and compatible Subversion binaries for OS X: homepage.mac.com/hiirem/svkbuilds.html

2

This is really clear but I fear that I’ll need a day off to work it out :laughing: or rather, for me to make it work …

btw I’d sack colleagues who took so long to send their comments back :unamused:

3

Jeremy
this looks very promising! Can you describe in more detail how to set up everything? I for one would be very interested and I am sure others are, too. Command line fear not a problem here.

I am taking the liberty to continuing your numbering scheme even though I may just be covering problems that nobody else is trying to solve.

  1. How could pure textfiles be incorporated into this workflow? I collaborate with Windows users (and other non-Scrivener users) on a regular basis and need something this flexible. Converting into plain text is the most comfortable for me. MultiMarkDown looks very promising indeed (plain text format!) but due to 3 and 4 I am not fully committed to actually taking the plunge.

  2. Oh, and if you could advise me how to incorporate FileMerge or some other tool to highlight changes into this workflow…that would be my writer’s nirvana. No more MS Word.

No worries, I a just dreaming. But if you could get me on the tracks with regard to solving your problems 1 and 2 I’d appreciate this very much.

Prion

4

I also use Subversion extensively and would LOVE to see it compatible with Scrivener, but in the meantime can you post the instructions on how to install SVK on top of an existing Subversion installation, if you have it handy?

Thanks!

5

Perhaps this is a naive question, but how do you keep the various auxilliary files in the project in sync–especially BinderStrings.xml, since that generally contains (and has to contain(?)) the text version of the RTF files and other project text?

Bryce

6

Sorry I’ve been AWOL from this thread for a bit.

cyberbryce re:How to resolve binder edits?:

If there were binder edits, you merge the binder files at commit time. This means going through and resolving changes, just like using Microsoft Word to merge changes. I’m particularly fond of using FileMerge (available free with Apple Developer Tools) to do this, but everyone has their own preferred setup. It feels weird to edit a private XML file, but in practice it works fine, although it is a serious disincentive to parallel editing. For a more serious problem, see my brief discussion at the end of the comment literatureandlatte.com/forum … =8176#8176 - “Problem: MyDocument.scriv/binder.scrivproj is binary”… hey, nobody said this would be easy.

7

agent99 and Prion re:SVK installation

Michael L.H. Brouwer maintains OS X binaries.

There are at this time no GUI tools for svk - either OS X or other - but there is an [svk Textmate bundle](svk.bestpractical.com/view/SVKBo … nistration).

If you do need to stick a Scrivener file directly in svn for some reason, TAR (don’t zip!) it - you lose all fine grained control benefits, but it works. If you want fine-grained control, however, commit via svk, and svk will manage getting it in and out of svn in a clean state.

Setting up SVN+SVK+Scrivener

During this process, do not run Scrivener such that it touches one of the docs you are attempting to put under version control. Leave it off.

  1. If you already have svn client tools installed on your local machine, you probably need to uninstall them. This will NOT delete your repos or working copies, and we will reinstall them - but we specifically need the version of svn client tools with perl bindings so that svk can talk to svn.

  2. Reinstall svn client tools with perl bindings (homepage.mac.com/hiirem/svkbuilds.html) as well as the additional and separate svk client tools (which use those perl bindings).

Note: If you have svn tools like svnX etc., after this process check to make sure they are pointing at your new svn binaries: probably in usr/local/svn/bin rather than usr/local/bin

Note: One concern I had was that the version of subversion with perl bindings was 1.4.2, whereas the version I was using before was 1.4.3. JUST IN CASE, I created exported versions of my repositories in ~/subversion/rebuild/…, but it turns out that was totally unnecc, as according to the docs the svn repo format didn’t change at all between 1.4.2 and 1.4.3.

  1. If you don’t already have a repository that you intend to keep your scrivener doc(s) in, create one. You can create one either locally or remotely - for this example, I’ll be creating a local repository.

$ cd ~
$ mkdir ~/subversion
$ mkdir ~/subversion/repositories
$ svnadmin create /Users/me/subversion/repositories/scrivenerdocs

  1. Check out a working copy from the server repository to any place on your hard drive - for example, a special section of your user folder. In Terminal:

$ mkdir ~/subversion/checkouts
$ mkdir ~/subversion/checkouts/svn
$ svn co file:///Users/me/subversion/repositories/scrivenerdocs ~/subversion/checkouts/svn/scrivenerdocs

  1. Add a Scrivener doc to your working copy and mark it as added. In Terminal:

$ cp -r ~/Documents/MyDocument.scriv ~/subversion/checkouts/svn/scrivenerdocs/
$ svn add ~/subversion/checkouts/svn/scrivenerdocs/MyDocument.scriv

You can add as many Scrivener docs as you like right away. Note that from now on, the .scriv file in /checkouts will be your active version - the original one in /Documents you can now delete (or archive).

  1. Upload the file to your repository

$ svn commit ~/subversion/checkouts/svn/scrivenerdocs -m “Initial import of my very first Scrivener file”

  1. Delete the svn working copy, and create a clean working copy with svk

This creates a copy of the entire scrivenerdocs repository that svk stores internally in ~/.svk

$ rm -r ~/subversion/checkouts/svn/scrivenerdocs
$ svk depotmap --init
$ svk mirror file:///Users/me/subversion/repositories/scrivenerdocs //mirror/scrivenerdocs
$ svk sync //mirror/scrivenerdocs

  1. Create a clean working copy that won’t break when you edit it.

$ mkdir ~/subversion/checkouts/svk
$ svk co //mirror/scrivenerdocs ~/subversion/checkouts/svk/scrivenerdocs

At this point you can edit the Scrivener doc in …svk/scrivenerdocs to your heart’s content, and use svk co and svk commit whenever you like. Since svk keeps its data in a depot (rather than the working copy) it is much friendlier to Scrivener packages (and RTFD files, and Inform compiles, and… and… etc.). Using svk commit automatically mirrors changes back up out of the ~/.svk depot and to your original svn repository.

8

Hmm, could you elaborate on this? I have no clue how to use Subversion and don’t have time to learn right now (although if you can give me a less-than-10-step intro to testing its use with a Scrivener project so that I can test what you mean by the above, that would be very useful). Where are the .svn folders saved? Inside the .scriv file, or inside the .rtfd files? If the former, I am surprised that Scrivener deletes it - in fact, I wouldn’t think so. If the latter, then that is nothing to do with me. RTFD files are RTF directory files and are entirely the domain of Apple code…

All the best,
Keith

9

Right, that’s part of it, but not exactly what I was getting at.

It seems like you could do damage to your project if binderstrings.xml is not in sync with the text in the RTF files. For example, based on a bug report that someone once submitted in the forums (and my subsequent peeking at how they are stored), I think that “Scrivener links” for example may require that binderstrings.xml is an exact replica (at the character level) of the plain text in every RTF document. This would imply that merging collaborative efforts entails consistent updates (and possibly decisions in case collisions emerge) for both the RTF files and for binderstrings.xml. Morever, you’d have to ensure that binderstrings.xml remained consistent with the other files down to the character level.

[Edit: my memory is returning – I think this may be complicated by the fact that Scrivener links are (or were at some point?) tied to exact character positions in the text, so I’d add that if you merge two different collaborators edits upstream from the Scrivener link, then you’d have to re-calculate its position. I hope, for your sake, I’ve imagined all of this… :wink: ]

Now, perhaps you could ask your collaborators never to use Scrivener links and ask them to never (unilaterally) change the structure of the project in the binder. Still, there may be other things besides Scrivener links which require that the text in binderstrings.xml matches those in the various RTF files. Again, maybe this isn’t as big a problem as it sounds, but you’d want to rule out these issues before relying on SVN for collaborative editing within Scrivener.

If you simply wanted more control over versioning your project than the snapshots system offers, then I suppose these problems wouldn’t affect you. SVN will just dutifully update all of the files.

Bryce

10

Keith, the OS X RTF(D) libraries are what is breaking svn.

The situation is that the svn version control system keeps .svn folders in every directory under its control - that means inside the .scriv package, in particular in the .rtfd folders. Then, any time an .rtfd file is edited, it automatically resaves with no .svn folder, deleting the version control information and ‘breaking’ svn.

If you wanted to take this on, there is a published workaround that we discussed over in another thread ~Feb 23:

literatureandlatte.com/forum … =8320#8320

It is possible to fix, however to be honest I don’t consider this to be your problem - I consider this to be Apple’s problem. Right now, Scrivener breaks svn for the same reason that TextEdit does.

11

Ok, I take part of it back, I imagined part of it. I have no idea what the copy of the text in binderstrings.xml is used for, but it didn’t seem to break the project (or the Scrivener links) when I forced that text to be out of sync with the RTF.

However, w/r/t the comment I made about Scrivener links requiring an exact character position in the text, that seems to rule out their use in projects where you collaborate, no?

Bryce

12

Thanks Bryce, I didn’t know about the scrivener-links-by-character-positions issue, as I don’t use the Scrivener links feature.

I think there are three main points here:

  1. If you need parallel editing collaboration, then you have to merge the xml. Merging a project XML file is just like merging source code - it is a nervous and potentially brittle process that, unlike editing text, has low tolerance for mistakes. If you merge your binderstrings, get it right! (and Scrivener links may not work even if you do it right - caveat emptor)

BUT

  1. If you need version control, no problem - all these scary issues only apply to the collaborative editing case

YET

  1. Reverting individual files even as a sole user could likewise create binder sync problems with scrivener links / full text search etc.

STILL

In practice, my experience is that the full text is not that brittle - it is probably out of sync in the binder briefly, but then gets rebuilt next time I touch the file. Binder ordering is the showstopper.

If more people start using Scrivener under source control, we may have more data on what can go wrong.

13

For the record, binderStrings.xml is used to store the synopsis strings, and to provide the strings for searching - so that neither the search field inside Scrivener nor Spotlight need to open up every RTFD file just to do a quick search.

Regarding the published solution at svn.haxx.se/dev/archive-2003-02/att-1321/fixsvn, I have difficult reading Python code, so if anyone can say in plain text exactly what that is doing, I would be grateful. E.g. Is it something like this?

  1. Upon save to RTFD, peek inside the RTFD directory to see if there is an .svn folder (or .cvs or something?).
  2. If so, move the folder elsewhere (or copy it in memory).
  3. Do the RTFD save as normal.
  4. Copy the folder back.

If it’s something like this, I could probably enhance the RTFD saving method to include the above steps. If I have misunderstood, please provide your own simple workaround steps that would be required that I could (possibly - no promises) translate into code.

Of course, the binder is much more problematic, given that it is currently written out in binary form. I don’t really see any way around that, as that is generally the way that custom classes such as are used by Scrivener to hold info get written to file. And should the file format change, that is a lot of projects out there that would need converting…

Best,
Keith

14

Keith, your steps 1-4 are correct - caching the folder and then restoring it after save is precisely the workaround. The two folder names to check for are “.svn” for Subversion and “.cvs” for CVS.

I concur on the binder binary - its a can of worms. The above fix on its own however gets you full svn support for single users, including per-document version control - it just means that binder updates still function like snapshots and that collaboration requires great care.

15

Keith, I don’t think that the file format would have to change. For those who aren’t faint-of-heart: A script could convert binder.scrivproj to XML, prior to versioning, using “plutil”:

plutil -convert xml1 binder.scrivproj

Then it could be converted back:

plutil -convert binary1 binder.scrivproj

I just checked, and it seems that Scrivener will still open the project without a problem even if it were left in XML format (even if you didn’t convert the file back). I have used a similar strategy to version other kinds of files that are stored in such a format for my own personal use. If it’s correct that it doesn’t matter in what form such files are stored, then I’ve got to say: those wizards at Apple just make these things smart, don’t they?

However, I imagine the bigger problem for collaborating is logistical, i.e. how to actually handle conflicts should they arise. The binder.scrivproj file looks pretty darn complicated… :frowning:

Bryce

16

My writing partner and I have been doing a simple form of collaboration: I save my daily work on a Scrivener file in a backup ZIP, post it to her via e-mail, she opens and makes her revisions and returns it to me. We are able to avoid overlapping revision by working in different time zones, and times of day. So far, no confusion because each ZIP file has a date and time stamp in the file name. I realize this won’t work for everyone, least of all writers who want near-real-time file sharing.

17

Regarding the binder.scrivproj file, if you ctrl-click on it and open it in, say, TextWrangler, it should come up as an XML file. A very weird XML file, yes, but XML nonetheless. Does this not happen? Can Subversion not deal with this?

Best,
Keith

18

Hi Keith,

No, as far as I can tell (unless TextWrangler is smarter than my text editor) binder.scrivproj will be XML only if you run “plutil” on it first like I described above. Otherwise, Scrivener always writes it as a binary file, by default. Moreover, if you convert it to XML, and then open the project, Scrivener will write it out as a binary file immediately upon any modification to the project.

Yes, if you convert it to XML prior to committing, subversion will then deal with it nicely, only saving differences with each version.

Bryce

19

Ah… Turns out TextWrangler is smarter. TextWranger apparently converts binary files to XML upon opening if it thinks it can.

Anyway, it turns out that it is more straightforward than I had thought to convert the .binderscriv file format to XML. So as of 1.04, binder.scrivproj will be an XML file (it will look just as complicated as the one you saw by converting via plutil, mind). I will probably have to convert the snapshots files in the same way. It will mean that pre-1.04 files will need converting, but Scrivener will handle that automatically.

Then it is just a matter of hacking the RTFD/.svn/.cvs stuff. So, before I go down this path, please clarify the following:

If I make these changes:

  1. Have binder.scrivproj save in XML format (no matter how unreadable it will be to humans);
  2. Convert snapshot files to XML format (same as binder.scrivproj);
  3. Grab .svn or .cvs folders inside RTFD packages before saving RTFD files and replace them after the save;

will Scrivener then be compatible with Subversion? Will that be enough?

Thanks!
Keith

20

Keith - I haven’t played with SVN or CVS enough recently to definitely confirm your query, so I will leave that to someone else.

But, I will say that making Scrivener compatible with these version control systems will be a HUGE bonus.

I’ve toyed with making a drag and drop program to make setting up something like this easier, but never got around to really sketching out the requirements. But being able to collaborate on files like this would be phenomenal…

Thanks for considering this!

Fletcher