Scrivener Collaborative Writing and Version Control

[cyberbryce]

Just a disclaimer: I don’t currently use SVN with my Scrivener packages, nor do I aspire to in the future.

@KB

Anyway, it turns out that it is more straightforward than I had thought to convert the .binderscriv file format to XML. So as of 1.04, binder.scrivproj will be an XML file (it will look just as complicated as the one you saw by converting via plutil, mind). I will probably have to convert the snapshots files in the same way. It will mean that pre-1.04 files will need converting, but Scrivener will handle that automatically.

Might such a change impact performance?

developer.apple.com/documentatio … lInfo.html

If performance is a concern, one workaround would be to have Scrivener write projects as XML only just before the application is quit. Similarly, the use of the XML format could be optional.

Bryce

[AmberV]

I agree, speed testing should be done, and if it does cause a slowdown to occur, some form of version tracking compatibility option should be provided, with a warning that you might want to turn auto-save down (or off). With a normal program it might be an acceptable performance loss, but not with one that saves itself every 2 seconds that you pause…

Testing will tell. I wouldn’t be surprised if impact is minimal.

I really like this idea. I’ve been wanting to set up a repository of my projects, located on server that is running Subversion. Additionally, secure text-only retention of Snapshots would be a welcome change.

[cyberbryce]

@AmberV

I agree, speed testing should be done, and if it does cause a slowdown to occur, some form of version tracking compatibility option should be provided, with a warning that you might want to turn auto-save down (or off). With a normal program it might be an acceptable performance loss, but not with one that saves itself every 2 seconds that you pause…

If people only version with the document closed, then you could limit saving in XML to occur upon closing the document – which is what I meant above but stated rather poorly.

Bryce

[KB]

You’re right - I hadn’t thought about the performance hit. I will look into whether making it optional is a possibility. It should work fine.

Snapshots can be safely saved as XML without worrying about any performance hit, given that they are only saved very occasionally.

Now, if someone can answer my questions about whether this would be enough to support Subversion, that would be great.

Best,
Keith

[KB]

@KB

If I make these changes:

1. Have binder.scrivproj save in XML format (no matter how unreadable it will be to humans);
2. Convert snapshot files to XML format (same as binder.scrivproj);
3. Grab .svn or .cvs folders inside RTFD packages before saving RTFD files and replace them after the save;

will Scrivener then be compatible with Subversion? Will that be enough?

Tumbleweed drifts across stage

Anyone? I have this flagged as my work for today, but I need some clarification before I waste my time…

Best,
Keith

[fletcher]

In an effort to TRY and answer your question, I dug around some Scrivener files and from what I see this appears to be true.

Obviously, any images or other multimedia will be binary, but these files won’t be edited so much as replaced, so this should be ok.

As long as Scrivener “respects” .svn or .cvs directories inside all folders (the main .scriv folder, snapshots, all of the RTFD’s, etc then this should work.

I’ve done a bit with SVN and CVS, but just not with Scrivener. They’re really pretty straightforward programs, they just don’t work well when those data directories get removed unexpectedly…

Fletcher

[peterdcox]

Pour les encouragement des autres … etc

I am REALLY keen for a good collaborative model, so welcome everyones’ efforts to achieve this. As for Subversion - did a quick internet look and ran away … looks geeky to me … Size of files is an issue 50 meg for a current project even after removing all the media files … So Keith, a solution would not be wasting your time, sorry I have nothing to contribute except expectation

[KB]

At this point in time, Scrivener itself will not offer any collaborative writing solutions. Right now I’m just trying to ensure that Scrivener is compatible with other solutions, such as Subversion and CVS.

This is what I have so far:

1. In 1.04, there is now a checkbox in the Preferences entitled, “Enable Subversion/CVS-compatible saving (slower)”. If this is checked, binder.scrivproj is saved in Apple’s XML/.plist format - that is, as plain text. Note however, that just because it is plain-text, that does not make it human readable. It contains a lot of data about internal Scrivener classes, so editing it manually would be as fatal as trying to edit the binary data - your project will be ruined. However, this should (as I understand it) at least allow Subversion to track changes to the file.

2. I have also implemented a workaround for saving RTFD files so that .svn or .cvs subdirectories within the RTFD wrapper are preserved - I hope. Initial tests seem to indicate that it is working (using Terminal and following Jeremy’s instructions in that other post). See below for the geeky implementation if you are familiar with Cocoa.

3. The next step is making snapshots plain text in the same way as binder.scrivproj. These will be plain text no matter what (the preference option won’t affect them) - given that they are only saved very occasionally, speed isn’t an issue as it is with binder.scrivproj.

Because of these changes, when you open 1.04 for the first time, you will be told that your project needs updating. It is a very minor change, though.

Right, on with step 3…

All the best,
Keith

Geeky Cocoa Coding Bit

For anyone who understands Cocoa, this is my workaround for preserving .svn and .cvs directories within RTFD packages:

The Cocoa class for creating and writing packages to file is NSFileWrapper. You normally create and write an RTFD file to disk like this:

NSFileWrapper *rtfdWrapper = [[textView textStorage] RTFDFileWrapperFromRange:all documentAttributes:nil];

[rtfdWrapper writeToFile:path atomically:YES updateFilenames:YES];

Thus, I have just created a category on NSFileWrapper which provides an alternative method for writing a file wrapper to file, as follows:

@interface NSFileWrapper (SVNAdditions)

- (BOOL)writeToFile:(NSString *)path atomically:(BOOL)atomicFlag updateFilenames:(BOOL)updateNamesFlag preserveSVN:(BOOL)preserveSVNFlag;

@end

@implementation NSFileWrapper (SVNAdditions)

- (BOOL)writeToFile:(NSString *)path atomically:(BOOL)atomicFlag updateFilenames:(BOOL)updateNamesFlag preserveSVN:(BOOL)preserveSVNFlag
{
	if (preserveSVNFlag == NO)
		return [self writeToFile:path atomically:atomicFlag updateFilenames:updateNamesFlag];
	
	NSFileManager *fileManager = [NSFileManager defaultManager];
	
	NSString *svnOrCvsPath = [path stringByAppendingPathComponent:@".svn"];
	
	// Check to see if the wrapper contains an .svn directory. If not, check to see if it contains a .cvs directory.
	if ([fileManager fileExistsAtPath:svnOrCvsPath] == NO)
	{
		svnOrCvsPath = [path stringByAppendingPathComponent:@".cvs"];
		
		if ([fileManager fileExistsAtPath:svnOrCvsPath] == NO)
			svnOrCvsPath = nil;
	}
	
	// If the wrapper contained an .svn or .cvs directory, cache it.
	NSFileWrapper *svnCache = (svnOrCvsPath != nil ? [[NSFileWrapper alloc] initWithPath:svnOrCvsPath] : nil);
	
	BOOL success = [self writeToFile:path atomically:atomicFlag updateFilenames:updateNamesFlag];

	// If we cached an .svn or .cvs directory, restore it now.
	if (svnCache != nil && svnOrCvsPath != nil)
	{
		[svnCache writeToFile:svnOrCvsPath atomically:YES updateFilenames:NO];
		[svnCache release];
	}
	
	return success;
}

@end

Using this category, you can create and write an RTFD file like this:

NSFileWrapper *rtfdWrapper = [[textView textStorage] RTFDFileWrapperFromRange:all documentAttributes:nil];

[rtfdWrapper writeToFile:path atomically:YES updateFilenames:YES preserveSVN:YES];

[Maria]

@KB

…
For anyone who understands Cocoa, this is my workaround for preserving .svn and .cvs directories within RTFD packages:

The Cocoa class for creating and writing packages to file is NSFileWrapper. You normally create and write an RTFD file to disk like this:…

Keith,

Thanks for sharing your codes, on the blog as well.

One more suggestion (in fact, an older suggestion): If you start considering making differences in saving XML (binary during runtime, text when closing/actively saving), it might be a good idea to start considering packing the RTF files in a project as one zip file when closing / actively saving a project. It saves lots of space and time while moving the packages.

All the best,
Maria

[fletcher]

@Maria

One more suggestion (in fact, an older suggestion): If you start considering making differences in saving XML (binary during runtime, text when closing/actively saving), it might be a good idea to start considering packing the RTF files in a project as one zip file when closing / actively saving a project. It saves lots of space and time while moving the packages.

But wouldn’t that defeat the purpose of converting the plists to XML? The main idea is that text files are more amenable to version control and collaboration systems. A zipfile is binary, and therefore not readily amenable to CVS, SVN, etc.

That seems like it would be a step backwards, or am I missing something?

[Maria]

@fletcher

…
But wouldn’t that defeat the purpose of converting the plists to XML? The main idea is that text files are more amenable to version control and collaboration systems. A zipfile is binary, and therefore not readily amenable to CVS, SVN, etc.

That seems like it would be a step backwards, or am I missing something?

Fletcher,

if we are of different opinions, then it is most likely that I am missing something .

I see that OpenDoc uses zip files, and so I suppose that CVS etc. are doable, with opening the zip files at the beginning of a session. I suppose so because these guys are most likely to include such features into their products.

But this is guess work, please correct me if I am wrong.

Best,
Maria

*** a bit aside: ***
As a whole, I don’t think that CVS or Subversion is of major importance in a writing application, just because it is too technological. Only few people who write fiction or work in the humanities would set up a system like this and really use it. But this is only my impression: Version control and collaboration tools are helpful, but writers need it set up in a non-technological way. As for natural sciences, all I know use Linux and TeX exclusively for their writing, no Mac, no Scrivener.

---

[fletcher]

@Maria

if we are of different opinions, then it is most likely that I am missing something

Not necessarily…

@Maria

*** a bit aside: ***
As a whole, I don’t think that CVS or Subversion is of major importance in a writing application, just because it is too technological. Only few people who write fiction or work in the humanities would set up a system like this and really use it. But this is only my impression: Version control and collaboration tools are helpful, but writers need it set up in a non-technological way. As for natural sciences, all I know use Linux and TeX exclusively for their writing, no Mac, no Scrivener.

---

It wasn’t that long ago that you could argue that writers didn’t know what a bulletin board was… When something becomes easy to use, the number of people who wonder how they ever got along without it grows.

There are a growing number of web sites and applications geared towards collaboration and version control (which are really different sides of the same problem). It never hurts to be forward thinking, and anticipate future uses for your product (e.g. Scrivener).

There is obviously a non-zero number of Scrivener users who have gone to great lengths to try and get version control working. All it takes is one of them to create an easy to use tool to enable version control/collaboration, and that solution can be shared by everyone. Programmers are used to version control systems, because it is nearly impossible to create massive projects without them. I suspect that when the tools are a bit easier to use, writers of all sorts will come to rely on them just as much.

This could end up being one of Scrivener’s “killer” features down the road for a LOT of users, if done correctly. I for one am willing to contribute to creating a tool that could make these features easy to use, either as a stand-alone utility, or as something that could eventually even be embedded in Scrivener (with minimal effort by you, Keith!)

On a more technical side, having to unzip a file before running it through a version control system brings up a bunch of problems, and I am not aware of any systems that do this automatically (though I am by no means an expert).

[Maria]

@fletcher

…
There are a growing number of web sites and applications geared towards collaboration and version control (which are really different sides of the same problem). It never hurts to be forward thinking, and anticipate future uses for your product (e.g. Scrivener).
…

This could end up being one of Scrivener’s “killer” features down the road for a LOT of users, if done correctly. I for one am willing to contribute to creating a tool that could make these features easy to use, either as a stand-alone utility, or as something that could eventually even be embedded in Scrivener (with minimal effort by you, Keith!)

On a more technical side, having to unzip a file before running it through a version control system brings up a bunch of problems, and I am not aware of any systems that do this automatically (though I am by no means an expert).

(1) You are right, it is no reason not to start something new just because nobody uses it yet.
(2) I would be one of those happily using your user friendly implementation of version control and collaborative features. Like I do with your MMD, thanks by the way!
(3) This of course, I cannot judge. If you say so, it must be difficult.

I speak for myself if I say that I use Scrivener for my personal work. I would like to have a message about which file in my project was dealt with last time and when this last time was, when I open a project (asked for it somewhere else, waiting for comments) because it makes keeping control over projects on 2 computers easier. This is all I need for my purposes. I would rather prefer zipped project documents, since they are about one third of the file size now and quick and easy to copy. This is what I need most often: space and quick copying with a little help on which is the actual file.

So if I had to choose, I would give away CVS, but I would like to have both Other ideas?

Best,
Maria

[fletcher]

@Maria

I speak for myself if I say that I use Scrivener for my personal work. I would like to have a message about which file in my project was dealt with last time and when this last time was, when I open a project (asked for it somewhere else, waiting for comments) because it makes keeping control over projects on 2 computers easier. This is all I need for my purposes. I would rather prefer zipped project documents, since they are about one third of the file size now and quick and easy to copy. This is what I need most often: space and quick copying with a little help on which is the actual file.

Let me see if I understand properly. Are you working on the same project on two different computers, and manually copying certain portions back and forth to keep the two copies in sync?

If so, then this is a perfect example of when a version control system would make your life easy. You can think of it as iSync for your document (basically, iSync is a version control system of sorts, minus a bunch of features). Whichever computer you are on, you hit a button, and sync your work to the other computer automatically.

Or I could have totally misunderstood what you are doing.

As for copying speeds, are you copying over a network? Most of the file copy protocols have built in compression, so it really shouldn’t matter whether a file is zipped or not, it is effectively zipped before transfer. I don’t know for sure whether Apple’s File Sharing has this or not, but I would suspect it does.

[Maria]

Fletcher,

thanks for your advise. My workflow is/was most simple:

I used to copy those files I worked with between iBook and Desktop with a USB stick.

Something aside: I could use our LAN, but there are Windows computers mixed which always start stopping their work when something is changed in the fragile setup. I am fed up repairing Windows machines with 400 character explanations in their set up wizards with badly displayed Japanese fonts, so I came to the conclusion to use the LAN only for my Internet and repair Windows machines in those rare cases when they have to be restarted after sudden shutdowns, which have become rare with XP, thank God.

The old method with copying was so insecure that I now have my actual projects and files on a thumb drive, which works fine. I zip backup on both machines before I take the thumbdrive from the actual machine to work on the other.

As for copying: There is a significant difference in speed even on my G5 hard disk between copying a Scrivener project and a zipped file of the project.

It is interesting to hear that files will be compressed when copying via LAN, so a CVS solution sounds really great. I understand that my problems are very specific, so considering a CVS for the normal user seems to be the better solution.

Long post, sorry, but you convinced me.
Thanks for taking your time,
Maria

[AmberV]

One very quick thought on transfer speeds: Most systems only transfer what is required to bring the target system up to date. In this way, massive projects (consider developers working on the Firefox browser, OpenOffice, or the Gnome desktop environment), can be kept synchronised without waiting for the entire current project to transfer.

Typically modern versioning systems work at a granularity that is even finer than the file level. So we are not even talking about entire modified RTFD files getting transferred, but just the instructions on what to change on the target machine to make the two files match. In fact, it is these delta fragments that a system like SVN actually stores. In a way, it is similar to the method an MPEG movie is compressed. A base frame is calculated, and then for the next pre-specified number of frames after that, only the changing or moving data in the picture is considered for storage.

You only need to transfer the entire project once, to a new machine, or between collaborators. After that point, everyone just shares what amounts to complex change logs. Putting everything into a zip file would accomplish nothing, save but for reduce hard drive space and increase processing time.

Yes, this stuff is all pretty geeky right now, but I’m with Fletcher on this one. That is something that will change. Apple’s Time Machine technology is the first step toward a world where people no longer view data in a two dimensional way.

[dafu]

[Warning: Extremely peeved system administrator speaking after a second 6 hour 250M multi-personal-computer accounting database repair has failed]

Buy a mainframe.

If you want seamless version control, have one version on your spiffy 360/70 with byte-level locking and all the stuff they figured out 20 years ago.

[End SA irritation]

Dave

[Maria]

That is all interesting.

AmberV, thanks for further explaining the benefits of such a set up. They are close to what I set up earlier with Chronosynch, although this software does not work below file level, but copies the whole file if necessary instead. What drove me away from the solution was the fact that it takes some time for Chronosynch (and I am sure, for a geeky version control system as well) to check the thousands of Scrivener files for whether or not they have to be updated.

Dafu’s comment was more of the kind that reflects my experiences: These systems work in theory and if all parts have the same level of knowledge and passion. I am more on the KISS side, thus simple and stupid copying of quickly copied zip files – without caring for set up and maintenance of the network.

This does not mean that I am against trying to keep up with promising developments in technology, at least, developments now should keep in mind that these features may have to be integrated. AmberV and Fletcher were quite convincing with their comments. But I am sceptical about the immediate and successful implementation. As long as they are not Maria-proof (i.e. extremely easy to handle), I prefer a simple solution. For the time being, and this may be some years…

Anyway, this discussion is inspiring for me, and I appreciate your patience!

Maria

[KB]

As for zipping stuff up in the project directory: the trouble with this is things could get very messy. Years ago I wrote a small Windows app for a hobbyist games programming engine (yes, my geek credentials are out in the open now) which used a folder structure to store its data, with all of its resource files open for anyone to nab. A lot of folk hated this, so my app just zipped it all up and handled unzipping it to a temporary folder at runtime and then removing the temporary files when the game ended. But this is exactly the trouble - you end up with files everywhere.

I would hate doing something like this for Scrivener. When you click on a document in the binder, Scrivener gets its internal ID number from the binder information (stored in binder.scrivproj) and then looks in a dictionary of open documents for that ID number. If the ID number is there, it displays the text associated with it. If it’s not there, it looks on disk inside the project directory for the ID number + .rtfd. If it exists, it loads it into the dictionary of open texts and opens it; if not, it creates a blank text.

If things were zipped up internally, every time Scrivener wanted to open a document, it would have to unzip it - which would be slower, for a start. And upon saving, it would have to replace the file in the zip with the file that has been unzipped somewhere - presumably to a temporary folder. Or, all files would have to be unzipped on opening the project and zipped back up on quitting. There are lots of circumstances in which these files could end up not getting destroyed. Or, if things went wrong - a crash, for instance - the file might not get restored to the zip and thus not saved properly.

Moreover, the main reason for the files being open within the RTFD package is for safety. If your .zip file became corrupted somehow - unlikely, but possible - your project would be lost to you, so things wouldn’t be much better than having a proprietary file format.

As for version control - I have put in place all the steps I mentioned for 1.04, so hopefully Scrivener will be compatible with Subversion and CVS from now on. Any proper version control within Scrivener, however, is well out of the scope for 1.x.

Thanks and all the best,
Keith

[jeremydouglass]

Tumbleweed drifts across stage

(Sorry I’ve been gone! - I’ve set watch on this thread three times, but for some reason I no longer receive reply notifications. Shoot.)

A delayed response regarding those three changes:

Yes, the changes described should make Scrivener compatible with Subversion, CVS, and almost any other version control system you care to name. In fact, this makes Scrivener something of a version control model citizen for all the other multi-RTF(d) cocoa apps out there - almost all your journalware and authorware.

If you would like me to do testing for you with SVN etc. for the next release, just let me know.

(Probably best to do it by email - for some reason I’m not getting notified on this thread.)