Backup Advice

Hello folks,
I have been happily using the windows beta for scrivener, which is working well for me, but I have run into a challenge I’m hoping somebody can help me with. I work off of a desk top and laptop computer, both of which I keep synchronized using a service called “Syncplicity,” which is similar to Dropbox, but not exactly the same. Syncplicity keeps all my files up to date on a server, and all installed computers, as I work, and it also keeps versions of file changes going way back. This helps with my writing because if I’m working on something and want to see a draft I created ten days earlier, I can go to the server, download the old draft, and read through it. If I decide to revert to something from the old draft, I cut and paste, adding it to my new draft. Even though I rarely use it, I love having this instant archive of all my thinking on a project, going back as far as I like.

Now for my problem: twice during my time with Scrivener, I have wanted to find a version of a paragraph that I wrote and discarded, and I haven’t been able to do it. If I navigate to my server, I can see versions of all my scrivener files going back for weeks, but the folder/file structure is byzantine and indecipherable to me-- somewhere in there, I know, there is a text file that has the words I’m searching for, but I don’t know what its name is, or how to find it. I tried downloading a couple .rtf files just to check, and ended up with a file that was one character wide (as if the left margin were 4 inches and the right margin were 4.4 inches, and my entire document was in a verticle column), hard to read, and not the one I was looking for. I could restore my entire scrivener folder from the server, but that would restore only the most recent version. This is great for a “my computer crashed or got stolen” scenario, but doesn’t help me when I already have the folder on my computer, and I’m trying to get at a previous version of a part of the draft that I wrote several days before.

The fact that I’m having so much trouble navigating/locating the text files that scrivener creates has me pretty concerned, both as if relates to me managing my project, but also as it relates to recovering data. Is there a simple way to access the text files I have generated in scrivener? I really don’t want to take snapshots every five minutes of everything I write. I really don’t want to back up my whole scrivener project every five minutes. I already have a very reliable backup/synchronization service taking regular snapshots of all my work, and the “previous versions” exist on the server. There should be some way to do this. Right? :confused:

A simple way of producing an item-by-item backup with clearly named files, arranged into folders (and even including meta-data options) is the File/Export/Files... menu command. This works by selection. So if you want to just back up your WIP, select the Draft folder in the Binder, use this command and select a location and folder name for the export. You can point it at your Syncplicity folder (or however it works) to keep them backed up on that service, if you want.

Of course, a more “Scrivener friendly” way of backing up is just File/Backup Project To... which will create a complete duplicate of the project (optionally zip archived) and datestamp it. The file export method is better if you anticipate losing Scrivener access, but if you don’t, slicing off your current project state in its complete form is by far the superior method. Since it produces a separate project, you can later open it up alongside the main project and restore the bits and bobs you need.

Have you looked at the “Snapshot” facility? Snapshot as currently implemented (in the Windows Beta, anyway) allows you freeze snapshots of the state of a file within a project, and to roll back changes to previous snapshots, keeping a list of all the snapshots so far which you can add to or delete as you like. You can make snapshots of individual folders and files, but not of the whole project. It’s worth playing with. I’m not sure what the point of making a snapshot of a folder is, since it’s not recursive. You can preview each snapshot in the right panel, but I don’t see any facility for searching through the snapshots. I’d like to be able, for instance, to visualize the entire snapshot tree for a project.

What you really need, though, is an integrated version control system, where you can open a named version, work on it, save snapshots of it, and then later navigate through the tree of snapshots looking for the material you want, and, ultimately, choose which snapshots to include in a compile, and define that as a Release. Perhaps I have two versions of a story, one rated R, another rated PG. Perhaps one market requires a certain format for the title page, and another asks for a different one. I should be able to use version control to do this. Ideally, it should be inside Scrivener, but it doesn’t have to be.

Ad-hoc solutions using Export, or worse yet, poking around in the project directory, are worse than useless, IMHO. Over time just remembering what is what will become a real problem. And generally asking a customer to stand on his head and spin to get something important like this to sorta-kinda work is a bad idea. When a customer tells you he’s poking around in the project files, you know you’re doing it wrong. He isn’t. The whole version-control thing is just too complex to approach that way. And please note: backup is not the same thing as version control. Version control is a kind of incremental backup in which meta-data about what has changed are saved along with the backup. It also typically provides ways to compare different versions side by side. There’s a lot more to it than that, of course, but simple backups aren’t going to touch the problem you’re having.

I think that a solution for Scrivener is to embed a real version control system, like CVS or a similar open-source, mature system, and implement the snapshot facility using it, rather than using the current linear version stack. The Mac version may already have something like that; the challenge of developing platform independent software is freeing the software from the comfortable tyranny of the original system and all the cool OS hooks. OS hooks are cool but not inherently maintainable. There are always other solutions for things like address books, configuration options, and version control. It’s not like these are new problems, are they?

Frankly, it seems that Scrivener is going at developing its Windows version the wrong way. I don’t mean this at all as a judgement of Lee’s work, and it’s a fabulous, ground-breaking application, but what should happen in my opinion is that Scrivener should develop version 3.0 in a platform independent way; re-factor the software in layers so that machine-dependent libraries are pushed down to the lowest level possible; define the semantics of the internal interfaces to those lower layers as cleanly as possible, taking into account the full book of use cases for both the latest Mac version and its planned successors. These are sound and well-understood software engineering principles. This means re-writing the whole project, sure, but in the end you’d have something that will run anywhere which will be easy to update. I shudder to think what the coming nightmare will be like for Lee and company when it comes to supporting a semi-kluged Windows version limping along trying to keep up with the Mac version, with the Linux version dragging along behind as a poor bastard stepchild. Never mind the iPad and Android versions… I may be completely off base about this and would be happy to be corrected; but I’ve developed software in both big and little environments, and quick-and-dirty has always been a very popular development methodology. I am by no means saying that is what L&L is doing, but certain kinds of problems do sorta point in that direction…

Back to the matter at hand…

If you as a user have a technical bent, you can use CVS yourself to do real version control, keep track of multiple versions, make notes about what the versions are, etc. There’s an open source Windows version here, and there are various other implementations around. You put your repository on your server; run the CVS client on each of your machines to check out the version you want to work on, save intermediate versions of, etc. But there is this caveat: it won’t ever be as good as version control inside Scrivener, because you’ll have to be dealing with the entire project folder as a unit, and won’t be able to drill down to individual files.

The other thing to note is that employing version control in your work will of necessity change the way you work. You will be thinking about versions, about backups, about remembering what you did, much more than when your work is freewheeling. And you’ll have to think not only of making backups in general of your work (either with the backup facility or by compiling versions) but you’ll need to backup the repository as well, because when you save a version you also save knowledge about what has changed and why the version was created that you’ll want to keep. So you need to accommodate all of this into your work process. And I’m sure that’s something that doesn’t appeal to everyone.

I hope this helps, or at least provides food for thought.

Nathan

Some people put notes directly into their folders, or even use folders to store introductory text to a section. Folders in Scrivener are not like normal folders. They are really just text files with a different icon and some coded in behaviour differences (like always preferring group view modes when you click on them, or alternate compile formatting rules). So if you use them to store text, Snapshots are useful for them as well.

Some of this can be accomplished with the new Front Matter feature, as yet to be developed for Windows. This lets you set up a folder outside of your Draft that is selected in the Contents portion of the compile phase. Any items stored in that folder will be prepended to the content list. Hence, you can create multiple font matter folders for different compile targets. This technique is demonstrated in some of the new fiction templates, where a submission has the typical cover page design with contact & agent info, word count, and title—but the e-book front matter folder has other material more suited to publishable output.

Those are going to be the areas that change the most when using multiple compile targets. A more widespread solution that uses Snapshots or something is probably going to be much less needed, while its an interesting concept, I don’t think it is necessary. Again, to cite a Mac feature, there is a way of doing this already, and in fact the Scrivener User Manual demonstrates it. The Draft folder has both the Windows and Macintosh manuals embedded within it, since about 75% of the articles are relevant to both. The 25% which represent divergent material are coded with Labels, and the compiler is set up to filter out one or the other to produce a platform specific manual. Filters can also source off of collections (dynamic or curated), so it would be possible to use a variety of different content sources if you need say, an abridged version of the book for the youth market. You can already kind of do that with the Windows compiler, which lets you pick a Collection as the compile group; though this feature is better for proofing as it drops hierarchy.

Both of these are features which will be coming to Windows in time; I just wanted to point out that advanced multi-publishing book control is something you’ll be able to do with greater ease than having multiple Draft folders or project files—and I would say, greater ease than CVS trunks.

On the development theories I don’t have much to say, except that Scrivener is a very small project. :slight_smile: Most of the stuff you are talking about would require a larger team, much larger in some cases. Just think of how many people are involved in some of these version control technologies, either closed or open source. They involve a lot of talent. You’d spend years working on architectures otherwise. I think “quick and dirty” is a bit harsh though. The Windows code base has been in development for, I think, about three years now, and the only reason it was delayed earlier this year was to polish and refine it. Spending ~ half a year on refinement for a 1.0 release is, I would say, above the call of duty for a guy with some contracted help. I might be biased, but I sure think it shows. This will be a 1.0 release that is far more considered, polished, and featured than many 3.0 iterations out there.

I’ve got no disagreement with the general thrust of your argument. Version control is good stuff, but its also kind of complicated to use, especially for the average author out there—it is out of reach complicated. Even with mature systems that have nice and tidy front-ends, we aren’t talking about basic computer usage stuff here. Most writers just want to get on with their writing, not learn how to architect their data flow, manage trunks, and optimise throughputs. :slight_smile: And most books are single-stream one-shot. So adding a lot of interface and feature burden to accommodate complex books with multiple streams is going to intimidate those that just need “a book” out of it. I think our current implementation with Front Matter and logical filtering rules is just about right. You can do an awful lot with it, and it isn’t too difficult to visualise something like “don’t use blue files for this run”.

If some form of versioning were to ever be implemented in the application itself, it would have to be as seamless and easy to use as Snapshots. If you start cutting into the simplicity of that, people will stop using it, and then you are back to square one with no control at all.

nathanzal, I think you’re being too harsh.

The idea of version control and installing version-control software hooks in Scrivener for the Mac platform was discussed at length in the forum a while ago. It would be worthwhile looking up the threads. Of course version control would be valuable for some users, but less so for others; I’m not sure I’d use it myself. If I remember correctly, an obstacle was the package nature of Mac Scrivener projects, presumably an obstacle for the Windows version as well. Presumably the developer rejected the idea at that stage because of the effort that would have had to be put in to make it viable, as set against the returns in terms of user demand and satisfaction. Bear in mind the size of L&L — and that the Windows version hasn’t yet even reached 1.0.

An all-singing, all-dancing platform-independent Scrivener 3.0 might be attractive, but it might not. If we’re talking Java, I suspect many Mac users wouldn’t be at all keen. And again, bear in mind the effort involved, set against the capacity of L&L. And “quick-and-dirty”? :confused: How long has Lee been working on the Windows version? And how long did Keith labour over the Mac version 2.0?

I bought a Mac to use Scrivener something like four years ago. I did so because having tried all the candidates I could find on Windows, from the likes of IdeaMason to Ultra-Recall to Word, I could find nothing capable and reliable for long-form writing. Even then Scrivener was pretty wonderful. And I think it’s even more wonderful now. I know of no sensible rivals. Of course, in the medium and long-term all sorts of improvements and enhancements would be desirable for users like me; one of the attractions of Scrivener is that one can see possible road-maps stretching off into the far distant future. But to say the least, isn’t it a bit premature to be saying “Start all over again”?

Dear Hugh et all,

[edited for increased calmness]

In response to various comments about the supposed harshness of my remarks I would like to point out that I said many things in that post, many of them glowing praise of the product. The problem came, apparently, with the term “quick and dirty,” which apparently some of you read as fightin’ words. That is not at all what I intended:

This is not saying that Scrivener is in any way Q&D. Please don’t put words in my mouth by jumping on that one phrase. I said I’m aware of the temptations, having developed software. I’m happy to learn more about the company and the fine work it’s doing, but I’m making a more complex point. My role here is to find and point out problems as I see them, not cheer-lead. There is no need to jump in and defend L&L. At this point I’m trusting all of my writing to this project; that surely means something.

Tell me I’m wrong, if you want, and tell me why, but please, give me the benefit of the doubt. We’re on the same team here. At least I hope so.

Thanks,

Nathan

First off Hugh, I did not say “Start all over again.” You did. Refactoring and defining interfaces for platform independence is not “starting over again” by any means. I was talking about long term software development issues well beyond the time horizon of Windows Scrivener 1.0. It’s never premature to think in those terms; it may even be too late, but that’s for L&L to figure out. I see from the “about” blurb that an iPad or iPod reduced-feature version is not out of the question, so it would seem that platform independence is relevant after all on at least some levels.

Best,

Nathan

Nathan, I do apologise, I didn’t originally see your disclaimer line either and jumped to the wrong conclusion about the gestalt of your statement. I’ll have a longer response for you soon.

Wow-- I went away for a few weeks, assuming that the question I originally asked had been too mundane to be addressed, and returned to discover this long discussion. I am grateful and imporessed by the care that goes into all the talk on this forum-- as a tech hobbyist, I am not used to that level of consideration on other forums.

As it happens, I stumbled back on this post because, upon my return from vacation, I decided to confront and sort out my versioning problem once and for all. The thing about versioning, as it exists in my current writing world, is that it is a “set it and forget it” system. My Cloud backup service continually backs up my work, every fifteen minutes or so. So as I’m mashing around my file, trying and discarding ideas, many of those attempts are “snapshotted” in the background. Once in a while I will think, “Oh, crap, I really should have kept that part about the city…” and then if I go hunting on the cloud, I can almost always find a file that contains that scrap of tested and discarded prose.

To repeat this experience in Scrivener, I would have to become much more self-conscious about taking snapshots of my work every ten minutes or so, forcing me to become distracted by the back up process, as Nathan suggested, and creating a bloated snapshot file that would be pretty hard to manage. (Creating a repository of random, rather than deliberate, snapshots.)

Speaking to the value of the program itself, I spent this morning testing some other software, in hopes of preserving my “lazy persons” backup system. NOTHING came close to the ease of use of Scrivener, and so I returned, determined to make this work, even if it means changing my own habits. No small task, as I’m sure you all know. :wink:

Okay, and, one last thought: what about an “auto-backup” option? Then I could set scrivener to automatically create a zip archive of whatever part of the project I choose every 15 minutes/hour/on close. That would totally do it.

Hopefully this may become available in the future as it is a feature found in Scrivener 2.0 for Mac. Is it unlikely that we will see it until after Windows 1.0 is officially released.

Problem with adding features is that you get bloatware. That, and other people may prefer other kinds of things that the software does. (emacs anyone? I already run a shell. But I don’t think AmberV wants a text editor war breaking out…) As programs get bigger, they become more and more unreliable. Look at the most recent version of MS Office.

Another thing to keep in mind is that the beta group is attracting a lot of people who’re good with computers. I’m willing to bet that 90% of the current beta users either have used CVS before or could easily figure it out. Once WinScriv is released to the general population, I don’t think people would be as adept, and I think the general computing population would be horribly confused. That isn’t to say that they couldn’t get it in time, but it would be another thing they just wouldn’t use.

Back to the question at hand: what about compiling drafts? I’ll compile a draft of whatever I’m working on before I make any major changes. That way if I want to majorly change something that I’m not sure is going to work, I’ll have something to roll back to. Or I’ll duplicate the individual file in the chapter, fork it to a non-draft file then make my change in the duplicate. (Usually move it to an “axis” or “old version” file under research. Bonus points if anyone catches the Big Finish reference.)

Well here’s my latest contribution what has turned out to be a surprisingly long conversation! The more I thought about it, the more I realized that I can make snapshots work for me reasonably well, by taking snapshots of important revisions and naming them (“Second Draft”), and taking snapshots every ten minutes or so and NOT naming them. That way I can easily locate and delete my scrap pile, and keep the milestones.

HOWever, it still seemed like an unnecessary step to me-- not for the first kind of intentional snapshot (First Draft, Second Draft) but for the second “always on” failsafe kind of snapshot. I don’t want to always remind myself to take a snapshot “just in case,” especially when I already have an excellent versioning system in place, storing gigabytes of prior versions automatically on a remote server. And what if, for some reason, in the heat of the moment, I forget to hit snapshot?

On the other hand, I take the point that the more extra features added, the less well Scrivener functions. The more I use it, the more I am dumbstruck by its brilliant simplicity. So as I fussed and futzed at home, I have finally come to a solution that works for me, and want to offer a related suggestion:

I discovered that I can open the search.indexes file, thereby discovering the numerical .rtf that corresponds to a document I am trying to investigate. Then I can go on to my cloud storage, and download as many versions of that .rtf as I need, to go hunting for my lost prose. This solution is really the only thing I needed-- I RARELY use my versioning backup (once every month or more), and don’t want a robust versioning system built in to Scrivener-- especially when I already have one working.

My suggestion: what about adding a file properties information item to all scrivener documents, which would allow a person to see where the file lives on the hard drive if they want to? I know Picasa does something like that, and I suspect many other file management systems do too. Just a thought.

For me, at any rate, the problem is solved. I know where to find the files if I need them, which I almost never do.

What I don’t understand is the argument that (a) taking snapshots is difficult to remember to do, and (b) unnecessary. How is this any different than pressing Ctrl-S to save every ten minutes or what have you, when working in another program—except that in this case you get an automatic record of your save log as you save? That’s all that’s going on here. Different shortcut, sure, but I fail to see how this is difficult or not automated. It is automated already; it’s a log of saves automatically compiled and “named” for you by datestamp? It’s no more “just in case” than Ctrl-S.

Good find with the search index; that’s a very useful file once you know its just raw XML and combines the plain-text material with an ID number. The .scrivx file itself can also be pretty useful in a pinch, but you need to know the binder name in order for it to be useful. But, given that information is there, if one knows a little scripting and XML, it wouldn’t be hard to make a script that extracts this information you request and dumps it to a .txt file.

I’m not sure if a feature for this would be “wise”. It might be useful in rare scenarios, but it might also encourage people to edit the RTF files by hand (something that is already tempting to do). An XSLT or something that can extract a binder name : ID list, made available to the forum, on the other hand might be better.

Just to clarify: I don’t think snapshots are unnecessary at all. My own version of snapshotting is central to my work flow. BUT it appears that I am much lazier than I realized: I have not hit “control + s” or control + anything in years, and the prospect of returning to that habit left me a little panicky.

When I begin a major revision, I make a backup of my original file, and have at it. If I am drafting, I just plug away, and let auto-save do the work in word, and my cloud backup service do the versioning. In my current workflow, I never think about saving, which turns out to be, for me, quite freeing. Even better-- my versioning is so granular that, on the rare occasion that I think “oh my gosh I should have kept that part”, I can almost always find it with a little digging. (And as I’m sure everybody knows, the part I find is rarely as miraculous as I thought it would be…) It’s more like homeowners insurance than a way of working.

As I said, now that I know how to make my old habits work in Scrivener I am fully and joyfully onboard. I will take snapshots (instead of backing up) when I start an on purpose rewrite, and let my existing backup systems work when I’m just drafting, knowing I can dig for the .rtf files if I need to. I agree that having the information as an option in scrivener could be risky-- I have made a hash of my directory structure in PIcasa on more than one occasion. This is okay with me, as it’s how I learn, but not for everybody. When you started talking about XML extraction you lost me, but the result sounded cool. Maybe some more technical person is following this, and can take it on. :slight_smile: .

Okay! Yeah, I guess this is just one of those things that comes down to preference. Keith (and I for that matter as well) prefer intentional systems to automated ones for things like this. I’d much rather have checkpoints where I said to myself “good enough” and hit the snapshot button (or Ctrl-S as I would have in the past, to write the RAM changes to disk). Or inversely, to decided the direction I’m taking something is rot and revert to the last save—with snapshots I have more history to work with though, so it’s nicer than just Ctrl-S over the same spot on the disk all day.

I guess some people, like yourself, prefer automated stuff. I never have because of the flood of chaff it produces (versions which are not useful; sentence-long if that, or recording mundane experiments with words). So for myself, I’ve never liked it, but it sounds like you do. It might work better with different writing styles as well, come to think of it.

Anyway, maybe it will some day be feasible for one-man-band style operations to develop fancy stuff like that, too. It’ll probably need to be a function of the OS itself, though.

Sounds like you have a decent work-around though with the RTF file access. So long as you just use the RTF file to reference and find the missing pieces and then paste them into Scrivener rather than rolling it back in place, that shouldn’t cause any problems.