Seeing Dropbox right...

I would suggest that if zipped archives are desired as a format for project storage, the default be left as is and the zipped format be selectable as an option. Changing the default format to zipping probably would not be a good idea. I think the majority of users would prefer to keep things text-only. I don’t use Dropbox for collaboration, just cloud-based backup, so I prefer text.

But making compression a project-based option sounds like a reasonable thing to allow for the subset of users that would prefer it.

Throwing this out there, but more ‘advanced’ users will find ways and other apps that allow them to work with Scriveners better, at least as they are currently designed. Zip is/could be nice, but certainly aren’t the only means.

I for one, put sensitive projects into a TrueCrypt Volume that I toss into Dropbox and then mount the volume as needed. This would also work perfectly for Scriveners’ files, but more as a solution if the current format of Scriveners aren’t altered rather than as something that Scrivener could integrate as a solution.

TrueCrypt with Dropbox won’t allow for simultaneous editing at all, but Dropbox isn’t the tool for that sort of thing anyway. Toss the scriveners into a TrueCrypt volume, mount it, alter it, unmount it, then the changes will re-sync and you’re good to go and secure to boot and I believe that other users would be locked out while you have it mounted (though I’m not positive on this count). Slight issues might be long upload on the front end and/or the possibility of incorrectly judging the size you need for the Volume and thus either wasting space in your Dropbox or running over within your Volume itself.

I guess the end of this is that I thoroughly plan on putting my Scriveners into something that makes Dropbox think it is a single file anyway, but it could be nice to have it treated as a single file straight out of the box.

The thing is, we’re not talking about compressed, just “combined”. On the mac, a scrivener project’s individual files are mostly hidden from users because the project’s directory is named in such a way that OS X treats it and it’s contents as one file. Making use of zip for the default file format will in no way lock you out of seeing the individual files (which you aren’t supposed to touch under normal circumstances). If you need to get to them, rename the all-in-one scrivener project to scrivproj.zip, and use a free/built-in zip utility to extract the bundle into individual files & directories. Anyone who really wants to see those files would be capable of getting to them with minimal effort, while the vast majority of people who are used to dealing with a single Word file will never be faced with the baffling array of files like 1.rtf, 2.rtf proj.scrivx settings/ snapshots/ …

This isn’t like changing the format into something that can’t be accessed by mere mortals, like the MS Office file formats, it’s just a handy way of keeping all of a project’s files together on any platform where Scrivener works.

This all seems to more a job for .tar, than .zip. Compression is nice when you need to preserve space, so it is great for backups, but for a working format—I guess I don’t get the point of compression, either. That’s a ton of processing overhead, and it would effectively cripple auto-save for even a small-ish project of 20mb. Tar is a lot faster, and it’s pretty easy to extract and update individual component elements on the fly. It would still be slower than a file-system by a long shot, but in most scenarios the slow-down would be barely noticeable—and presumably anyone using this method would be aware of the fact that one huge file is not going to be as much of a performance demon as files and folders.

As far as collaboration in a group or enterprise environment. Why reinvent the wheel? It seems like distributed source control solutions, such as Git and Mercurial, have already solved this. Why not simply open up the the file format and have the source control repository take care of the heavy lifting? By using a DVCS enables me to save and version locally. I have tag and branch certain ideas I might have and then merge them locally. If I want to share them with someone I can use something like GitHub.com to share by thoughts or changes with the rest of the world. You could even comment and collaborate using this type of tool.

Just my thoughts but I am a software engineer by trade, so this may be overkill for the average user but having said that you could abstract the average user from the complexity of source control tool by supplying the same constructs through the main UI of Scrivener.

Food for thought :slight_smile:

Joe, I think we went over this ground earlier in the thread.

  • The first thing is, the basis of Scrivener is formatted text. Thus a source control system isn’t going to work for this, unless you know of one that does edit/merge for RTF.

  • The second thing is that Scrivener also has another level of structuring – the scrivenings in binders. A special-purpose editor would have to be written to merge the XML files that control this – and it would need to be integrated sufficiently with the formatted text edit/merge that complex edits on a project would be fully visible and manipulable.

It would be instructive to dream something like this up, but a great deal of effort to make it complete and workable. A look at Word’s document compare/merge will give an idea of the level of complexity just for the RTF merge aspect.

I haven’t so much more to say here, except that the idea has been to hew to what non-technical persons are comfortable with and practiced at using, so as to not get in the way of anyone’s writing, and thus not losing track of Scrivener’s purposes. I think that means Dropbox and Zip files, doesn’t it?

I also appreciate what Ioa is saying above about tar, etc., but really, a non-compressed zip just would do the job, wouldn’t it, including high-efficiency Dropbox synchronizing?

Adaptation always seems the actually sophisticated thing; at least in experience.

Regards to each,
Clive

Viable for what? If your goal is collaboration, use of a zip archive doesn’t even begin to address the issues. It helps avoid out and out file corruption, but does nothing to help with versioning and change management. And let’s not even talk about trying to do simultaneous editing in real time.

Katherine

The point I’ve been convinced of is that .zip can be used just like a virtual disk without compression , but on Windows, Linux, as well as Mac; Am I wrong in my understanding that newer versions of Windows come with a built-in Zip utility, but not tar? Using zip without compression on the project keeps all of your project’s files looking like just one file, so Drop Box no longer mucks with the internals of the project when there’s a conflict. Instead, it creates an entirely new “conflicted” project, which is much more noticable than a 324(conflicted).rtfd file appearing in your project. But again, without the compression, it shouldn’t slow down saving (you can tell a zip utility to replace individual files), and drop box will only transmit the new bytes, not the entire archive.

Viable for using as a “file format”… the discussion has drifted a bit (though it’s incredibly on topic for these forums). It’s also helpful in that if something gets conflicted, you get a new copy of the project (equivalent to a blaring siren), instead of files within your project that can remain unnoticed for days, weeks, or even months.

As for simultaneous editing; no, zip archives are not the answer. The answer is for Keith to hire about 10 more Mac programmers (bringing the total to 11), and devote a goodly chunk of their time to shoe-horning in the ability to simultaneously edit a project. Until Keith wins the lottery, or Scrivener rivals Angry Birds in number of sales, I think you’re going to have to give up on simultaneous editing of Scrivener projects. If that’s your primary concern, Google Docs seems the way to go from what I’ve read in this discussion.

Aye, Google Docs. For rapid collaboration. :wink:

Great thoughts from everyone; this is one of the most conflicted challenges in computing (the collaboration part), and it probably takes a Google to get the software right. Even they falter on the user view (e.g. Waves).

Hope I’ve taken right tones through this for everyone. Writing is my interest too, among others in that space. Technology one may understand able to be of certain help, as Scrivener; beyond such a point, often there be dragons. As we have met.

Regards to each,
Clive

For those only wanting to use Drop Box as an automatic backup service, it might be worth considering not saving to a Drop Box folder directly, but instead using a synchronization tool to perform a one way copy of files to a folder which is then synced with Drop Box. This would prevent any file sharing or other issues while still providing one of the basic benefits, offline backups. There are many good free sync tools that can be used, so it may be something people may want to consider given some of the potential sync problems.

Dan, the thing is, doing this would unfortunately then reinvent the problem.

Whenever the files in a Scrivener project are separated, then Dropbox can and will create the differently-named ‘conflict files’, every time two or more Dropbox contents are changed.

Then portions of text, and also whole scrivenings, depenwill be apparently lost.

That’s the problem we’re trying to assure won’t happen in future, and the only way that logically will do it, as far as I can see, is:

  • have the Scrivener projects in single files

  • do a manual merge of the enclosed projects, whenever Dropbox indicates that these single files have accumulated mutual differences from the original, which it will do by making its renamed copies.

Regards,
Clive

I’m not sure I completely follow what you mean by files being separated. Using this method, the files are copied to another directory that should contain all the files, not just the changed ones. It should be an exact replica of the actual scrivener data folder. It might be just be that I’m not really sure how the scrivener file structure works. Is scrivener reusing/renaming filenames? If so, this would certainly cause this. I will say, based on what I’ve seen, it would be tricky to piece together a project based on the files with the information I have found. I would like a better understanding of the files in case I did ever have to attempt to recover a project manually.

Dan, there are so many issues here, actually. I have background both in the general problem and in some specific things I was asked to do on a consultancy once for the BBC.

There are two fundamentals:

  • one is the need for what is called ‘atomicity’ in replications. This means you deliver nothing but a fully integrated and consistent result: the total state of a multi-component Scrivener project at the point you saved it. This is not something Dropbox or the like can possibly provide, the way they are designed, unless the Scrivener project is archived into one single file for Dropbox to deal with.

  • the other is the need for intelligent merging. Only a person, and somewhat an informed person at that, can successfully compare and merge two versions of a Scrivener project. It takes knowledge, after you have the RTF and structure editors that Scrivener provides. And you only get the chance to do it if the first condition is fulfilled.

Without these, you will at any mysterious point get text that can’t be found, text that appears to be lost, even though at other times the Dropbox seems to work.

The simplest case for it is when one project-internal RTF file, which holds a scrivening, gets modified at both ends after the naive project directory is put into Dropbox (or closed from Scrivener there). Then Dropbox will create a re-named version of that RTF file, which you will never see in Scrivener. You could only manually find such files, and then, guess what, manually edit them in – which you can’t do from Scrivener itself.

There are all kinds of variations on the problem, and in proposing a solution, I found in reflection this morning that I’m still depending on Dropbox getting certain potentially tricky things right. Probably they do – as it’s a well-recommended program for many persons.

If you need another example from the field, that would be what happened when Dropbox-like programs first appeared 15 years ago, and the first thing that smarty people thought to do with them was run their Quickbooks accounting through files shared this way. Instant unreliability, for similar reasons, that a database also requires the ‘atomic = all at once or refuse changes’ approach. To actually have a database do this over distance is extraordinarily demanding, as someone’s husband above has mentioned, and costs it from Oracle or others.

Why do people think Dropbox just ought to work? Well, naive Dropbox and Scrivener can work, indeed – so long as their users are absolutely not naive.

If one or more persons clearly understand and absolutely accurately follow the sorts of absolute limitations with close, consistent communications Keith has outlined, then you can avoid the areas where trouble can come.

One mistake in that regime, and then trouble is perched on your shoulder. As in, two persons writing in the same area of a document – or one person doing so from two locations without remembering to have closed Scrivener down fully before they left the one where they’re not. Or additional scrivenings added, same fault circumstances.

In the end, choices are often a matter of trading off apparent convenience with risk. I’ve hoped the discussion here would end up with a way to take the risk out, but still leave Dropbox useful. That we’ve done that and raised Google Docs as the way to do true 'writing together, seems worth the candle.

Best to each,
Clive

Ah, I think you missed a key statement in my original post. I was only referring to situations where drop box is to be used as a backup method, meaning the files are not accessed on multiple machines and syncing is only one way. With this, there is the option to even capture the entire scrivener project and drop it into the drop box folder. This can even be done as a zip file. I was not intending this be used in any way to access Scrivener on multiple machines. This is simply an easy way to achieve offline storage for the project in case a laptop gets lost or stolen but you don’t want to save to the actual dropbox folder.

Dan, you’re quite right - I did miss that crucial point. Hence going through the thoughts again, possibly not a complete waste :wink:.

If you wanted to do automatic offsite or backup storage this way (not a bad idea for cases), then would think doing it from Scrivener backups would be a nice way to go. You can use your drop-folder to re-drop into Dropbox, and get the benefit of the unique names, so that you not only back up, you back up over time.

So that’s a nice idea, and sorry I mistook it. I do think it’s better than simply backing up to the Dropbox, because you have the local copy which won’t be affected by anything done at the other end of the Dropbox, hence real diversity. Diversity is good, in backups and time-saves, my experience.

Best,
Clive

I like the thought of knowing nothing will be messing with my files, but having an offline backup is crucial. Ultimately, it would be nice to have an automatic ‘backup on Scrivener exit’ feature so it drops a backup of the project into the drop box folder anytime I close the program (or at other scheduleable intervals). The downside would be a brief (hopefully) pause on exit while this is done. It’s doable with other tools, so it’s certainly not a mission critical feature but it would be nice.

Already exists in the Mac version. (See the Backup option under Preferences.) I don’t know about the Windows version, but if it doesn’t have it now I assume it’s coming.

Katherine

Great! Then I’m sure it will eventually come to the Windows side as well :smiley:

Sorry for the late hit on this, but I’m just finally getting caught up on forum posts.

The modern (.docx, .xlsx, .pptx, etc.) Word file formats are doing this exact trick with using the outer file as a ZIP-format wrapper. There were several reasons that Microsoft decided to use this format:

  1. Office docs are often bulky, so the same document saved in .doc and .docx often shows a substantial space savings in the newer format.
  2. Windows includes the ability to open/parse ZIP files in Explorer natively, so the appropriate routines are in place from Windows XP forward.
  3. The new Office docs are in fact just collections of files – mostly XML, but things like imported pictures, etc. are simply imported into the ZIP archive in the appropriate place and linked via XML. As a result, new APIs could be created to manage contents of the new file formats without requiring the use of Office COM objects.

The decision to use the ZIP-format wrapper has certainly turned out to be a good one for Office 2007 and 2010. It’s actually reasonably performant (although, granted, Word’s not auto-saving everything as you go). Even for netbooks, the ZIP compression algorithm doesn’t require a ton of CPU. It also reduces the number of active filehandles required, and I believe I remember hearing one of the Office developers talking about how that actually coding for file I/O much simpler.

It would certainly be nice to see this as a (non-default) option in some later version of the program.