Scrivener for iOS needs a better Sync model than DropBox

Big disappointment that Scrivener for iOS mandates usage of DropBox (which I don’t use), doesn’t support iCloud syncing in the platform, Google Drive, or other cloud syncing solutions, and is manual in its sync model.

There is no encryption support on DropBox for your intellectual property, something writers I’ve worked with care about when using Cloud syncing.

So given I’m complaining, how about I suggest a solution. You should probably look at architecting a sync model divorced from a given platform with translation layers, and try to figure out how to sync at a more granular level than files, while allowing concurrent editing across devices/platforms/users. They are large blocks of data which are difficult to shove around and have large locks on them. I notice a bunch of your early users are taking an hour or so to sync their file collections.

The better Cloud syncing platforms allow encryption and granular document editing. For example check out the Google Realtime API that helps you develop apps which use the same technology as Google Docs to allow concurrent and granular editing of files. I’d suggest thinking of importing your regular files into something like Google Drive (which has encryption in the platform) and then using the Realtime API to track edits. You then get concurrent user support for multiple authors on documents and all that goodness. developers.google.com/google-ap … e/overview

Much as I love Scrivener, which I have supported for years, I am left with no option but un-purchasing until the app properly supports syncing. Unusable in it’s current form for me.

Let me know if you need more help with this, I am happy to try and help,

Simon.

If you have a team of server and sync experts you can lend us, we’d be happy to oblige!

scrivener.tenderapp.com/help/kb/ … g-with-ios

EDIT: I note you replied to the thread I started directing users to that Knowledge Base article. If you actually read the article, you will find that I have explained in detail why we use the sync model we do. And no, other apps have not “figured this out”, because other apps don’t have to sync the sort of multiple-interdependent package-based projects that we do. We did not choose the current sync solution because it was easy or convenient (it wasn’t - there are thousands of lines of very complex code involved); we chose it because there is no “magic” solution to what we do. Read the bottom sections of the linked article for more info.

I’m sure Apple or Google could develop a real-time sync solution (although even then there would be inevitable problems with half-synced projects and internet connection drop-outs), but we do not have their resources: I’m a single coder.

While I understand this must be frustrating to hear criticism, I love Scrivener, and I don’t want to be a downer. I just want to help you get this fixed.

Google especially has done a bunch of work to make this easier for you so you don’t have to write your own file syncing libraries, but can instead leverage the real-time APIs. Also, they support Drive offline access so you can mitigate the internet drop outs you refer to. Honestly I don’t mean to sound facetious by saying other people have figured this out. Its a lot of hard work as you have discovered and I’d like to help you avoid it.

I used to run large parts of Developer Relations @ Google. I can try to connect you with some help from that end if you like, though I no longer work for or represent Google. If you send me an email I would be happy to try and introduce you to some folks who could help.

Until then, check out this QuickStart Article to see how you can hook up to the Syncing engine Google Docs uses developers.google.com/google-ap … pplication

A quick perusal doesn’t reveal anything there that specifically addresses all the problems I have syncing Scrivener files (multiple inter-dependent files in a package). Also, we have to recommend users not to use Google Drive because it has a tendency of losing data and corrupting regular Scrivener packages, and that’s just with Google’s own desktop client dealing with things and doing nothing on our side:

scrivener.tenderapp.com/help/kb/ … e-advisory

Moreover, given the lack of Objective-C API, it comes down to what I said before: as a single developer, resources are limited. I would need one or two more coders working on that. And even then, with the Realtime API, they would have to go low-level and use what is there to come up with a very complex solution. I think you are underestimating what is involved, or overestimating the scale of L&L.

If we can expand the team at some point and bring in some sync coders, that would be great. But our first priority was getting out Scrivener 1.0 for iOS with a sync that worked. I made some tough decisions when implementing sync, and I knew that not all users would be happy with them. That’s why I have been very up-front in telling people before the release how the sync would work. I chose stability and precautions against data loss over something that seemed “magic” but risked losing data.

As I say, I understand that not everybody will be happy, but then we can never please everyone.

Does Keith really have to listen to this? Help, someone!

…saying 1) you are no longer at Google (one might wonder why) and 2) you are not a coder? You are new to the forum, new to Scrivener perhaps, but already have solutions to problems you didn’t know about yesterday. Impressive…m :unamused:

I’m just trying to help. I coded for a long time on a bunch of things. I’m sorry if I’ve upset anyone, that’s not my intention. Syncing is hard, and I can appreciate the resources you need to do it.

I’m not suggesting Google Drive per se, but a realtime sync API so people can edit at a more granular level than monolithic documents and don’t have to spend a long time syncing. Some of the hard work has been done by Google that I thought might help Keith/L&L development.

Sometimes companies help with resources and people to support their development community for popular apps like Scrivener and while I can’t speak for Google it might be worth asking their Developer Relations teams about it. At least they might build you an Objective-C or Swift API. I can help try and connect you to extra resources and support if you want.

I just want Scrivener iOS to be successful, and was trying to think of ways to support you in that goal. I don’t think it’s there yet with the DropBox sync model.

I’ve been poking around a thing they have set up called the Realtime API Playground which looked interesting… realtimeplayground.appspot.com

So - again I’m sorry if I’ve upset anyone. I just wanted to help.

Suggesting on Day 2 since the release of the app that it is not gonna be successful is not very helpful, especially since there seems to be quite a lot of happy customers out there. The few that didn’t understand the extremely simple syncing process were an extreme minority.

Besides, you claim in your first post that Scrivener

which is a strange statement, especially since Mac Scrivener saves/updates the project automatically on close, and iOS Scrivener automatically updates any changes on open. The only thing needed is for the user to say Yes, and to save/update before closing the iDevice.

Looking at it that way you must surely think that most software is very manual, as you are required to manually Save and manually choose where to save, and manually Open files, etc. I actually can’t think of any software where everything is automatic, in the way that I don’t even have to manually choose what to Open, Save, Close, etc. Do you?

I’m happy to be able to work on my iPad on Scrivener. So, kudos to Keith for pulling this off. Keep in mind that this is 1.0 release. I’m sure there’ll be many improvements, etc., coming in the future. For those coming improvements, I’d also like to advocate for more options when it comes to online storage and sync.

I’ve been using Box.com for a while now and my writing folder, including all the Scrivener files, is backed up to Box automatically. I don’t know the ins and outs of syncing the complex files Scrivener creates to online storage, but my Scrivener files back up to Box just fine. So the complex nature of the files don’t seen to be an issue there.

Maybe in a future release there will be more options. For now, I’ll use the iTunes method to get my files back and forth. In itself a bit cumbersome, like the copy and paste I’ve done in the past, but now I have the entire project with me and that makes all the difference.

So with the greatest respect (sincerely) I am a customer, and because I don’t use DropBox the product isn’t usable for me as a result so I should be able to express that without recrimination, and I think that should be helpful to the authors of the software. I think all customer feedback is helpful so you get to know what’s important for dev priorities. Maybe my concerns are an edge case. I think it’s too soon to say whether the people having problems with syncing will be a minority but prior experience tells me where there’s file syncing there’s usually problems. Please understand I’m trying to air a user perspective not sling mud and if I didn’t care about the product I wouldn’t bother saying anything. I get where you’re coming from, but not every paying customer is going to like being told their feedback isn’t helpful. It runs the risk of making them not want to buy the product or provide feedback, and that only hurts the product.

As a former developer a long time ago in a galaxy far far away I wanted to suggest some different approaches for replication/syncing. I’ve written database synchronization and file synchronization code before and I know it’s a pain. I think that’s too much for one developer to have to wrestle with on their own. Keith is right when he says I assumed L&L had more development resources. So my coding hat comes on and I wanted to convey “this is easier if you don’t use files and sync lines and data to a store”. We call it ‘shredding’ the files into a store. If you do it that way you can use the storage/replication APIs on commercial databases (with a local and remote cloud store) or something like Google’s RealTime API, and the file conflicts and all the nastiness from file systems goes away. Most of the file syncing problems that arise in my experience come from the problem of the file system not being a transactional store.

I think I’ve been using Google Docs and cloud tech too long, I don’t press Save buttons anymore on anything I use, I just kind of expect it to stay synchronized. Again, this is a user perspective. I might be an outlier but I think the feedback is still valid.

Thank you for taking the time to reply to me lunk, I hope we can get along a bit better.

1 Like

These are all well made points. I wish I had been more diplomatic in my earlier posting, you are right it is a 1.0 release and Keith deserves kudos for pulling it off.

They say most people project their own experience and fears onto others. I think I’ve wrestled with syncing so much across my career in software development it makes me see red and I go at it again like a pitbull. I kind of get a ‘oh no not this problem again’ feeling, “well you fix this like this”, and that can come off as a bit of a bull in a china shop.

Most of the problems that show up with syncing have nothing to do with the code you write as a developer but are rooted in the fact file systems are non-transactional unlike databases, so you get into all sorts of inconsistent states. Add to that merging changes and it becomes a world of fun, especially when you add multiple users making changes. Thats without factoring in hardware going wrong and networks failing.

It would be much simpler if I had said “Good job on v1.0 Keith, I’ve wrestled with file system syncing my whole career and its horribly painful. Can I help you think about ways of evolving this functionality for v2.0 or v3.0?, I have some ideas on how to lift the burden on you, and I might know some people who can help”.

Lesson Learned.

Yes, it would. :slight_smile:

What you suggest would require the whole structure of Scrivener to be changed, making it something different.

Besides, even on Google Docs you need to name your file manually and decide if you want to keep the changes you’ve made or not and make a manual Save at some point.

Why not get yourself a free Dropbox account and test how it works and feels? You’d probably get a better feel for it then.

One factor to keep in mind is that a large portion of the Scrivener user base would be ill-served by a file format/sync system that requires always-on Internet. Many of us will intentionally sync up, then pull the plug so we can focus on writing. Scrivener’s file format also degrades nicely in the rare event something goes wrong – it’s a collection of XML and RTF files, so even a relative neophyte can find and recover their work even if something causes the Scrivener file structure to go wrong.

I’d just like to say that I do value all feedback, positive or negative, when it is constructive, which most of this thread has been. I do admit that my hackles were raised by your first post, because of phrases “big disappointment”, “I am left with no option but un-purchasing” and suggestions that would take us a lot more resources to implement, combined with first day release excitement and no sleep. I’m sure that’s the only reason a couple of other users also had their backs put up, too, but everything since then has been perfectly polite, reasonable and constructive, so I’m glad the thread is now continuing in a more helpful tone on all sides.

As Devin says, the problem is that you are suggesting a fundamental change to the underlying file format of Scrivener. I’m not averse to such changes in the long run, but we have to evaluate how they affect features. Scrivener has to work offline and never be reliant on a cloud service. And it needs a file format that can easily be read into memory by the iOS frameworks. For this reason, it uses a “file package” format. Each document you create on disk is an RTF file inside the project folder. The binder is an XML file representing the structure. And so on. Another advantage of this format is that users know that their data is in a format they can always access, even if we disappear off the face of the planet (again, as Devin points out).

I’m not convinced a database format would work as well. macOS and iOS support Core Data for a database-style format, but I looked into that long ago and it’s really not suited for Scrivener. So we would need some custom solution.

Ultimately, it comes back to resources: we would probably need a small team working on this one problem, coming up with a different file format that could still retain all the benefits of the current package system and writing an ad hoc syncing solution based on whatever platforms are out there. Syncing is always going to be an issue, and I always knew that it was going to be the one thing that users would be less keen on in the implementation of our iOS version. I’m sure the pressure will only build, too. The only way I see out of it, however, is either to move to a city, rent offices, hire a team and hope they can do it, or sell up to Amazon or Google or someone. Syncing solutions such as iCloud (which is what many users want) work amazingly for simple file formats, and of course, users don’t understand the differences between file formats and the problems involved. But they don’t work so well for us. I think syncing is going to make it much harder in coming years for small companies such as us to provide truly complex and deep apps and still meet user expectations. Unless there is a breakthrough with something like iCloud that provides something easier for indie developers with such problems.

All the best,
Keith

Hey Keith,

Obviously I for one would love iCloud synch, but read and understood your statements on this long ago. To be honest iCloud was pretty lousy in earlier implimentations though is now robust in SIMPLE file management.

Dropbox while not my platform of choice works a treat and it’s been a painless experience working with Dropbox, first to enable me to work on Scrivener on multiple Macs/PCs, and now with the absolutely amazing iOS Scrivener.

Thank you for not running around like the headless chook, trying to accommodate every (some crazy) whim (iCloud, Google Docs, Box, One Drive, TVoS, Watch oS etc) and instead concentrating on producing a top app that ‘just works’ to borrow a phrase.

Off topic - reviews of Scrivener for iOS in the tech press have all been rightly AMAZING, and here in Aus today 2nd Grossing productivity app, beaten only by Word.

Take a bow Keith.

I know you unfairly copped some serious shit over the past year or two, (and some complaints/‘suggestions’ the past couple of days that made me shake my head) but you shook it off and came up with the goods in style. I still don’t know HTF you stuffed it all into 7+MB

Thanks Keith. Let me know if I can help as you think about synchronization for future versions of Scrivener.

All the best.

I would like to second that Box Sync be considered as another sync choice besides Dropbox. Many universities now have volume license to use Box Sync for their university population. For instance, major universities in the USA including Carnegie Mellon and Duke University provide their university population Box accounts.

I don’t think I have heard of a university in the US where they offer their users Dropbox accounts. It’s usually Box or Google Drive (and OneDrive, which comes with the Office 365 subscription).

For the record, I do know of a (large, public) university (in the US) which offers (secure) Dropbox accounts to all its faculty. gr may or may not be connected to said university. :wink:

Hi,

Can I also add my vote to a encryption feature request? I am a Barrister and the iOS version is incredibly useful tool for drafting legal documents. A proper encrypted sync would be the icing on the cake for this product!

Haha. I was doing some digging and looks like Arizona State U is a customer of Dropbox :slight_smile:. I’m at Duke where everyone is given Box accounts :slight_smile: Good to know there are other universities using Dropbox.

Here’s an interesting read on Dropbox vs Box: computerworlduk.com/it-vendo … d-3631134/
I guess the impression I had that big enterprises used platforms other than Dropbox was formed because indeed Dropbox is a latecomer to enterprise compared to Box.