How to Fix Scriv's Biggest Problem?

Scrivener Scrivener for Windows
1

I’m a big fan of Scrivener. I’ve used it for nine books, and I’m working on my tenth.

There’s one process that’s excruciatingly tedious–and scary, too.

Every time I finish a book, I compile it to a doc file and send it to my proofreader. She makes perhaps 200 corrections, and I accept or reject each. I then I have to get the document back into Scrivener.

I’ve done it scene by scene, but it’s hard, error-prone work: copying, pasting, setting default formatting, etc.

My scene structure typically looks like this:

Scenes.jpg
Scenes.jpg318×853 74 KB

I know about import and split, but that wouldn’t work because I’d lose all the Part designations, Chapter designations, synopses, document notes, etc.

If there’s no better way to do that, here’s a suggestion:

Add a compile switch that embeds information in the DOC file. This information could be right in the output, like ##Part I$Chapter 7$Scene 3##. The proofreader would be instructed to not touch any of that.

The corrected DOC file could then be reimported into the existing project, and Scriv would simply used the embedded information replace the text in the appropriate existing scene.

Thanks for listening.

2

The proofreader would be instructed to not touch any of that.

Well that’s one problem right there, one of the lesser problems at that. Even if one is aware of the markers themselves it would be easy to accidentally damage something like that. Consider a seemingly innocent search and replace using anything at all that might be in the title—such as from your example replacing “Eric” with another name. The method would be very fragile and require the utmost care in protecting those separators.

Then you’ve got cases of duplicates (copying the Draft as a “first draft” for example). What of items that don’t have a name? That will become much more common a practice I am sure, once binder titles act more like they do on iOS (if you aren’t aware, if a title is left blank it uses the first few dozen words or so from the file or synopsis).

It’s why we can’t depend upon the names of things with the folder sync feature either, and use internal ID numbers to keep things together.

That is of course an approach, using internal IDs—and right now that wouldn’t be too awkward, but once the format gets updated to use UUIDs for better robustness in a sync environment, you’ll end up with some rather ugly dividers in your file, like so:

<ID:D765AB88-FA38-4BC1-A382-5F8BD522716F>
Chapter One
<ID:C2BB7606-193E-46AB-A32E-5381E2EAD08F>
- Midnight -
LOREM IPSUM DOLOR sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
                       * * *
<ID:A1BC1A79-3BDD-4104-A4F7-41E52F1684A0>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

That’s not the end of the world, but now that we have a practical example to look at, you might also spot some other deeper and much more difficult to solve problems. This is one of those ideas that sound good on the surface, until you start going down the feature list in the compiler, and Scrivener in general.

What happens with all of these headings and separators the compiler can insert into the text? Do the asterisks and scene heading that were dynamically inserted between scenes from the Separators and Formatting pane now end up in the scene text itself? The next time you compile it would look like:

<ID:D765AB88-FA38-4BC1-A382-5F8BD522716F>
Chapter One
Chapter One
<ID:C2BB7606-193E-46AB-A32E-5381E2EAD08F>
- Midnight -
- Midnight -
LOREM IPSUM DOLOR sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
                       * * *
                       * * *
<ID:A1BC1A79-3BDD-4104-A4F7-41E52F1684A0>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

And that’s only considering additions. The compiler can also manipulate the output with substitutions and subtractions as well. How does Scrivener know those first three words in the chapter were converted to uppercase by the compiler, not the typist? What about all of those auto-numbering and meta-data placeholders now turned into static text? What happens to your comments and inline annotations if you did not compile with them enabled? What about any use at all of the Replacements option pane? There will only be more questions such as these as time goes by and Scrivener evolves, adding new ways for the compiler to generate output: custom meta-data, the ability to <$include> text from one item into another, etc.

In short: compile is just that… compiling. It’s a small bit like compiling software from source code: you can maybe, in the best and simplest of cases, get back to some semblance of source code using a decompiler (common in reverse engineering), but you will never get the original source used to create the software from the .exe itself. Compiling in both cases is a destructive (though beneficial) process.

So something like this would only work in the most dirt-simple cases, where one is hardly even using Scrivener’s features to their advantage. All chapter titles and headings typed into the editor, with every number manually typed in, all formatting already set in the editor. No scene-by-scene files—you get the picture: we’re starting to describe a Word file. :slight_smile:

Jumping back to the top of my response: I mentioned a feature called folder sync, have you had a look at that one? It’s the File ▸ Sync ▸ with External Folder menu command, and it is designed to export a folder full of files that can be edited in other programs. Later on when you load the project it will check this folder for modified files and automatically inject the text back into each binder item where it came from. It was originally designed for people to work mobile—but it works just as well for collaboration.

The sync folder itself can be tossed into a cloud share folder that your editor has access to, and now you’ve got a central editing area that your project stays in sync with and keeps up to date with your own changes automatically.

The downside of course is that your editor is now working in 91 files instead of one—not everyone would be amenable to that, but if you provide a compiled copy to them as well they could use that for reading and make edits to the corresponding files as they go.

Lastly of course there is getting them to use Scrivener. :smiley: Yeah, neither option is something many editors may jump on, I am aware, but really—there is no coming back from a compiled file without necessary data loss to the extent that import & split really probably is the best way to handle it in the end (it would take a human mind to solve many of the above problems anyway); or as many do, call the point where things transition into editing the point where you conclude work in Scrivener for that project.

3

Thanks for the detailed analysis, Amber.

As a retired programmer, I see those problems as slam-dunk easy to solve.

For example, before importing, Scriv runs a check. If there isn’t a one-to-one correspondence between scenes, it will give an error.

My proofer makes no changes to the files, and definitely no search/replace. I go through her list of issues and accept/reject the suggested fixes, like this:

Editing.png
Editing.png1122×828 73.6 KB

It would be easy to avoid changing any of the “header” lines, and Scriv would catch any of those errors before importing.

What happens with all of these headings and separators the compiler can insert into the text? Do the asterisks and scene heading that were dynamically inserted between scenes from the Separators and Formatting pane now end up in the scene text itself?

That’s the easiest to solve, as long as you think of the edited DOC file as a data file. That is, it is just a database of scenes, and those headers and separators are simply markers telling scriv where that scene belongs. Nothing fancy or complicated. If, during the check, Scriv sees any irregularities, it flags them.

The database happens to be editable by a human, in the same way the .ini files may be edited.

Your comments made me realize that this process could be done outside of Scriv proper. The “database” doc file might look like this:

…
I thanked them for the conversation and coffee and left. They’d played me like a fish but failed to set the hook.
##HEADER–DO NOT MODIFY “C:\Users\Al\Documents\ST\Sanity’s Thief.scriv\Files\Docs\236.rtf”
The Newburn Dixie Diner was a full-course meal of old-timey charm. I’d read that in the early 1900s, diners were prefabricated and delivered to small towns on rail cars.
…

And the import procedure could simply replace the RTF file with the text for that scene.

Bottom line, right now, I grab each scene from the DOC file, and copy and paste it into the corresponding scene in Scriv. That’s the kind of procedure that computers are very good at.

I hope you’re developers will think about this with an open mind.

Al

4

With app developing being basically a one-man job for Mac and Windows, respectively, I can see several problems. The biggest is maybe that all Word users who correspond with Scrivener authors do not use the same technique for making changes and comments in the text.

Wouldn’t it be easier for Microsoft to make Word able to read and write Scrivener projects? They do have the resources and lots of developers… :wink:

5

Al, with a good deal of hesitation, I’m going to jump in here and suggest approaching the problem from the other end. If you’ve done 9 books this way, I’m in no position to give advice; but that has seldom stopped me before. :wink:

My understanding is that Scrivener, despite its compiling capabilities, is not really meant to be used for the final writing and formatting of a project, only for organizing information and ideas and creating a draft, to be finalized in a full-on word processor. Since you’re compiling to MS Word and your proofer is working in that program, and since at that point it’s presumably not a matter of placing entire chapters somewhere else in the book, but only dealing with relatively minor adjustments–why import back into Scrivener at all? Why not just do the whole proofing/correcting process in Word, since it’s going to have to end up there anyway?

I myself do zero serious writing in Word, and I yield to no one in my loathing for it as a program for writers. My process is to organize in Scrivener and write in Nota Bene, then export from there to Word if necessary. Final adjustments can be made in Word, even though I’d never want to write anything major in it. (Admittedly, if I want a final state in NB it is easier to reimport into it than into Scrivener.)

I guess I just don’t see the need to do the final processing in Scrivener rather than in Word, even if one must hold one’s nose while doing so. Enlightenment, please?

6

@DavidR: Trombone probably has his reasons, but here would be mine as I always ensure that I keep all my work as the top-copy in Scrivener. I’m an academic, so perhaps this is specific to us, but Scrivener is the place where ideas and words are locked together, and often (e.g. academic grant proposals), one project is the well-spring of new project. I sync changes my collaborators make back to Scrivener always, stopping fragmentation of my papers and proposals — the Scrivener project is the ground truth. I developed this after doing the Scrivener=draft, LibreOffice=final for a couple of years, as I ended up with lots of problems trying to work out exactly where I made changes, where different bits were. My current workflow uses named snapshots and it therefore keeps a history from first idea to published paper all in a central place, with all the rough ideas, relevant papers, figures and metadata also live. Like Trombone, I sync back to Scrivener manually, and for my workflow I think this is the lesser of two evils.

I use MultiMarkDown in Scrivener, and the major benefit is you do not need to do much in a word processor, offering a far more powerful post-flight workflow.

A better way to sync back to Scrivener would be appreciated, but this is a non-trivial technical problem unless you enforce a very strict workflow on your collaborator / editor and ensure only certain formats are allowed to sync back. Still, I think this is a major weakness for many Scrivener users who collaborate with others and do not want to cede their projects to end up in Word, so a creative solution would be great.

7

That’s interesting, nontroppo. I’m also an academic, though I only found Scrivener around the time I retired. I’m still researching (though I seldom collaborate), but I don’t think I’d want to use Scrivener for writing an academic paper, much less a book. It’s footnoting capacity seems fairly limited, and bibliographical management nonexistent (which is fine; that’s not what it’s for). But of course not having tried it I can’t really speak to the reality of using it in that context.

EDIT: FWIW, I thought it might be interesting to further explore what seem to be differences in approach to tools here (one tool for most everything, vs. a variety of tools for various purposes). See this thread if you’re interested. I’d especially love to hear from TromboneAl and nontroppo.

8

Presumably the .docx file is changing though, otherwise there is no point in importing it. It doesn’t really matter who makes the changes, and such a feature needs to be more robust than what one person’s very specific usage of it might entail.

Ultimately this would probably need to bypass compiling entirely and be some kind of special export option that produces a very strict output that may not in some cases look anything like the eventual compiled result—but it would at least match what is in the project. <$label> wouldn’t get turned into “Eric”, <$n> wouldn’t get turned into “21”, there would be no checkbox to convert italics to underscores, no Replacements, no options to remove comments and annotations, etc. etc. etc.

There are no simple and reliable ways to reverse forms of transformation and substitution such as these—and I would contend that the other insertions are not so easy as you’re anticipating. Sure, if one only uses Separators to insert “* * *” that’s one thing, it could even as you suggest be considered a data delineator (but not always—consider the “single return” separator option which is indistinguishable from a paragraph break), but everyone is free to now and then insert their own scene breaks by hand. Again we can’t make assumptions based on how one person uses the software for something that is as potentially destructive as sync.

So I think a simpler form of export that works a lot more like dumping a Scrivenings session of the Draft folder to a text file would be better. The question is whether or not having one’s 18pt bright green (because they use a black background) Consola editor text is what they want to be handing out as advanced copies. :slight_smile: Again, we can’t just operate on the assumption that a raw editor dump will look roughly like an ARC—and conversely that what one wants that to look like is what they want imported back into their editor.

Aren’t we now talking about something much closer to the folder sync feature I referred you to though? Sure it’s not in one file, but separate files much reduces the risk of splitting things back up.

9

Thanks for thinking about this, guys.

Aren’t we now talking about something much closer to the folder sync feature I referred you to though?
Yes, with one important difference. My proofreader doesn’t need to go through many different files.

Really, all we’re talking about is a concatenation/deconcatenation of files. If I were still a programmer and had my tools, I could write a usable version of this app in two days.

Remember that during proofing, the only changes are made by me. As seen in the above screenshot, I move down through each change suggested by the proofreader, and (thoughtfully) decide whether to accept or reject it. Even if she made some change to a header by mistake, I wouldn’t accept that change. Most changes are commas, hyphens, or spelling fixes.

My understanding is that Scrivener, despite its compiling capabilities, is not really meant to be used for the final writing and formatting of a project,
Right. But like nontroppo, that’s exactly how I use it. Even after the book is published, I like to have the project available in Scrivener. Just today I copied a character from an old project. All are a click away:

Projects.jpg

I make zero changes outside of Scrivener for my ePub, MOBI, and PDF output. The only time I work outside of Scriv is the formatting for the paperback book.

10

Well then, with all due respect, and assuming I really am understanding you right, and granted that it certainly makes sense to copy material from one Scrivener project to another: is it reasonable to say, “I’m using the software in ways for which it is not designed, and why can’t it do that better?” (Let alone to say that this is the software’s biggest problem!) I mean, I wish I had a dog that could go through my e-mail and delete the spam, but you know…

11

DavidR: the problem Trombone has is not to do with final formatting, but with being able to easily work with other people. This applies equally to someone who is collaborating at the early, middle or late stages, or an with an editor working at the late stage. Most people normally encounter this problem (how can I merge the changes someone has suggested), at the late stage (thus temporally close to final compile, and people giving up and moving out of Scrivener). But if you have a first draft that has been extensively edited, you still need to spend several more months in the creative phase, which will be in Scrivener — this problem can be experienced at any stage of a project.

Scrivener is incredibly powerful in many ways, and though its focus is not on document layout (rightfully so), it is a writers tool, and writers do not always work in a vacuum. Trombone has IMO identified a well known major weakness at the core of Scrivener’s creative workflow that is regularly discussed in these forums — many other posts raise the same issue. Scrivener is brilliant for creative writing, but the way it is designed it is very hard to collaborate and merge collaborators work into your own. To fix this properly (revisioned document system) would require huge resources that L&L do not have, and so the kinds of short cuts to help manual merge (as proposed by Trombone) are where we are left.

12

At the core of this lies the assumption that because very many use Word and Word has a ‘track-changes’ function, Scrivener should learn to handle this. IMO it is just as reasonable to request that MIcrosoft adapts Word to handle this but I guess that no one has sent that request to MS?

So, I must say I agree with DavidR. Maybe one could even say that this is Word’s biggest problem, that it can’t handle Scrivener projects.

Personally I don’t like Word’s track-changes, especially with many involved in the writing. I prefer that editors and colleagues make their comments in a pdf and then it is up to me to change the writing. I don’t “merge” anything.

13

This doesn’t really have anything to do with Word. Trombone’s original post was to do with “smart” import of an edited document with intention of merging the changes made by a proofreader. That the proofreader may or may not use track changes is irrelevant. If the proofreader used LibreOffice, or Mellel, or Nisus: the underlying problem is identical.

14

Yes it does

The original post explicitly mention a doc file and “accept or reject”, i.e. Word.

Requesting that Scrivener should be able to read every other file format is as (il)logical as requesting that all other writing apps should be able to read and write Scrivener projects.

15

Thanks for this thoughtful response. I’ll admit that I do tend to work in a vacuum, which is not always a terribly sensible thing to do. I can see that trying to use Scrivener in a collaborative environment, in which my colleagues do not use it themselves, would be problematic, given the functional characteristics of Scrivener as they’re being discussed here. But I also tend to use Scrivener as a tool for the earlier phases of the creative process rather than for merges or corrections or other adjustments at a later stage. I do think that’s how the software itself was envisioned, though of course this could change as users’ needs change. I guess I’m still stuck a bit on the title of the original post: it doesn’t seem to me to be Scrivener’s biggest problem that it works well as designed, and not well as not designed. I would see it more as a feature request for future redesigns than as a problem in the current design.

But the world is too full of quibbles over what people meant to say or should have meant to say. I lay down my quibbling tool herewith; may it rust in peace! :slight_smile:

16

Lunk: I’m not trying to pick a fight. What Trombone is suggesting is to use embedded codes to allow the document structure to be read for sync back to Scrivener. The codes do not use anything specific to Word; it does not depend on anything related to track changes in word; it does not require Scrivener to read native Mellel format or whatever you think I may have suggested (which I certainly didn’t). Embedded codes are PLAIN TEXT, therefore you could import a text file, mellel file, word file, libreoffice file etc. Even if Microsoft (or Mellel, or LibreOffice etc.) made their app load Scrivener projects, this would not solve Trombone’s (and many other users) problem: how to best get edits made by a collaborator not using Scrivener back into a Scrivener project…

AmberV has given a detailed reply as to the technical challenges of using embedded codes: primarily how fragile they are to user error. Actually because Scrivener’s project format is a bundle, and uses UUIDs and the base format is RTF, it would already be possible for someone to use Python/Ruby/Perl or whatever their preferred poison to write a tool that could take the codes and do this. All we really need is Scrivener to output a UUID during compile. As I understand it, Scrivener 3 will also have expanded Placeholder support so perhaps the basis for an add-on tool to do this is not far away…

DavidR: Yes, Scrivener is clearly not designed for collaborative writing, and therefore I agree that this problem sits outside of Scrivener’s original design scope. I do disagree somewhat with the idea that Scrivener was not designed for final output though — many users get Compile to do what they need. And for those that need more, Scrivener has MultiMarkDown built in which means automating a complex post-production workflow with high production demands is doable (for those who know how to code, or can employ someone to help). That the Scrivener manual itself is built from a Scrivener project (via MMD driving LaTeX) should remind us all that jumping early into Word (which is itself a compromised document layout system), is one of several options, not a necessity for all…

17

If I understand it correctly, all that is needed is for someone to use Phyton/etc on the output from External Sync, write code that stitches it together for editing in an rtf editor and afterwards breaks it up in the original parts so Scrivener can sync it back. Right?

18

lunk: I was originally thinking about a tool to edit the Scrivener project directly (where unique IDs (UUIDs) would be robust), but it may be the case that utilising the existing external sync would be a smarter way to make such a tool! 8) Indeed with external sync, the tool could create its own UUIDs to use to ensure the correct RTF chunk gets back to the correct document. So yes, right!

19

Not quite what’s being requested, but one way of approaching this in Scrivener 3 will be as follows:

  • Scrivener 3 has an option to include link-backs to each document in the exported Word or RTF file. When this is turned on, links are added at the top of each section that link to the document in the Scrivener project.

  • When you import a file containing such links, they get converted to internal links.

  • So, you could import such a file and click on the links to bring up the Scrivener document, making any edits you want to keep in there, then delete the imported file once you’ve finished all the edits. You just work in split screen with the imported document on one side, clicking the link at the top of each section to bring up the original section associated with it.

This provides an easy way of accessing the documents that were associated with the sections in the text. You’ve still got to go through and make the edits you want to accept manually, but that’s inevitable. There’s no good way for Scrivener itself to be able to merge back in such edits, because even a single comma change makes it very difficult for a computer to accurately recognise text.

All the best,
Keith

20

I am very excited to see this feature, and find out whether it solves the challenges I face. I realize I’m late to the discussion, but thought my usage scenario might prove elucidating, both to the developers and to those who can’t imagine why you need to reverse-engineer, returning to Scrivener after the draft is done:

I write novels and screenplays. For novels, the usual Scrivener workflow is great-- you can “write until publication” and then polish everything up in word. Because of how much I love Scrivener, I have gradually begun to do all of my screenplays in Scrivener as well. This works great when I’m working with producers-- just compile to .pdf and they can read, and make notes, which I insert into scrivener. But I have to export to .FDX as soon as we go to production. Often I will get HUGE notes of the “second act needs to be restructured” or “not following the arc of this relationship” kind. Strangely, I find it increasingly difficult to THINK in more traditional, linear word processing documents, including Final Draft (the document navigation features are both clunky and archaic compared to Scrivener…). So in desperation, I will sometimes reimport and split the document back into Scrivener, so that I can “see” it, and work with it. This works okay (I lose my act breaks and snapshot history, and have to recreate them by hand) until we have a locked production script, in which case I need to keep all of my changes in the SAME Final Draft file, which contains the revision history that is required by production…

To be honest, I don’t see any way for Scrivener to solve this problem. It does a remarkably good job interfacing with Final Draft already. And while I don’t think of it as Scrivener’s “biggest problem” (I’m not sure there is one, for me), I do find myself wrestling quite a bit when I get into production. It’s a symptom of how good the program is, really.