Compile Replacement applied multiple times

OSX: 10.11.5
Scrivener: 2.7 (26106)

Hello.

Scrivener has been excellent for all of my projects, but with one project in particular I’ve hit a hiccup.

I’m trying to compile a .txt file with two line breaks between every paragraph. I’m not sure the best way to do this, but I’m trying to use the Compile Replacement feature, except I end up getting four line breaks between paragraphs. After experimenting with various Regex/non-regex attempts, it seems like Scrivener is applying replacements multiple times.

When I tried the following replacement: c → cX

I’d expect something like this:

What I get is something like this:

And sometimes I even get three X’s, although I’m not sure why. Is this a bug? Is there a better way to put two line breaks between paragraphs?

Otherwise though, Scrivener is an amazing piece of software.

-Flibbles

Yeah, unfortunately that’s a known limitation in the current implementation. Replacements run several times at different phases of compile, so recursive replace:with pairs will get messy. This will be fixed in the future hopefully. It is a tricky problem though since they really do need to run multiple times.

The solution is to trick the engine. If you replace a bare paragraph break with two, then the next iteration turns it into four. So what you want to do is construct a replacement that first converts a single carriage return to one surrounded by placeholders, and then a second replacement that only looks for paragraph breaks surrounded by the placeholder, and converts that to two returns without the placeholder. So something like this:

Replacement 1 (RegEx)
Replace: [^@]\n[^@]
With: @[RETURN]@

For the [RETURN] part, use the Opt-return trick. For some reason the regex engine won’t take \n as a break in the replace field, but it works fine in the search field.

Replacement 2
Replace: @[RETURN]@
With: [RETURN][RETURN]

Leave regex off for this one. I didn’t thoroughly test it, there may be cases where that still ends up iterating over itself again, but in brief testing it seems to work fine.

There is another approach that you could use that may work in some situations. If you don’t mind the look, try reformatting your source text so that there is a visual line of space, using the Format/Text/Paragraph and Line Spacing… menu command, after paragraphs. Then in the Transformations compile pane, enable Convert to plain text, and set the drop-down to “Paragraph spacing”.

The downside is that your source text has to look that way, but on the positive side (over Replacements) you have more control over the matter since you can opt out of inserting an extra space in special situations by simply removing the formatting from the text. For something like a bullet list that might be a better output than an empty line between each.

Your solution will work a lot better than the current crazy-ass solution I’m using now. Too bad about the limitation, but I know how it goes.

Thank you,
Flibbles.

Problem persists on 2.8.1.2 under OS X El Capitan.

Replacement 1 needs a little work, in any case. It truncates the last and first characters of each paragraph. To work around that, the regex should look like:

[] Replace: ([^@])\n([^@])
[
] With: $1@Opt-Return@$2

But I’m still getting two blank lines instead of one (in other words, three returns instead of two). I guess this is an improvement, because I was getting three blank lines before. I also tried “@RETURN@” (verbatim) as a replacement, but it didn’t make a difference.

This has to affect a lot of people moving (or trying to move) to MultiMarkdown processing. I can work around it using vi and awk, but I doubt that’s an option for a lot of Scrivener users… especially on Windows, where you have to install handy things like that manually. And this needs to work right, without external massaging, for MMD->HTML or MMD->LaTeX processing.

Good point, my fault for testing with garbage data. However I don’t think it is possible to solve this problem from that direction. What was causing my solution to “work” is that the second run was culling the extraneous carriage returns which were the adjacent characters being truncated in that scenario. Here’s another take that avoids regex and just compensates for where the engine will fail:

  • The first replacement is straight-forward: one CR replaced with two CRs.
  • This of course results in four CRs, so we simply search for four CRs and replace instances of that with two.

I think that will work, it looks promising.

I think perhaps this issue is being conflated with what you are referring to. If one is truly migrating to a markup workflow then the simplest and best answer is to do the \n doubling on the source text with basic search and replace tools—you can even do that right in Scrivener with Project Replace or on a per-document basis. Nothing fancy needed, you don’t even need to drop to plain-text tools for that (though if one is inclined that is certainly an approach as well).

This is only an issue if for whatever reason one isn’t truly migrating to the workflow and wishes to retain their source material in a word processing suitable format, relying upon dynamic systems in the compiler to attempt to convert that text to something suitable to markup, but other times using that source for rich text based outputs. Scrivener isn’t designed at all for that kind of usage, nor is Replacements, so we run into issues like this (though as I pointed out above there are probably better solutions than messing with CRs manually as that is a problematic solution on other grounds than just Replacements).

My advice, as it always has been, is to just go all in—in the manner most conducive to the software design. Pandoc and Multimarkdown can produce a staggering array of high-quality document formats together, and so for all but very niche situations, there is little reason to retain a rich text source in Scrivener.

Hey, that worked! Thanks much! I was afraid it would end up doubling the returns again, but it didn’t.

The thing is, Scrivener beats pretty much everything out there when it comes to organization. So to me, it makes sense to at least start with Scrivener (using its Markdown integration). Maybe, in the final stages, moving it out to pure Markdown would be more efficient. I’ve been experimenting with Jekyll for processing Markdown; it’s wicked-fast and (at least for HTML-based publishing like eBooks) produces fantastic results. It takes several hours for me to clean up a novel-sized EPUB, produced directly from Scrivener, to meet my admittedly anal standards. :slight_smile: Using HTML out of Jekyll, I don’t have to clean it up. But I haven’t found a Markdown-savvy text editor that has anything approaching Scrivener’s Binder to organize (or re-organize) a book. Tradeoffs everywhere.

What I meant there is to use Scrivener to maintain your pure Markdown text, from start to finish; that is what I have always done anyway. When I create a new project, I start the initial drafting of it using Markdown, I go through various revisions and editing using Markdown, and when it’s all ready to compile or export, I don’t have to make any conversions or use Replacements to coerce the text into another format—it’s already there and everything about the compiler is built around the assumption that the text in the editor is pure Markdown (with a few exceptions, like footnotes and images which are all of course opt-in).

That’s exactly what I meant by the above, if you use Pandoc/MMD Markdown from the very start in Scrivener and then make use of Pandoc’s ePub2/3 generator at the end of the process, you’ll probably end up with a much higher quality starting point than you’d get using RTF → HTML conversion routines in any program, including Scrivener.

Likewise getting to docx/odt for those in a conventional publication workflow is going to produce a much cleaner output.

Well I will grant you that point, that Scrivener’s editor blithely ignores any kind of markup conventions while you’re writing. Personally, and as a matter of taste, I’ve never been overly fond of that kind of stuff anyway, so for me that has never been a major downside to the software—I am perhaps a little old school in that I never had a problem manually typing in LaTeX by hand into Vim, so a couple of asterisks here and there is barely noticeable.

The main constructs where I find myself wishing for some textual automation is table building, but very few dedicated Markdown editors even try when it comes to tables, anyway. My experience there has been to just type it out “ugly” and then use a script to clean it up after it is keyed in. I can do that in Scrivener just as well as any other editor, and for that and a few other “clean-up” type routines, that is what I do (mainly via automation supplied by Keyboard Maestro).

I do get the overall sentiment to be clear. Anyone used to Vim feels bound, gagged and tied to a milling block by comparison in any macOS text engine based editor (and that includes those trendy Markdown editors!). Although I don’t mind writing directly into Scrivener, and for many things I do, there are still many things I start elsewhere and then copy and paste into Scrivener. If one is very tied to the performance of a particular Markdown style editor for drafting initial text, I don’t see why that wouldn’t be a feasible approach for similar reasons. The main pity here is that Apple has cut all cross-communication between software down to the point where editing the contents of a buffer in one program using another is no longer easily achievable—but I digress. :slight_smile:

I maintain an open-source outliner, and added support for a “text” type for its native format and OPML (not an extension, since OPML doesn’t define the values or meanings of the type attribute). My Markdown export recognizes it, so it knows the difference between headings and content. That lets me add snatches of dialogue, and generally more description than _note (which is a non-standard extension). Then I can import it to Scrivener as MMD. I tried it for a book I decided to outline more than I usually do, and it worked great!

I also have Vim hardcoded into my fingers. :smiley: Except I used groff. Never could quite get my head around LaTeX.

Lately, I’ve been experimenting with Jekyll, which (through Liquid templates) extends Markdown with variables, conditionals, and looping. Any of the web developer-centric editors (Brackets, Atom, Webstorm) should work; I’m using Brackets with a few add-ons. I have to number the file names (like 100_Chapter_1.md, 110_scene1.md…) so things stay in order, but the HTML is nearly perfect. I experimented with the same setup for writing documentation at the day job; my conversion script just punted on tables and used HTML fallback… I’ve begun to think we abuse tables a little too much at work anyway. But I digress. I pulled several hanging-fire stories into the Jekyll system, and managed to finish them, yay! But as I said, Scrivener’s organizational capabilities are far beyond it. At least (thanks to you!), I now have a way to move stuff back and forth between them. Sometimes, for me, changing the writing environment can unblock me.

Tables being as awkward as they are in nearly every context they exist within (spreadsheets being the notable exception) often does make me think twice about using them. I think sometimes tables can be a bit of a semantic crutch like bullet lists can be.

As for extending Markdown, I’ve always been a bit rough about it. I just hack together some Ruby scripts to handle custom syntax as a preprocessor, and I also do a lot of extension via the subject of this thread: Replacements (see the Scapple user manual, it isn’t pretty but it gets the job done). The next version of Scrivener, the version I’m in the process of documenting right now, will have even more capabilities for virtually extending markup languages—at the outline (as in outline chunks can themselves become semantic structures) and text blocking levels. Markdown is a great platform, but for a big documentation project I almost always feel the need for a “DSL” of sorts. I look forward to seeing it put through its paces.

As for the main topic of the thread, it’s been said before, but the next version will allow for full hybrid workflows as well. Those that don’t like writing with markup could absolutely avoid it and still output DocBook, groff, LaTeX or whatever Pandoc and MMD can do. The current version of Scrivener had a system designed before all of this got popular, and is definitely built in mind for those that actually enjoy using syntax over font variations and paragraph spacing. It’s a bit hipster that way.