Set Length of Lines/Break Lines in Markdown and Latex Text Output

Scrivener Markdown & LaTeX
,
1

Hello:

Referring to my earlier questions, and thanks to Amber’s great advice, I now have a pretty decent workflow that allows me to write Markdown in Scrivener to compile it for Multi Markdown or Word Docx using pandoc and some tricks including the script @AmomentOfMusic sent me here (thanks), or LaTeX. Whichever output is good for the time. This is super to keep a master document that is text and separates formatting from the content as much as possible and relatively easily generate any final file format I need.

I might have additional workflow questions, but for now I’m wondering about line length on plain text documents output by Scrivener compile.

Question: For all of the “plain text” outputs (markdown, LaTeX) it seems useful to me to break the line at 75 characters or something (but only on whitespace so that words didn’t break in the middle). Is this possible? Or have I overlooked something and it’s less useful than I think?

Right now my output breaks at the end of a long paragraph which is slightly complicated to work with. My paragraphs insert an extra beak (blank line) to separate paragraphs. I think this is common with Markdown in most cases.

I assumed this might be a scrivener feature already, but I don’t find how to make it happen.It might also be possible to use a well-known post-compile script to break the lines. I say “well-known” because I don’t want to write such a script myself, and I haven’t gone searching for one. If you can provide pretty straightforward instructions on using that Id be grateful. (I have used all my brain power to get the rest of the workflow going and I’m looking for an easy answer.)

The actual question: Can you help me work this out? How can I get a compiler output that breaks paragraphs into shorter lines with a blank line between paragraphs?

I searched in the forums, but if this is answered there I didn’t use good search terms cause I kept coming to answers for other questions. :slight_smile:

Thanks for your thoughts and hints on accomplishing this!

Cheers,
Steve

2

And! Just after I hit send on that question, I tried one more thing. Is the solution to convert the document to plain text in transformations with the option: All whitespace"? That seems to do the trick! :slight_smile:

Is there better option for this?

Thanks for helping me answer my own question :smiley:

Screenshot 2022-05-24 at 16.38.28
Screenshot 2022-05-24 at 16.38.28754×343 43.4 KB

3

If you just mean you would like your text to present in the editor with a maximum column width, then there is a preference for that: Scriiv > Prefs > Appearance > Main Editor > Use fixed width editor. There you can set a preferred line width in points.

If, on the other hand, you mean you want to have a hard return in the text every 75 chars or so, then I am just puzzled why that would be useful.

1 Like
4

Well, It may not be as useful as I think it will be. But I’m collaborating with people who don’t use Scrivener (but who do know how to manage source code). So my thinking is that treating the documents like plain text file (much like source code) is useful. And lines of text that are about 80 characters across (like punch cards lol), might make the text files more manageable to work with in git$(or even vi).

I hope to keep the master documents in Scrivener. Work with collaborators to revise them and merge revisions back into Scrivener. We’ll see how well that works in practice. I could be wrong about how useful it is. Then when it’s time to compile I can use Scrivener (or we can use Pandoc) to produce the publishable object (latex, markdown, or word depending).

5

Those people usually know how to work with a plain text editor (which displays the lines appropriately). :blush: You’re actually making life harder for them with fixed line lengths.

1 Like
6

You know what. Thanks for your comment, but with all due respect I work with the people you are making assumptions about … they are my writing partners. I know our toolset, and I know what’s appropriate for our workflow which is why I asked the question that I did.

This is also a feature of scrivener, so apparently, I’m not the only person that has a use case for it.

7

I’m sorry. I won’t bother you again with my absurd ideas. :slightly_smiling_face:

8

Convert to plain text all whitespace is a good find. I had no idea. As long as there are no single (i.e., not doubled) carriage returns in your source text, looks like you are good to go and should be able to “uncode” those single line breaks later by replacement.

best,
gr

9

Pretty much has me sorted and good to go, I think. :slight_smile: (And I already have the double line breaks where I need them in my source, so good there.)

One Question for the forum: when you use the Convert to plain text all whitespace in compile settings, how does it determine the white space? It seems to me that it uses the line/paragraph spacing setting from the format bar, which feels imprecise and more complicated than necessary for my case. (Have I worked that out correctly?) The other white space features are nice for creating some files (like readmes), and I can live with it. But I really would like to have (or know about) a few ala carte options, like

  • “replace tabs with a single space — or s user defined number of space”
  • “split long lines with single new line—at a user defined line length”
  • Remove extra line breaks (more than too new lines is probably not needed in many cases)

I think I could get fancy and do most of this in styles, but my wish list would be to have compile flags for it.

Despite me wanting more, this feature IS working for me. So thanks, ya’ll. I’m really grateful.

Best,
Steve

10

For items one and three you could easily set something up in the Replacements area of your compile settings.

I can only guess about where the convert setting is getting its idea about where to cut lines, but I was assuming it was just looking to the editor pane. If so, you would get different results with different window widths.

The Scriv pref I pointed to earlier would give you a precise way to set a maximum line length (before word wrap happens) in the editor. The (potential) downside of that is that it is not something being imposed at compile time, but would effect your working environment in Scrivener, too

I wonder if some RegEx maven could contrive to do the whole line break thing as a replacement — literally putting a return at the first space break beyond so many characters. I don’t quite see how you’d do that, but maybe?

11

This may not work for you, but for completeness to this question, for those using Pandoc as the markdown engine for output, it offers options for rewrapping text for text formats Pandoc - Pandoc User’s Guide

--wrap=auto|none|preserve
Determine how text is wrapped in the output (the source code, not the rendered version). With auto (the default), pandoc will attempt to wrap lines to the column width specified by --columns (default 72). With none, pandoc will not wrap lines at all. With preserve, pandoc will attempt to preserve the wrapping from the source document (that is, where there are nonsemantic newlines in the source, there will be nonsemantic newlines in the output as well). In ipynb output, this option affects wrapping of the contents of markdown cells.

--columns=NUMBER
Specify length of lines in characters. This affects text wrapping in the generated source code (see --wrap). It also affects calculation of column widths for plain text tables (see Tables below).

I use this to generate a plain text version of my papers (including a full bibliography, tables, lists headings, and even some equations) which are wrapped per requirements of one colleague, an example of this scrivener output here: scrivomatic/sample-output/workflow.txt at master · iandol/scrivomatic · GitHub

2 Likes
12

The “All Whitespace” setting is documented in the user manual, §24.12, Transformations:

…with the exception of full justification, this mode will attempt to faithfully preserve all whitespace, including alignment, right-indent offsetting, block indentation, hanging indents, and so forth.

So in short it attempts to reproduce the intended look of the rich text formatted content as closely as possible. Bear in mind the originally intended purpose for this feature was the production of standard screenplays to plain-text. The measurement conversion operates on 10-pitch monospace as a basis for this calculation, thus 7.25pts is considered a single horizontal space—hence 29pts is four spaces, or the standard indent for Markdown, triggering code blocks or subsequent material for the preceding block.

What constitutes rich text formatted content is all of the stuff that ordinarily goes into that equation via the compile settings. Editor formatting is where it all begins, but of course what comes out of the compiler can end up looking nothing like that at all. Section Layouts can transform the paragraph layout, so can the settings in the Styles pane. The indent by level Transformation setting right above can override all of that by increasing the indent offset by binder hierarchy (this is how the Markdown Outline compile format can create a nested bullet list). And that, among other uses, is precisely why we have left the rich text formatting controls enabled for Markdown-based output!

Given that, we can clean undesirable formatting from the editor, using the compile format’s facilities for doing so. Unless you really want a code block for example, you probably don’t want any indenting greater than 29pts anywhere. While normally the Override text and notes formatting flag is of little use in Markdown-based processing, here it can be useful for keeping text flush left, and paragraph spacing set to 0pts on top and bottom to avoid inserting extraneous empty lines.

The hard wrap length is 65 characters, which may seem odd, but when you consider the whole 10-pitch equation it makes perfect sense, as that would equate to a 6.5in wide line, which fits neatly into the standard 1in margin on a US Letter page.

In other words this isn’t really a coder friendly setting so much as a printer friendly setting for those using a 10-pitch font. It’s not a bad width for those using text editors though, as modern text editors tend to add a fair amount of gutter space on the left for line numbering, and maybe even a document map widget on the right. So all told an 80 character wide editor with these elements subtracted probably has around 65 characters left for line display. I find it a comfortable line width for reading, at any rate, and probably because it does go back to how many characters you would fit on a line using a typewriter.

Info dump aside, I agree with @nontroppo that if you want hard-wrapping with more flexibility and less complexity, the Pandoc flags are the way to go here! Scrivener’s setup is going to be better if you want single-spaced paragraphs to be properly double-spaced for LaTeX/Markdown output, or to translate visual indenting to code blocks and list nesting. That’s not something Pandoc will be able to help you with, but word-wrapping lines at 75 or 80 or whatever doesn’t need any access to the original or derived formatting.

13

That’s super helpful and I like all the background. Makes “perfect” sense.

I appreciate @nontroppo’s pandoc suggestion a lot. I’m new to pandoc, and it’s been really useful. still discovering everything it will do.

14

Are you compiling to Markdown or to Txt with Pandoc? If you compile to txt it will likely hard wrap your text, if I remember correctly, but not if you compile to Markdown.

1 Like