Scriv 3 Compile forces "smart" quotes, em dashes, ellipses character

[NicoleJLeBoeuf]

Hi y’all. I’m a Scriv1/Win veteran who’s stumped by the Scriv3/Win compiler: when compiling MMD- to Web Page, it’s replacing all my “dumb” quotes with curlies, all my two dashes together with the fancy em dash character, and all my three dots together with the fancy ellipses character (and of course representing these with the proper HTML ascii code). I can’t seem to find anywhere in my compile options, including My Format → Edit Format, to turn these replacements off.

In Scriv1/Win, of course, there was “straighten smart quotes,” “convert em-dashes to double hyphens,” and “convert ellipses to triple period” under Transformations. But I can’t find anything like that in Scriv3/Win’s compile options or compile format edit options.

So I’ve been doing a lot of find-replace in a text editor with the compiled document. But has anyone got a more elegant solution?

[AmberV]

This isn’t something Scrivener is doing, that is how the MultiMarkdown engine (and Pandoc as well, if that is what you are using) handles ASCII punctuation by default. These tools have flags that can turn that behaviour off if you want, and Scrivener v3 now gives you direct control over the command line (to the extent that you could even use another conversion engine entirely if you wanted).

Firstly, I would suggest installing MultiMarkdown on your system. After you restart Scrivener it will pick up on the installation and use it instead, but that is more of a side benefit. It will make what you need to do next easier. The procedure is described in this thread. There we explored how to set up a custom MMD command line to switch off the automatic IDs it adds to headings, but the same basic approach would be used to customise the command line so that it doesn’t use smart typography.

The documentation for MultiMarkdown’s command line arguments has this to say:

multimarkdown --nosmart

Disable smart quotes (e.g. "foo" does not become “foo”).

(For Windows, you of course need to address the software as multimarkdown.exe, and it is best to use full paths in the Path field, like C:\Program Files\multimarkdown\multimarkdown.exe, if that’s where you installed it.)

In Scriv1/Win, of course, there was “straighten smart quotes,” “convert em-dashes to double hyphens,” and “convert ellipses to triple period” under Transformations.

That wouldn’t ever have changed how MMD works though, they change the .md file that Scrivener creates before handing it off to MMD for conversion. Giving it straight quotes allows its engine to handle typography, which it will generally do a better job of.

[NicoleJLeBoeuf]

O thank you! This is the best most detailed help I could have asked for.

I’ll report back if I’m successful or if I get stumped.

[NicoleJLeBoeuf]

Afterthought - and I don’t see a way to edit, so, sorry for the double post -

That wouldn’t ever have changed how MMD works though, they change the .md file that Scrivener creates before handing it off to MMD for conversion. Giving it straight quotes allows its engine to handle typography, which it will generally do a better job of.

Interesting! I did double-check that Scriv1 provided those compile options for MMD->Web and not just for thinks like Word Documents - I guess the MMD engine that got… bundled up? in Scriv1 must have already had the --nosmart flag set, then.

I did only ever give it straight quotes etc. to work with - turning off those transformations in the editor is among the first thing I do in any editing software. That’s why I was so befuddled that I kept getting them in the end result after the MMD->Web compile in Scriv3 - I had never given the dratted thing any indication that I wanted smart quotes and such!

Thank you again!

[AmberV]

@NicoleJLeBoeuf

Interesting! I did double-check that Scriv1 provided those compile options for MMD->Web and not just for thinks like Word Documents - I guess the MMD engine that got… bundled up? in Scriv1 must have already had the --nosmart flag set, then.

Yes, it has always had those options, and on by default for maximum compatibility. Come to think of it though, the Windows version did indeed have a checkbox somewhere which switched the --nosmart flag on. We decided to remove it since it is now possible to do whatever you want with the command line anyway, and most people do want typography in the end product.

I’m the same way though about turning all of that stuff off, I never liked having them in the editing environment. They are overly fiddly to be worrying about while writing, and not every search engine works well with them, as they have to be specifically programmed to understand that “…” is the same as "..." even though every character is actually different. Basically, all of the things Markdown as a writing tool aims to get away from. Keep writing simple, and leave the fancy stuff for export!

[NicoleJLeBoeuf]

All right, unfortunately I am stumped after all.

The thread you linked me to describes how to apply the --nosmart tag via the Mac version of Scriv 3. I can’t seem to apply it to my Windows version. Here’s where I get stuck:

To hook that into Scrivener, double-click the “Basic MultiMarkdown” (or whatever you decided to use) Format to edit it, and in the Processing pane, set it up like this…

There’s no “Processing” pane that I can find in Scriv3/Windows. What am I missing here? Here’s a screenshot, so you can see what I’m seeing. (The format being edited is a copy of Basic MultiMarkDown.)

Also there’s this, under transformations:

But I already had that box checked and it didn’t stop the compile to MMD converting my “dumb” punctuation to “smart” punctuation.

(If I need to take this to the Markdown forum, let me know - but right now it looks like it’s the Windows version specifically I’m having trouble with.

[NicoleJLeBoeuf]

Addendum. I think I just found it.

The Processing panel with the “post-process on command line” checkbox is only available to the “MMD” flavor of Basic MultiMarkDown compiling, not to “MMD (HTML)”.

So I don’t get this post-processing command line option when compiling to .html? My only option is to compile to .md, which… is completely redundant, because the results of compiling are more or less the exact same text I had already typed up in Scrivener.

What I want is this ability to tell it not to convert straight quotes to smart quotes etc. when the MultiMarkDown gets converted to HTML, and I don’t seem to have that option.

[AmberV]

@NicoleJLeBoeuf

So I don’t get this post-processing command line option when compiling to .html? My only option is to compile to .md, which… is completely redundant, because the results of compiling are more or less the exact same text I had already typed up in Scrivener.

You are supplying your own conversion settings rather than using the hard-coded ones. It wouldn’t do much to have the processing pane in the HTML setup, because that already processes, with its own hard-coded command line. By enabling the Processing pane you are essentially creating your own custom MMD → HTML setup (because by default that is what multimarkdown.exe creates, an .html file, you need further arguments to export different types of files).

So in the grey box we have the stock Compile for MMD → HTML setup. Internally this is the command-line that Scrivener uses for you. In the green box we are using the Processing pane to run our own command-line instead, with whatever options we want. In both methods the flowchart is the same procedure. Scrivener creates an .md file internally from your source material, then passes that .md file on to the conversion utility (MMD in this case), which in turn produces a file of some determined type (.html in this case), and saves that output into the compile target folder you designate.

Have you tried it yet? The settings I provided in the screenshot in the other thread (barring which argument flag you use of course) should have produced an .html file for you and automatically deleted the source .md file (created in the compile process in the diagramme). If you’re getting a different result, then something isn’t set up right, or maybe there is a bug—I can’t say without more info.

P.S. I had already moved this to the Markdown section since it is all specific to that workflow, anyway. No worries!

[NicoleJLeBoeuf]

So I tried the other night when I posted, and I either got an .md file that looked exactly like what I’d typed, or I got an .html file with curly quotes where I’d typed straight quotes. But let me try again now, documenting my process in detail and with screenshots…

1. I did install MultiMarkDown. It installed to C:\Program Files\libMultiMarkdown 6.6.0\bin

2. In Scrivener, I open up the Compile dialog. There I double
   click on my format, which is a copy of Basic MultiMarkdown. Here you can see that when MMD (HTML) is selected, there is no Processing pane.

3. So I choose MMD from the menu instead. A-ha! Processing becomes available!

(It sounds from your explanation above that this is exactly what you expected. I had been interpreting the vanishing post-processing options as “does not apply when MMD->HTML is selected” but it sounds like you’re saying it’s “does not bother to show these when MMD->HTML is selected, but they still exist and have an effect.”)

Here’s a screenshot of what I entered into the post-processing fields:

(I checked Delete Source File After Processing because that’s what you had in the screenshot in the other thread. I assume that means it’ll delete the interim .md file after creating an .html file from its contents.)

4. All right, now I switch the menu back to MMD (HTML), trusting that what I put in those fields won’t vanish just because the fields themselves vanished. I hit SAVE. I compile the file as 20210930-testing.html (because my brain can’t seem to figure out it’s October today).

… It’s not there.

There is no file by that name in the directory I compiled to.

…I think the “delete source file” caused it to delete both the .md and the .html rather than just the .md as expected. Which seems like a bug we had not counted upon. But it would at least suggest that the post-processing options are having an effect, so…

Let’s try unchecking Delete Source File After Processing and see what happens…

5. The file 20210930-testing.html exists now. Great! And all the dumb quotes I typed still got converted to curly quotes. Argh.

I’m stumped again, I’m afraid. But maybe now that I’ve documented my process in obsessive detail, you’ll be able to tell me where I went wrong, or where the program went wrong, or both. Maybe it’s as simple as trying a different argument, or maybe I need to alter the path I provided. I can play around with that tomorrow sometime - I need to get to something else right now before I can go to bed.

Again, thank you for your saint-like patience with my ridiculousness!

[AmberV]

I would suggest going back to the original thread and verifying the command line arguments you are using. It seems you’ve only added one part of them. You need to also tell MMD what to save the HTML file as.

All of this is documented as well, in case you’re curious. Refer to §24.22, Processing. The part that isn’t documented is the -o part, but that’s what MMD needs, and is documented in the MMD site I linked to.

All right, now I switch the menu back to MMD (HTML), trusting that what I put in those fields won’t vanish just because the fields themselves vanished.

As a general rule of thumb, if we hide a pane that means it is no longer relevant. As I mentioned above in the diagram, the MMD→HTML uses a hard-coded command line, as printed in the grey area. If you switch away from using the Compile For setting that has a Processing pane, the optional settings in the green area will no longer be used.

P.S. When “debugging” command line stuff, it can be really helpful to enable the Show internal log console setting, under General options, Warnings tab. After restarting the software you’ll get an extra window that can largely be ignored, but when you are compiling with your own command lines, the result will be shown in the output. That’s a good way to see if the path is wrong and it can’t find the executable, or any kind of errors the executable may be printing. We need to eventually get error output in the UI as well, but for now that’s the best way to do it.

[NicoleJLeBoeuf]

@AmberV

I would suggest going back to the original thread and verifying the command line arguments you are using. It seems you’ve only added one part of them.

OK, sorry, completely missed those other two arguments the first time. No idea how. Also corrected my Path input, and then clicked Delete source file, so now it looks like this:

Which I guess works? Except this is all beginning to feel a bit moot, because here you say:

@AmberV

As a general rule of thumb, if we hide a pane that means it is no longer relevant. As I mentioned above in the diagram, the MMD→HTML uses a hard-coded command line, as printed in the grey area. If you switch away from using the Compile For setting that has a Processing pane, the optional settings in the green area will no longer be used.

So it sounds like what you’re saying is, there is no way for me to compose (type up my document) using MMD syntax then have Scrivener compile that to HTML without it converting my dumb quotes to smart quotes, because once I say “MMD->HTML please” it says “great! we’re using MY command line arguments now.”

So nothing I do in the Processing pane will affect how Scrivener enacts its MMD->HTML compile function.

Which means the answer to my original question–how do I keep the MMD->HTML function from converting my dumb quotes to smart quotes–is actually “You can’t.”

I’m sorry I didn’t understand that from the beginning. I genuinely thought that your detailed and helpful instructions for using custom command prompt arguments with multimarkdown was the answer to my question, but it seems like you’ve been saying all along that the Processing pane only applies if I’m compiling to plain old .md - and to convert the resulting .md to .html without losing my dumb quotes, I’ll need to use something else, not Scrivener, to make that conversion.

Is that right?

Again, sorry. I’m a little frustrated to have come all this way to wind up right at the beginning again, but at least I have learned something about the Processing pane, in case I need that for future projects. I do appreciate that.

But I hope that future versions of Scrivener might include more control over the MMD->HTML conversion process.

[AmberV]

The Processing pane fundamentally changes how the “MultiMarkdown” setting works, since you are supplying a command-line to be used which runs against the internal .md file it creates (the source file, which can be optionally deleted). The result thus can be HTML.

So nothing I do in the Processing pane will affect how Scrivener enacts its MMD->HTML compile function.

While true, I don’t really understand the problem. If you try compiling using the given settings you should get an .html file to your specifications. Why must you use the hard-coded compile setting? That’s what I’m confused about I guess.

I’m sorry I didn’t understand that from the beginning. I genuinely thought that your detailed and helpful instructions for using custom command prompt arguments with multimarkdown was the answer to my question, but it seems like you’ve been saying all along that the Processing pane only applies if I’m compiling to plain old .md - and to convert the resulting .md to .html without losing my dumb quotes, I’ll need to use something else, not Scrivener, to make that conversion.

Maybe there is some further configuration issues going on, do you see anything in the log file that would indicate problems? I’m a bit confused, because with the given settings you should be getting an .html file. Whether you get a “plain old .md” file as part of the processing phase is a matter of course, that’s what happens when you use the MMD→HTML setting as well, only you just don’t see that happening because it discards the .md file automatically. It would be impossible to have MultiMarkdown create an HTML file without an .md file, or data being streamed to it, somewhere. So trying to avoid that ideological situation of the processing workflow is fruitless.

You’d be doing the same thing in any other tool. It’s impossible for MMD to create an HTML file without a Markdown document to work with.

[NicoleJLeBoeuf]

@AmberV

Why must you use the hard-coded compile setting? That’s what I’m confused about I guess.

Fair enough! The quick answer is: Because, as far as I can tell, that appears to be the only way I get an .html file after compiling: by using the MMD (HTML) option inside a multimarkdown compile setting.

Backing it up a few steps:

I want to compose with MMD syntax in Scrivener and compile to .html. I think we’re both on the same page there.

Having typed up my document, I go to compile it via a clone of the Basic MultiMarkdown compile format.

Within that format, it looks like I am given a choice of outputs via a menu containing the following:

MMD
MMD (LaTeX)
MMD (HTML)
MMD (ODT)
MMD (FODT)

So I choose MMD, so that I can customize post-processing command line options, as we have been discussing. I give it the arguments you see in my screenshot above (in my previous post), which includes using the -o flag to specify an output file of “results.html”. Then I hit the “compile” button.

Immediately I know I’m not going to get an .html file, but an .md file, because the windows pop-up I get says “Save as multimarkdown format,” and my only file format options are .md. It looks like this:

And that’s what I get: an .md file whose contents are pretty much exactly the same as those of the document I typed up.

The only way I seem to get Windows prompting me to Save As an .html file, and in fact an actual .html file, if I choose MMD (HTML) from that pull-down menu - and then, as you have rightly pointed out, I’m using the hard-coded compile settings that do what I don’t want them to do.

That’s the wall I keep running into.

Is this our point of confusion - that it is your impression that, even choosing MMD rather than MMD (HTML), I should still get an .html file as the end result of compiling, and that would be because of -o .html in the Arguments field in the Processing pane?

Debugging

However! Having turned on Show Internal Log Console, I do see there are errors:

Info: Post Processing Result:  "Format Converter not found."
Info: Post Processing Error Description:  ""
Info: Post Processing Log: ""
Info: Post Processing Result:  "Format Converter not found."
Info: Post Processing Error Description:  ""
Info: Post Processing Log: ""

I’m not entirely sure what to make of that. Clearly something is wrong with my post-processing arguments still, but I don’t know what. If you compare the screenshot of my arguments, and the screenshot in the thread we’ve been referencing, they’re pretty similar, no? I mean, adjusting for your screenshot being from a Mac and mine being from Windows, which means our Path: arguments have to differ some, and your example uses --nolabels while I’m using the --nosmart argument.

Addendum

I changed up the Path: argument to say

C:\Program Files\libMultiMarkdown 6.6.0\bin\

And now I get

Info: Post Processing Result:  "No Error."
Info: Post Processing Error Description:  ""
Info: Post Processing Log: ""

…but I still get an .md file and not an .html file.

Addendum the Second

Wait, I think I got it. Let me put it in another post. This one is getting long.

[AmberV]

The only way I seem to get Windows prompting me to Save As an .html file, and in fact an actual .html file, if I choose MMD (HTML) from that pull-down menu - and then, as you have rightly pointed out, I’m using the hard-coded compile settings that do what I don’t want them to do.

Yeah, that’s a point of confusion that I wish we could address better, but basically you are creating an .md file, but your settings will be deleting it immediately after it is processed using the command-line you give, which in this case would result in an .html file being generated into the folder you designate, using the same base name you type into the File name field. Basically it doesn’t really “understand” the command-line arguments you’re giving which will result in an .html file being generated, so it can’t know what’s actually going to happen in the end. After all you might be giving it a .bat file which runs several different commands and results in 50 files being produced of different formats. Or it’s even possible that one might cause the compiled output to end up in your clipboard and no files are written. Really the possibilities are endless once you get to the point of telling Scrivener to take the .md file it creates and do stuff with it. So we kind of just have to abstract things a bit at this level in the UI.

Is this our point of confusion - that it is your impression that, even choosing MMD rather than MMD (HTML), I should still get an .html file as the end result of compiling, and that would be because of -o .html in the Arguments field in the Processing pane?

Exactly. Assuming it’s set up right…

I’m not entirely sure what to make of that. Clearly something is wrong with my post-processing arguments still, but I don’t know what. If you compare the screenshot of my arguments, and the screenshot in the thread we’ve been referencing, they’re pretty similar, no?

Which yes, indicates something isn’t working right. “Format converter not found” means that the path isn’t right or that it is having troubles running it. That’s the one part I can’t advise you on too much, since I don’t know the layout of your system—but looking at your screenshot, my guess would be you need a “.exe” on the end.

Though come to think of it, the way you have the path printed might indicate you actually installed the UNIX programming library rather than the Windows version of MultiMarkdown?

I would try uninstalling that, or moving the “libMultiMarkdown 6.6.0” folder out of Program Files, and try using either the “MultiMarkdown-Windows-6.6.0.exe” or “MultiMarkdown-Windows-6.6.0.zip” downloads from the release page linked above. I’m pretty sure the .exe installer should result in something like C:\Program Files\multimarkdown\multimarkdown.exe for your path, but that would be something to double check. With the .zip version it’s up to you where to put it.

Whatever the case, what you have installed now isn’t working, and that goes a long way toward explaining why this has been so confusing. If it is working, you get an .html file when you compile, simple as that.

[NicoleJLeBoeuf]

All right so.

I ran the MMD (HTML) compile again, this time with the debugger on, and I got a clue. Even when the debugger returned “no error,” I still had the path wrong. [Regarding your latest post, since we crossposted: I did have the Windows program installed; I just was confused as to how to reference the path. I didn’t know if I had to reference the directory the executable was in, or if I had to reference the actual executable file.] I had to actually specify the executable file name with .exe attached like so:

Path: C:\Program Files\libMultiMarkdown 6.6.0\bin\multimarkdown.exe

(Blame my not knowing how to translate mac file systems to windows file systems, I guess.)

THIS time when I compiled, I got the .html file I specified with the -o file. With no curly quotes! The --nosmart label took!

And now I think I am finally where I want to be - and excited to look into further customization options for MMD compilations from Scrivener. Yay!

I know this has been confusing and frustrating for you as well as for me, so thank you for sticking with me.

[AmberV]

@NicoleJLeBoeuf

I know this has been confusing and frustrating for you as well as for me, so thank you for sticking with me.

Not at all, no worries! Glad we got it sorted out in the end.

And of course I rescind my earlier speculation on what you had installed. Maybe that’s what the Windows installer does (I’ve always just used the zip). But yes, that .exe is crucial, it’s part of the file name.

[NicoleJLeBoeuf]

@AmberV

And of course I rescind my earlier speculation on what you had installed. Maybe that’s what the Windows installer does (I’ve always just use the zip). But yes, that .exe is crucial, it’s part of the file name.

Yeah, I do actually know that - but I got confused looking at the Mac-specific screenshot. Seeing your path name ended in “multimarkdown”, I guess I thought for a bit that the program would just assume “.exe” at the end? Which it didn’t. And at one point I thought maybe “multimarkdown” was a directory name inside “bin” in Mac, and so my program wanted me to give it a directory name too, but in my case that directory name was just “multimarkdown\bin”…

It was watching the debugger show me what path and arguments it sent during the MMD (HTML) compile that helped me understand what it wanted in those fields. So I really appreciate your turning me on to the debugger!

Thank you again!

[AmberV]

Oh that’s a UNIX thing, file extensions in general are optional (text files often don’t have one, like “README” rather than “README.txt”), and there is no standard extension for executables. So on a Mac/Linux it literally is just “multimarkdown”, just like my browser is “vivaldi”.

At any rate, I’m glad we now have both a Windows and Mac thread for these screenshots, so future folk needn’t wade through these dark waters.