How An “Index-Creating” Algorithm Might Work

By the way, I wouldn’t use the zip file examples in this thread. This is from several years ago, and at this point in time the setup has been further refined, and a part of the official Mac and Windows distribution itself. Look for the the “MMD OpenOffice Document” and “Pandoc Word Document” compile Formats in the left sidebar of Compile, once you select either MMD → ODT or Pandoc → DOCX. The only thing you’ll need to do to set up your project to work with these formats is create a couple of styles in your project (using whatever aesthetics you prefer):

  • “Index Key”: for inserting indexing phrases at the position the index should point to.
  • “Index Term”: for highlighting printed words that should be indexed verbatim.

Use of these will automatically link up with the settings in the compiler.

@Badja: PS Thanks for the ODT work around, but it’s still a work around. Looking at your example now.

That’s not what the phrase means, though. A “work around” is used to refer to a short-term or temporary detour required to avoid bugs and issues in the software.

This on the other hand is demonstrating a core capability of Scrivener: its ability to allow extension of its capabilities, and doing so in a fashion that allows for customisation and adjustment to the individual’s requirements. Although this represents the more advanced end of it, you’ll find that core design philosophy throughout the software, I’m afraid.

For example, for a tool that appears to be “only useful for fiction”, it has a complete lack of features you’d expect to find in fiction-writing software. It also has a whole section of templates for non-fiction writing, including technical writing, which would seem at odds with what you’re saying. In fact, I’ve said it before, but there probably are more actual features in the software for non-fiction writing than fiction writing!

As Bill Bryson once said, any non-fiction book without an index should go straight in the bin.

That really says nothing about software though, does it. If you’re using it as an argument that writing software should be capable of creating books in their final form as well, you’ll be hard-pressed to find anything suitable. Few people would want to use DTP software for writing, for example, and tools that try to straddle the line tend not to excel at anything, like MS Word. You wouldn’t want to make a book with Word, and it’s also only mediocre for large-scale writing. It can get the job done, but so can pen and paper.

Indeed, these days a lot of software aimed at the writing phase itself eschews 99.9% of the DTP model and focuses on using markup for some sort (often Markdown). Scrivener also fits into that category, though it can be used more like how one would write in a word processor, too.

1 Like

Hello @AmberV

I have just tested this.

Selected a group of three words (“theory of equivalence”) in my editor and styled it “Index Term”.

Created a section in the back matter, called Index, and in it, created a style called Index Key

Set compile to MultiMarkdown to ODT

Left column: MMD OpenOffice Document

Filled the two replacement columns: replace and with

RESULT:

In my ODT, when I run my mouse over the words, it says “Alphabetical Index: theory of equivalence”, so something must have happened right

BUT

In my Index section with Index Key style, I don’t have anything.

Did I just miss one step, maybe?

It doesn’t sound like you rebuilt the index field in LibreOffice. That is something you will need to do under default settings. There may be a way to have fields always updating, but I believe it is off for performance reasons. You’re looking for Tools ▸ Update ▸ Update All (or more specifically just update “Indexes and Tables”.

I’d have to double-check, but I’m also pretty sure you need to insert the index as well, wherever you want it.

Created a section in the back matter, called Index, and in it, created a style called Index Key

Wait, I’m a bit confused here. You would be styling the key where the index should take the reader in the main body. It sounds from this description you are trying to manually create an index by hand. The whole idea here is that you can mark indexing terms while you write, and later insert a completely formatted and automatically updating index at the end of the book.

Filled the two replacement columns: replace and with

As a side note, that sounds like the earlier advice. You don’t have to do that any more, the styles handle all of it.

1 Like

Sorry I misunderstood.

Soooo I have just watched a video about indexing on LibreOffice so as to know a little more and avoid wasting your time…

I have deleted what was in “replacement”, since the styles already handle it now, which is great.

I have found how to insert an index and update in LibreOffice.

What I do not understand is how the two styles function TOGETHER.

I suppose that I need to select a word / group of words in the main body and style it “index term”, as I did.

But then I do not understand where I should insert the “index key”.
Also, supposing I have styled the word “equivalence” as “index term”, does the index automatically detect the same word each time it appears in the main body? I suppose it would not make sense to select the word a hundred times to give it the style “index term”.

If it is too long to explain, forget it. You cannot spend your day on this.
It is great if I can build the index directly inside Scrivener, but if not possible, I will just build the index from scratch at the end after compiling. Sorry I am not more competent in these things. This is the first time ever I do this.

Okay great, yeah knowing a bit of how the word processor itself is going to be using this technology will help form the expectations you’d have for what you want to do in Scrivener.

I understand the source of confusion with the styles themselves, no worries! They were not fully described in the bullets, but described more thoroughly in the discussion leading up to it.

So basically the Index Key style is for inserting a key into the text that you don’t want the reader to see as text. Depending on your style of indexing, or even writing, this may be the one you use the most, or even the only one you use. For instance the point where a useful topical word may be found may not often be at the spot that you want the reader to come to if they are interested in that topic. The word itself might be a page after the heading itself, for example.

The “Index Term” style is for those lesser cases where the key you want indexed just so happens to be a naturally used word the reader will encounter as well, and in a spot advantageous to direct them to. It keeps things tidy in the editor if that’s all you need, as you don’t have to have text text like that, where there is a duplicate word alongside.

So, to answer your question they don’t work together in that sense. They both accomplish the same thing: adding a term to the document’s internal index. They give you two tools for accomplishing that, depending on the scenario.

Also, supposing I have styled the word “equivalence” as “index term”, does the index automatically detect the same word each time it appears in the main body?

No, thankfully not! That would result in a rather unusable index, and unlike the OP I don’t even think that kind of mechanical vocabulary word type listing would even be a good starting point to work from. You’d probably spend just as much time cutting down on chaff than you would building up a list of references to the topic, where each reference offers new and unique insight into the topic.

1 Like

Amazing! Thank you sooo much!

I have been testing this, and it works sooo well!

Honestly, I really can’t understand people complaining about Scrivener. It does just about everything needed. All you have to do is ask.
I think you are an amazing team. The idea of using styles to do this is awesome, really, so simple to use and efficient.

… and I completely agree with your last point: not a good idea at all, actually. I realise now that I had not thought this whole thing over, in addition to the technicalities.

Now that I know I can do that, I have a feeling that I should probably keep the indexing for the end instead of doing while I am writing, so I am going to leave this aside for now and go on with the writing.

I just have two questions:

  • If I want to have subentries, is this something that needs to be done later in LibreOffice? Or is it just not possible at the moment?

  • If I want to cross-ref, say, for “translating”, see “translation”, is there a way to do it? Should this be done in LibreOffice afterwards too?

I saw that both these things were possible in LibreOffice, but you normally do it as you select words and add an index entry, so in the present case, I wonder if anything can/has to be done in Scrivener before compiling.

Thank you for everything, really!

3 Likes

Thanks, glad to hear it is working out well for you!

Now that I know I can do that, I have a feeling that I should probably keep the indexing for the end instead of doing while I am writing, so I am going to leave this aside for now and go on with the writing.

You’ll find a lot of different opinions on that matter, conceptual as well as technical. Modern technology makes indexing during composition possible, whereas in the past it was thoroughly impossible until the book was done, because only then could you know the page numbers. So I feel lot of the received wisdom on when to index comes from this legacy, and that there probably aren’t inherent reasons to avoid it in a modern environment. It also was traditionally never done by the author, which is another reason why the question of doing it earlier is a newer one.

There are some good arguments though for saving it to last, in that even if we can dynamically generate an index at will, whenever and at whatever stage, it may still not be possible to know the final flow of information and where it will best be understood in terms of providing an index until fairly late in the writing process.

Whether to start early or wait probably also depends a lot on how you think and write. If you tend to move stuff around and write in a more “shotgun” approach where the final structure is unknown early on—I’d say indexing while writing would probably mean too much overhead. But some people can commit to a structure very early on and stick to it, and they would probably find it just as easy to do both things at once.

… and I completely agree with your last point: not a good idea at all, actually. I realise now that I had not thought this whole thing over, in addition to the technicalities.

If you haven’t already, I’d encourage reading some theory on it. I did some of that before creating the index for the Scapple manual, as there is an art to this task. It also benefits an author to step away from their work while doing it—what we find interesting isn’t necessarily what a reader will find interesting, I mean to say.

I saw that both these things were possible in LibreOffice, but you normally do it as you select words and add an index entry, so in the present case, I wonder if anything can/has to be done in Scrivener before compiling.

I’d have to take a look at the underlying XML that is created from this, to see whether it is possible. I think the main thing will be whether such grouping and aliasing is a function of the key declaration itself, at the point of the key being inserted, or if there is some kind of separate index elsewhere that handles that. If it is the latter, than I don’t think there would be a clean way of handling that without building some additional code.

At any rate that is one of the stated limitations of these styles for now. Maybe when I have some more time to look into it, I’ll see if I can expand its capabilities—but I think for more reasons than one, there should be an expectation that the tools in Scrivener are for getting a rough index in shape, and whether you adhere to the stricter protocol of waiting until it is done, or not, it will probably always be easier to do this once most of the work in Scrivener is complete.

2 Likes

For what it’s worth for those who might want another idea, I have had (for years) satisfactory results with both automated TOC and Index (marked words) using Scrivener’s “General Non-Fiction (LaTeX)” Template. Just works and with LaTeX plenty of opportunity to tweak what it does. The instructions provided by Scrivener and succinct, accurate, and useful.

1 Like

Yes, it’s a lot easier to accommodate LaTeX because most everything you would need to do in your index is driven from how you form the key. Typing in category!subkey, for example is all it takes to do that, and there are also ways of aliasing or creating “See also” entries in a similar fashion.

As for a ToC though, using these Markdown-based ODT and DOCX outputs makes inserting a ToC a piece of cake, as the heading structure is written into the document outline, which is what drives the ToC. In fact with Pandoc->DOCX you don’t even have to do anything but rebuild the ToC index upon loading the file in a word processor, because we’ve wired it up to have Pandoc insert all of the code for one by default. MMD’s ODT method does not have that facility, so you do still need to insert one in the right place—it’s a simple matter to do so though.

I guess what I’m saying is that one method does both for me. I accept and agree that there are numerous ways to make ToC’s, but for me there is value in pressing a few buttons and it all just works. I prefer using my time for writing.

@AmberV Thank you very much for this advice. I have had a quick look at this, which seems interesting. I have downloaded the 80-page pdf to have a closer look later before I start indexing.

https://www.asindexing.org/best-indexing-practices/

I totally hear your arguments about the indexing being done from an external point of view.

When I see if I really need subentries and cross-ref, I will probably come back to the forum to see who is doing what and how.

@rms Thanks. I saw that there were instructions on LaTeX in the manual. I will go and have a closer look. Since the object of the Scrivener manual is not to “teach” LaTeX, that would probably mean more IT learning on my part, and I don’t know if I will have time to do that.

Concerning the ToC, although I haven’t really looked into it, I was under the impression that Scrivener handled it automatically, just from how I structure my binder, so I am not sure I will need any additional work on this.

@rms Thanks. I saw that there were instructions on LaTeX in the manual. I will go and have a closer look. Since the object of the Scrivener manual is not to “teach” LaTeX, that would probably mean more IT learning on my part, and I don’t know if I will have time to do that.

Concerning the ToC, although I haven’t really looked into it, I was under the impression that Scrivener handled it automatically, just from how I structure my binder, so I am not sure I will need any additional work on this.

@AmberV Concerning the ToC, I see that you are talking about compiling Pandoc->DOCX, so I am just wondering: if I am compiling, as I did for the index, MMD OpenOffice document in “MultiMarkdown to ODT”, will it deal with the ToC as well?

1 Like

All the Pandoc option does is insert the ToC for you after the title page, and set the depth. For MultiMarkdown’s ODT you can get the same thing by inserting the ToC yourself in LibreOffice. That is the only difference between the two—the necessary structure for having one, the heading outline itself, is the same as in all Markdown, and if you have your compiler set up to generate headings for you from the binder structure, it’ll all come for free.

We’re talking about two seconds and a few button clicks, especially if you’re importing the content into a prepared template file with styles all set up the way you want them to look and function.

2 Likes

The instructions for using the LaTeX template are provided in any new project created by the template (at top). Read that. Those instructions are a world class example of clear, concise, and correct instructions. Frankly, it’s pretty simple and any informed Scrivener user should have the ability to figure it out. I don’t recall reading anything in the ‘Scrivener Manual’ to get started. I figured it out immediately. I’ve been using it for years now, so of course I’ve tweaked a few LaTeX settings, but the out-of-the-box (last I used it) just works.

3 Likes

Thanks! Will definitely look :wink:

I did most of my tech writing in Word. FWIW, treat index tagging as a kind of SEO. You have to read the readers’ minds and figure out what they need to accomplish to get the boss off their backs and get on with their actual jobs, which has nothing to do with my beautifully crafted documentation. No more, no less.

I suppose the same applies to any non-fiction that acknowledges the existance of an audience composed of people who are not the author or the author’s closest family.

I don’t know of any algorithm that can do that.

Sure, you can write a Word macro that tags every instance of the word “foo” and generate an index from that, but does it really serve the reader’s needs to know that the word appears 2525 times in your 200 page document? Beware of venturing into the alluring horror of field codes. They will rewire your brain and not in a fun way. Spend your time on the mystery of mutating bullet lists instead. Thank me later.

Aside: I just noticed that the forum software does ligatures. Slick. fl fi

1 Like