Workaround for Scrivener-LaTeX limitation on macro use within \index?

This is a request for feedback that may surprise those that have read my many messages about using Scrivener with LaTeX. For all the flexibility and capability that Scrivener+LaTeX offers, I have run into a full-stop roadblock that is apparently intrinsic to LaTeX, NOT Scrivener.

For all the flexibility and power that comes with Scrivener+LaTeX, there is a ‘feature’ of LaTeX for which I now believe there is NO workaround. The ‘feature’ in question is how LaTeX treats \index items that include macros. The issue was outlined in a 15 January 1998 document “Tips on Indexing LATEX Documents” by Nelson H. F. Beebe available at https://www.math.utah.edu/~beebe/talks/1998/idxtips.pdf.

As mentioned earlier, the argument of \index is treated to little more than a brace balance check, and written out directly to the .idx file. In particular, any TEX macros present will NOT be expanded. This is convenient, because those macros will be expanded when the .ind file is later read, and their premature expansion would clutter the .idx file and possibly interfere with sorting. (Emphasis added)

This LaTeX ‘feature’ effectively suspends the expansion of all macros within the LaTeX \indexes until the end of the compiling process when the \index is ultimately compiled.

I wanted to build a LaTeX \index that employs a macro that includes a LaTeX ‘Counter’ (similar to Scrivener’s <$n> placeholder but much more powerful) as a way to incrementally follow my use of ‘bounding boxes’ available from the tcolorbox package that uses TikZ. (that’s a mouthful!)

As I learned (and outlined above), a fundamental feature of LaTeX, is that a LaTeX \index does NOT, repeat, NOT, expand ANY macro included in a call to \index, until the end of the compiling process when the \index is ultimately compiled. Full STOP.

The following ‘BBoxIndexmacro includes a call to the \theBBoxnum macro that represents the ‘Counter’ BBoxnum that is incremented by 1 with each sequential ‘bounding box’ I have in my Scrivener project. The \Lemmatext macro includes a title caption that is unique for each ‘bounding box’.

\newcommand{\BBoxIndex}[2]{\index{Lemma!{\theBBoxnum{\Lemmatext}}}}

The following is the second of what I had hoped would be one of many ‘bounding boxes’ in my project, each with a separate reference in my project index.

My idea was to include the title caption “Lemma # 2: Thermal heat energy” that appears at the top of the above ‘bounding box’ in the index with the appropriate hyperlinked page number. Instead what appears in the index is:

image

At the moment, I have 10 ‘bounding boxes’ in my project. The caption at the top of the last (10’th) ‘bounding box’: “# 10: Lies, anti-science, and climate denial” is the only one (of the 10) that appears in the index. This happens as the LaTeX ‘feature’ suspends macro \index expansions until the \index is compiled resulting in the macro expansion of whatever is the last title caption for the ‘bounding boxes’. This results in the macro expansion for the last ‘bounding box’ as #10:

If the LaTeX macros were expanded dynamically during compilation, there would be 10 ‘bounding boxes’ (#1, #2, #3, … #10 ) with their title captions listed in the project index.

I’ve hit a wall and cannot see a way around the LaTeX \index limitation that prohibits the dynamic expansion of the unique BBoxnum and \Lemmatext macros for each ‘bounding box’.

Please note that, given the context of the Scrivener application as it currently stands, I am calling this out as a LaTeX limitation, NOT as a Scrivener limitation (or shortcoming) in any way.

Any feedback, thoughts, comments and/or suggestions appreciated!

Thanks for reading,
scrive
:thinking: :confused:

P.S. For those interested, in mathematics, a ‘lemma’ (as in dilemma) is a minor, relatively-easy-to-prove claim which is often used as a stepping stone for proving major results.

I may well not understand the nature of the problem, as this is something I have never extensively spent much time doing in any of my projects. But it occurs to me that the root of the issue is in trying to do too many things all at once, no? Wouldn’t the dirt simple solution to this problem be to put the index term outside and adjacent of the bounding box environment entirely? Here I’m using a different sequence of macros, but the basic idea should be adaptable to what you’re doing:

\newcommand{\indexbox}[2]{
	\index{#1}
	\begin{callouttip}{#1}
	#2
	\end{callouttip}
}

Hi AmberV,

Thank you for your suggestion, but I am bit confused as to what is being suggested.

My apologies for the question, but I have been working in Scrivener+LaTeX for so long that I am unfamiliar as to what functionality exists (or not) within a Scrivener-only project.

When I tried to test your suggestion in a new, standalone project without LaTeX (using the Non-fiction Essay (Chicago Style) Project template) the Index Key and Index Term options that I am used to seeing on the Style options pop-up, don’t exist in the Essay template. In fact, there is NO Style options pop-up at all that I can see in the Non-fiction Essay (Chicago Style) Project template.

So my question is whether there is a native Scrivener \index option in a Scrivener-only project, or is the ‘\index{#1}’ macro command in your reply represent a LaTeX only feature?

If the ‘\index{#1}’ macro command in your reply IS a LaTeX only feature, then (as best I can tell) the placement of where a \index macro is located will not affect if or when the \index macro is expanded.

As best I can tell (but I could be wrong on this), LaTeX simply passes the argument of any \index macro DIRECTLY through to the typesetter without any expansion until the very end when the index is actually compiled. Therefore the typesetting program will only expand the FINAL macro value when the index is actually compiled, and the typesetter uses the final macro value when the output is printed.

In short, the expansion of any \index macro is NOT dynamic during compilation.

Put another way, if within the body of a project, the value of a particular macro dynamically changes a thousand times during the course of the project, the argument of the compiled \index macro DOES NOT CHANGE over the course of compiling the project. The typesetter therefore only picks up the FINAL value of the macro as it appears in the compiled project.

My current understanding is that what gets passed to the LaTeX typesetter (and ultimately appears in the printed index) after compiling is the FINAL value of the expanded macro, just before the index itself is compiled.

I could be way off on this, and I’d like to be wrong as that would mean I can dynamically change the value of the macros that I am using throughout my project within my \index macros.

I am hoping someone will prove me wrong.

Thanks again for you suggestion,
scrive
:thinking: :pensive:

My apologies for the question, but I have been working in Scrivener+LaTeX for so long that I am unfamiliar as to what functionality exists (or not) within a Scrivener-only project.

In short, nothing at all exists. Scrivener as a piece of software is wholly unaware of LaTeX, and has no code to produce any LaTeX anywhere within it. If you’re referring to the project template, then all of that is a result of configuration, chiefly in four areas of the compile settings:

  • Section Layouts
  • Text Layout
  • Styles
  • Markup

So, if you want to create your own LaTeX output based on section types (see how it handles table and figure outline items) or styles, then these are the places to look for inspiration.

When I tried to test your suggestion in a new, standalone project without LaTeX (using the Non-fiction Essay (Chicago Style) Project template) the Index Key and Index Term options that I am used to seeing on the Style options pop-up, don’t exist in the Essay template.

The native RTF way of working has no form of indexing. I’ve added indexing capabilities to LaTeX-raw and the Markdown-based outputs for LaTeX, OpenOffice and MS Word, all using the same style-based convention.

So my question is whether there is a native Scrivener \index option in a Scrivener-only project, or is the ‘\index{#1}’ macro command in your reply represent a LaTeX only feature?

None of this has anything to do with Scrivener, save for what we do inside the software ourselves—it could just as easily be configured to print {{BANANA}} around the text, instead of \index{BANANA}.

The snippet I provided as an example of what I mean is pure LaTeX. That’s just how you can pass two arguments through a macro. The first argument is the box header, which we use as an index phrase and then as the box header itself, and then the second argument is the box content. The box macro takes two arguments (header and content), so our \indexbox command here is merely a wrapper of the \callouttip macro, existing purely to place that indexing line alongside the box.

And now you don’t have an index call inside the heading which is itself an argument call to a complex box drawing system of untold complexity.

In short, the expansion of any \index macro is NOT dynamic during compilation.

Which is why it is best to use them simply, from what I gather, rather than trying to do too much at once and inserting them into complex macros as arguments. There is no need for that, as far as I can tell. Having the index marker right before the box still brings the reader to that page and lists the page the box is on correctly. You don’t need the marker inside the box drawing code.

I could be way off on this, and I’d like to be wrong as that would mean I can dynamically change the value of the macros that I am using throughout my project within my \index macros.

Yeah I don’t fully understand what it is you are trying to do. And I might not really be offering advice you can use. I just saw what looked to me like making something more complicated than it strictly speaking needs to be. To put it simply:

  • You have a box with heading + content.
  • You want the heading of the box indexed.

That’s it? If so you shouldn’t need the index marker inside the box to do that.

2 Likes

Hi AmberV,

Issue Solved !!!

Thank you AmberV for your most recent reply to my posting. I had given up on finding a solution, but your posting prompted me to have another look!

By utilizing the Scrivener special auto-numbering placeholder that includes a name and a keyword, e.g.: <$n:[name]:[keyword]> in conjunction with the Scrivener placeholder for the document title in the binder that strips all spaces from the title, e.g.: <$title_no_spaces>, I have been able to have the sequentially numberedbounding box’ titles included in my project index, circumventing the LaTeX feature that passes the argument of the \index macro through to the indexer without any macro expansion.

The reason this works is three fold:

First, by using the Scrivener auto-numbering placeholder <$n:[name]:[keyword]> in lieu of a LaTeX Counter, Scrivener converts the placeholder to a number during compilation, prior to typesetting, so what the LaTeX \index macro sees is the dynamic numerical value of the placeholder that is sequentially calculated each time the Scrivener auto-numbering placeholder <$n:[name]:[keyword]> is called with a new [keyword].

This highlights the difference between how the Scrivener compiler treats a Scrivener placeholder versus a LaTeX Counter; they are NOT treated the same by the compiler. The Scrivener placeholder is converted to a value by the compiler whereas the LaTeX Counter is NOT expanded by the compiler, thus affecting what the typesetter ‘sees’ from the compiler.

Second, the Scrivener special auto-numbering placeholder that includes a name and a keyword, e.g.: <$n:[name]:[keyword]> allows me to make multiple calls to the placeholder without sequencing the placeholder (as long as the [keyword] is unchanged). I was not aware of this feature until I reviewed and re-read the description of the placeholders.

Third, I used the <$title_no_spaces> placeholder as the [keyword] when I called the special auto-numbering placeholder that includes a name and a keyword, e.g.: <$n:[name]:[keyword]>. I have at most onebounding box’ for each sub-section in my project, so I was able to make multiple calls to the special auto-numbering placeholder without incrementing the placeholder while still in the same sub-section.

The sequentially numbered ‘bounding boxes’ now appear in the project index. However, the list of boxes in the index is not sorting as I would have preferred, placing the listing for items #10-#13 before items #1-#9.

Any thoughts or suggestion as to how I might correct the sort order for items #1 through #13 would be greatly appreciated!

Thanks for reading,
scrive
:thinking: :pensive:

The sorting is leading-zero sensitive so a quick fix is using 001 to 013. I’m sure there is some weird setting to adjust this somewhere in the baroque heart of the index package, and as is usual for LaTeX, there are probably several different indexing packages (makeidx vs. imakeidx etc.)… Did you see if xindy can sort to your requirements? xindy Manual: Command List

Hi nontroppo,

Thank you for the suggestion. I considered the “leading-zero sensitive - quick fix” of using 001 to 013, but that would require my using a LaTeX macro to pre-append the necessary ‘00’ or ‘0’ to create a proper argument for \index, which I believe would put me back into the initial realm of not having a dynamic \index macro expansion from wherever my ‘bounding boxes’ exist within my project.

I’ve also been exploring the use of LaTeX sort key versus actual field nomenclature using the ‘@(actual) character in the following \index format:

\index{alpha@{\it alpha/}} [1]

There’s not much documentation on how to use the ‘@(actual) character, particularly in conjunction with the index level delimiter, ‘!’ to sort my Lemmas in the correct order. In theory, using the sort key and the ‘@(actual) character indexing nomenclature should work, but the index level delimiter, ‘!’, seems to complicate how they are to be used together.

After some trial-and-error using the ‘@(actual) character indexing nomenclature, I haven’t been able to find the configuration that gives me a sorted \index for my Lemmas in the #1, #2, #3, … #11, #12 … order that I am looking for.

As for ‘xindy’, I happen to already be using ‘xindy’ in my project for another feature, so I’ll also be looking into whether I can use it to give me the proper indexing order for my Lemmas.

Thank you again for the suggestion … let me know if you have any other ideas, or have access to additional documentation on how to implement the ‘@(actual) character when using \index with LaTeX!

Thanks again,
scrive
:thinking: :pensive: :confused:

[1] See “2.3 Index Processing” in https://ctan.ijs.si/tex-archive/indexing/makeindex/paper/ind.pdf

See also “4 Controlling the sort order and fonts” in https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.135.5517&rep=rep1&type=pdf

Hi nontroppo,

Good news! I’ve been able to work with the LaTeX sort key using the ‘@(actual) character along with the index level delimiter , ‘!’ in a trial-and-error fashion to have the \index command correctly order the entries for the ‘Lemmas’ in my project, as shown in the PDF screen capture below.

As I mentioned, there’s not a lot of documentation regarding the LaTeX sort key using the ‘@(actual) character with the index level delimiter , ‘!’, so I had to resort to trial-and-error to find out what works.

One requirement to have the \index macro correctly order the Lemma items in the index was to remove ANY and ALL non-numeric characters before and after the \index sort key:

<$n:lemma:<$title_no_spaces>>

in order to force the \index macro to place the #2 item before the #12 item in the index, for example.

Now I can go back to writing!
Yea!
scrive
:thinking: :relieved: :triumph:

2 Likes