I cannot reproduce it reliably, but if I fool around with the Metadata, Options, or Replacements section of the interface to edit custom project formats long enough, I often induce a spinning-beachball. Scrivener shows as “Application not Responding” and has to be force-quit. On restarting the app, the project is re-indexed.
I’ve seen the behaviour on two different projects (both very large = 200MB+).
While I can’t get an exact step process to reproduce it, I am emailing you the application “hang” report from console.app (it’s too large to paste!)
The strange thing is that the project size should have no effect on custom Compile formats, because Compile formats are completely independent of the project (even if you save the format into the project rather than making it global, it is just a set of settings that will be applied and contains no data about the project).