API or Hooks

Scrivener Wish List
1

My dear friend is writing a book in which she is compiling about 1,000 surveys. She needs to keep track of them in separate folders (by topic) in which there may be many entries. She needs to keep track of those in her database as well.

Importing those surveys into Scrivener, applying “quote” styles to all of them, and setting metadata on each one (so she can keep track of things like the record number in the relational database where the surveys are stored) would have been a tedious and time consuming task.

Poking through the Scrivener project “folder” I discovered the XML file and realized I could script the entire import process. I wrote a Perl script to take her tab-delimited export of hundreds of surveys, created binder entries for each one and folders for each topic, and a corresponding .rtf file formatted the way she needed.

This has been incredibly useful for her and has saved her a lot of time.

The minor issue is that, in order to do this, I had to close the Scrivener project, open the XML file, put a custom tag somewhere (I chose IMPORT ) where I wanted to insert all the new binder entries, run the script, then open the Scrivener project.

This worked great.

It would be nice to generalize these Perl scripts so someone else could benefit from them. It would be even more nice if there was a way to “hook” into Scrivener so I could do this while the Scrivener project is open, rather than “shuffling things behind Scrivener’s back” :slight_smile:

A simple way of saying to Scrivener: “Hey, I know you have parsed the XML and built the binder, but I need you to discard that and re-read the XML and regenerate the UI” would be cool.

I love the core philosophy you built Scrivener around: simple rtf files in a directory (rather than some proprietary file format) and some separate files for user settings.

However, in the process I discovered you are using a signed INT for each binder entry. Thus, everyone should be forewarned not to try to exceed 2,147,483,647 items. :wink:

(My friend may approach that limit. It’s a big project).

(I’m kidding) :smiley:

1 Like
2

Some kind of SDK or integrated scripting support are indeed things we’d like to tackle some day. Third-party macro programs like Keyboard Maestro (and AutoHotKey on the PC side) can help with in-program automation, too. I have a number of “features” I’ve added to Scrivener that just manipulate the existing UI in tandem with some Ruby scripts and other system tools.

The open file format does allow quite a bit though. One thing to note, you should have your friend run File/Save and Rebuild Search Indexes (Option key menu variant) on the project after building a lot of content externally.

Yeah, that’s a serious limitation that will be addressed in the future. The Mac’s HFS+ 8 exabyte file size limit is also troubling, but there isn’t much we can do about that. :mrgreen:

1 Like
3

Ah, thank you for that little tip about “Save and Rebuild Indexes” – that’s good to know.

I was wondering how Scrivener would know it had a bunch of new content. :smiley:

1 Like
4

On the road to a true API, these additions would make Scrivener much easier to script.

  • The ability to edit a document by its numeric ID, and the ability to obtain the ID of the doc in the editor. A clever scripter can do this, but it’s a bear.

  • The ability to obtain numeric values for screen geometry. AutoHotKey lets us search for an image fragment – to determine, for example, where to click to interrogate the synopsis. But this technique yields scripts that we can’t usefully share, and that vary with personalization even for the original user.

  • Menu items, with accelerator keys, for every visual control. That’s an accessibility refinement in its own right, particularly as an alternative to Inspector tabs.

  • A fast and reliable way to determine when a new document is active in the Editor. Changing the shade of a single pixel at known coordinates would convey this information; a script could reasonably test for the change three times per second. Perhaps an RGB of the Doc ID? :slight_smile:

Thanks for considering – Jerome

1 Like
5

For scripting, I would like to build a suite of tools that ran either inside or outside of the Scrivener front end – ideally, My two use-cases involve cross-application scripting.

Say that I have a tool which goes through a document (in Scrivener’s case this would be one of the rtf saved document files), and flags use of the words their/there/they’re, to/two/too, etc. Let’s call this script hnym_sniffer.pl. The script has additional abilities to make judgement calls on each usage, to determine if the homonyms are used correctly (“there were only to cars; he added them to the list, too”, it would flag the first, it would ignore the second and third).

[*] From inside Scrivener, I would like to click on a document in the Binder, and then run hnym_sniffer.pl via a button on a toolbar, or hotkey combination, and have the document be marked with comments that call out each use of a homonym, directing me to verify each one, outside the context of “I know what I wanted to write, ergo I see what I wanted to write, in spite of what I’ve written”. The comment markup might need to be done either via API (the script asking Scrivener to add comments to a specific word) or might be done inline in the returned text using some sort of markup, which Scrivener would then parse as it resaved the changed document).

[*] From outside Scrivener, I would like to call a workflow script that queries for files from Scrivener (from the command line), takes the incoming files, calls hnym_sniffer.pl (plus a word-count/frequency script, etc) and then returns the marked up documents (or an array of API call parameter lists, which the bridge script would pass on to Scrivener as API calls to do the markup/commenting)

Note that nowhere do I say ‘applescript’. My exploration of AppleScript is that it is a workflow tool not really designed for heavy text-based manipulation. For example, I could use AppleScript to implement the buttons/hotkey action to run the perl script.

But if I’m doing heavy text pre-processing, I’m doing it in Perl.

1 Like
6

Good to know… I was afraid to try and do things like this because the manual warns that editing the files outside Scrivener could screw up the binder. Are there any more guidelines or advice anywhere for people who want to do things like this? Really cool.

Was just trying to hunt down where the custom meta-data is stored, for example…

Actually, what you’ve described sounds reasonably similar to something I’d like to build. Since I have no experience with Perl, it would be super useful if you felt like making your non-generalized scripts available for me to learn/start from (I’m not a noob to scripting, just Perl).

I would love to see a thread or forum section for scripting, even if it is all “behind Scrivener’s back”. I wonder if there are enough enthusiastic scripters on here. Would be awesome to see a set of DIY Scrivener tool scripts emerge…

7

I just bought Scrivener a few weeks ago so I just stumbled on this thread.

As a developer, my first inquiry to L&L Sales included a request for info about an API, which sadly doesn’t exist. However, there are a few very positive elements at work here.

  1. A large (I believe so anyway) and passionate (no doubt) user/fan base.
  2. An open data format (XML and RTF).
  3. Nothing to stand in the way.

I primarily work with .NET, not exclusively, and I’ve already built a C# library around software that’s structured like Scrivener. The app needs to save the XML data, external code operates on it, then saves, and the app reads it back. If L&L can just make a few enhancements to automate a reload of data changed outside of the environment, some very cool things are possible.

I’ve also done a lot of work to tightly integrate with Word, Outlook, and Exel - the first one being of the most interest here. I saw a video recently where the solution to moving revisioned text from Word to Scrivener was to manually copy/paste everything. I was horrified. If the data is available in a format which can be processed programmatically, I would have no problem doing one-for-one conversions of Word feature to Scrivener. That would include things like section breaks, comments, revisions, and various field codes used for linking text within the document.

I understand that L&L hasn’t gotten around to this yet and that they would probably want to do this on their own. But the reality is that they simply don’t have time and they have other priorities. As with many other communities, from Eclipse to Minecraft, L&L would do well to build an ecosystem around this product that exists outside of their control, completely community-supported. Not only does that take a load off of them in both the short term and long, but it gives them even more bragging rights for Marketing. This is the model for which other companies hunger. Once a small ecosystem of developers emerges, say for a .NET library, others can grow simultaneously and collaboratively, for Perl, Ruby, Python, Java, etc.

Of course L&L would be interested/concerned to make sure this is all done properly and respectfully to protect the brand. That’s fine, as long as it doesn’t go too far in the opposite direction. I’ve seen companies exert too much control over a community that wants to build upon their platform. There is a Not Invented Here mindset, that unless the company builds a feature into their app that the feature shouldn’t exist. They themselves stifle the grassroots spirit and destroy all will. Community effort sometimes moves forward despite the best efforts of the vendor to suppress it, but that’s a battle no one wants to see (again). L&L doesn’t need to accept what the community is doing around them, but they should be careful not to suppress or sabotage it either. There’s nothing wrong with more than one API, and more than one language binding to support some functionality. This happens all the time and usually the best one wins out whatever the source. What I’m saying here is that even if some API doesn’t move forward as some individual at L&L might have done it themselves, it’s OK. Because until the company does it themselves there will still be buzz about what’s available rather than lament about what’s not - which is kinda where we are now.

So here’s my proposal. I’m hoping L&L will create a sub-forum for independent developers, as was suggested some months ago. I’m hoping someone from L&L can volunteer to discuss options with the community, without making commitments or providing code. All we need is some love and encouragement, and a little guidance to make sure we’re not wasting our time.

And who participates? Well, anyone who is motivated. The Free and Open Source Software (FOSS) industry is a graveyard of developers who started with enthusiasm, but then had the joy beaten out of them by ungrateful and demanding users. I feel a lot of that pain. There’s probably more dead FOSS projects out there than live ones. Look at SourceForge, CodeProject, GitHub, CodePlex, and all of the plugins that are no longer maintained for Drupal, WordPress, and every other platform. Many are abandoned for lack of interest by the developer, lack of community participation to provide motivation, lack of users, and/or lack of some kind of funding.

And on that last point, just because software is “free” as in liberty, doesn’t mean it needs to be “free” as in beer. And even if it is, developers need to get “something” for all of their effort. So personally, rather than building yet another community that’s all starry-eyed about doing good things for the benefit of all, I would encourage a model for funding to be built-in to a developer community from the start. “But it should all be Free” I hear people say. Yeah, that sounds nice as long as those folks don’t need to write code or docs. When it comes down to it, developers are people who have other things to do than to provide free labour to the masses. Most developers who are inclined to write FOSS have already done so, been burned, probably don’t want to do it again the same way. There needs to be some kind of reciprocation between supply and demand other than short-term gratitude. For myself, I run a software business, and I’m already working on a lot of FOSS. So my time is saturated. If I’m going to pick up on any project, I need to be motivated by more than warm fuzzies. Don’t get me wrong, I’m not being snotty or capitalistic - I’ve already done this “thing” I’m describing, and I’m trying to convey what works and what doesn’t. Insist on everything being free and the initiative is likely to flounder, but motivate people who do something good for you and the ball keeps rolling.

So when I said above that I think there is a large (quantity) and passionate (quality) user base, I’m thinking that’s justification to write code for utilities where some are free for personal use and others might have a nominal fee for commercial use or extended functionality. APIs and implementations can be completely FOSS but if someone wants functionality they can either write it themselves or they can “motivate” someone else to do it for them. If a developer forum is created here, one of my goals would be to ascertain what kind of demand there is for tools to augment Scrivener usage. Are there a lot of legal sites with end-users? I dunno. Medical? Tech writers? The average “I’d like to write a book” user cannot afford much for new functionality, but if a corporate site can save several hours for several people per week, they would be motivated to pay some amount as an investment toward savings. Maybe they wouldn’t pay for software but they might be inclined to provide a little funding just to keep developers alive to keep writing code - that’s a Very common model … it’s the reason why we get a chance to enjoy so much Free software. We simply don’t know if there is demand like that in for Scrivener without getting some community feedback - or some data from L&L.

Thanks for your time.

( Wow, that’s long, I should have used Scrivener to write that book. :laughing: )

1 Like
8

I can see a need for a back-end/compilation API, as well. I’m studying interactive fiction as a medium for storytelling, and I would love to be able to generate IF when compiling a project. Scrivener is, hands-down, my favorite writing tool, and I’d be willing to put some energy into writing a compile target for, say, z-machine code or Ink or something like that (I’m a software engineer with 30+ years of experience; I can do this).

9

If all you’re looking for is compile-time processing, that already exists in a limited form. You can build your system into the MultiMarkdown framework (which is based on XSLT, so plenty of flexibility there), but since that is itself a command-line tool with an optional shell script infrastructure—where Scrivener ultimately pipes STDIN to a script and waits for STDOUT to dump the results into a file—the sky is the limit what you do in between those two events.

I wrote a bit on the topic in this thread.

10

Thanks! I didn’t think of looking at this. Very cool.