I believe most of Apple’s current language tools run locally (nothing leaves your machine), so this feels technically feasible using smaller models without eating all your local RAM.
2 Likes
The problem is that models small enough to run locally generally don’t work all that well. (Not that the big models are flawless, of course.)
4 Likes
I put this question to Copilot. I think the answer advances the discussion:
I asked:
In an online discussion, we’re discussing the possibility of building into Scrivener an LLM AI that would run locally rather than in the cloud. Someone wrote:
“The problem is that models small enough to run locally generally don’t work all that well. (Not that the big models are flawless, of course.)”
What do you think?
It responded:
That’s a great topic, Stephen — and it’s one that’s evolving rapidly. The comment you quoted reflects a common concern, but it’s a bit of a moving target. Here’s a breakdown of the landscape:
Local LLMs: Are They Good Enough?
Historically, yes — smaller models struggled. But recent developments are changing that.
Advantages of Local LLMs
- Privacy: No data leaves your machine — ideal for sensitive writing projects.
- Latency: Instant responses without relying on internet speed.
- Customization: You can fine-tune or control behavior more easily.
- Cost: No subscription fees or API calls.
Challenges
- Performance: Smaller models (e.g. 3B–7B parameters) may lack nuance, coherence, or deep reasoning.
- Hardware Requirements: Even modest models need decent RAM and GPU (or Apple Silicon).
- Context Window: Smaller models often have shorter memory spans, which limits long-form coherence.
What’s Changing?
Recent models like Mistral, Phi-2, Gemma, and LLama 3 (8B) are surprisingly capable — especially for writing tasks. For example:
Model Size Strengths Runs Locally? Mistral 7B 7B Fast, multilingual, good reasoning Yes Phi-2 2.7B Great for writing, trained on textbooks Yes Gemma 2B/7B 2B/7B Tuned for helpfulness and safety Yes LLaMA 3 8B 8B Strong general performance Yes (with good hardware) These models can run on consumer-grade machines with tools like Ollama, LM Studio, or GPT4All, especially if you’re on macOS with Apple Silicon or a PC with a decent GPU.
For Scrivener Integration?
If the goal is writing assistance — brainstorming, rewriting, summarizing, or outlining — then a local LLM is increasingly viable. You don’t need GPT-4-level reasoning for most writing tasks. A well-tuned 7B model could be more than enough, especially if paired with clever prompt engineering and context management.
Would you want Scrivener to offer something like:
- A sidebar assistant for rewriting or summarizing?
- Context-aware suggestions based on your manuscript?
- Character or plot development help?
If so, we could brainstorm what kind of model and interface would best support that.
If L&L does anything with integrating AI with Scrivener, I do hope it is optional and adds no cost to those (like me) who don’t have interest in AI at the current time. DEVONthink did it that way. In any event seems like there is a lot of ways right now for writers to put their documents into any of the local or server AI services now, with no change to Scrivener.
7 Likes
My thought is that they should make any AI ingegration API modular in nature so the users can connect it to their desired LLM.
I’m sure there’s a dozen implementations out there waiting for a chunk of L&L’s profits this might attract.
Link from Scrivener to your favourite LLMs. The way operating systems can nip and tuck apps these days, “windowing” them into little boxes, would offer a similar experience to being part of Scrivener.
1 Like
Hear me out:
LLMs are functions: they take an input – a prompt – and output a screed of text – other LLM types are available.
Using this sort of function is trivial with many extensions. On macOS, for example, you could select text and let, say, PopClip call a function on it, or BTT, or – I’m pretty sure – espanso. I’m sure there are many others. None of these require changes to Scrivener, and you can use them today.
If you are really so keen for this functionality, then perhaps you should roll up your sleeves and do the work.
4 Likes
just saying - I am writing a piece of science fiction and the publisher explicitly says they will not accept AI assisted work. So would any use of a writing app with AI built in (like Scrivener ) be a no-no from the point of view of some publishers? Should using an AI be always a separate decision of serious writers using a serious writing app.
There is no AI in Scrivener whatsoever. (…Which is good.)
4 Likes
Just my opinion, but…
Putting AI into Scrivner is not the real issue. AI is here to stay. Most people will not care if a book was written by an AI as long as it is a good story. Everyone, including publishers, will realize that soon.
If you enjoy writing then write. You don’t need AI. If you are or are planning to have a career as a writer then there are plenty of YouTube videos to tell you how to write with the click of a button and you don’t need Scrivner. That will be the only way to compete when that market is fully realized otherwise you need to rethink your career plans. That’s a hard truth but writing is just another craft getting overtaken by “progress”. Think about woodworking, furniture making, clocking making/repair, etc. There will be others.
I really don’t know if this is a prequel to Doom or Utopia.
I am here to learn to write (without AI) and learn about Scrivner. In fact, I am using AI to teach me how to write but I’m retired and don’t need a career.
That’s all rather depressing, isn’t it?
Eddie
1 Like
I was responding to a suggestion to include such a thing in Scrivener improvements.
A friend of mine makes a pretty good living as a luthier. His clients can definitely hear (and are willing to pay for) the difference between his handmade instruments and what rolls off a factory line. And factory made guitars reach a much higher standard than AI-generated “writing.”
Speaking personally, not as a representative of L&L, my opinion is that AI-generated “writing” is objectively terrible, and that the business model for the companies involved doesn’t come close to justifying the money they’re pouring into it. I expect the entire space to collapse long before “everyone” decides that AI can displace writers en masse. (And I still fail to understand how people think clicking the same button that a million other people can click can possibly serve as the foundation for a “career.”)
From L&L’s point of view, our view of AI is similar to our view of bibliography software. We can provide mechanisms to access third party tools – as for instance we have already done with Apple Intelligence on the Mac – but are unlikely to go further than that.
10 Likes
I made guitars for a while and am familiar with the craft. Yes, there are people making a good living as a luthier. There will always be people who appreciate fine craftsmanship, including writing, who will pay extra but the number of craftsmen who can make a living from them are getting fewer. It will be so with authors.
I think L&L position is right. Let AI do its thing and keep L&L the good writing tool with the necessary for third party interfaces.
1 Like
I agree. However, having spent time on things like the “trending” / “popular” feeds on platforms like YouTube I think it’s not an overstatement to say that the general public have equally terrible tastes and either won’t be able to tell the difference or may even prefer the garbage these tools produce.
If you want to see how bad public taste gets… go to your Instagram account and delete all the saved data about your content preferences (WARNING: this is a terrifying and non-reversible action ). Then click on the little magnifying glass icon to see what gets recommended to you now, based on no info other than what is currently popular. Any residual faith in or hope for humanity you have will dissolve faster than a dismembered limb in hydrofluoric acid.
I hope you’re right, but I think what we’re seeing right now is countries bending over backwards to make the necessary changes to infrastructure, regulation and incentives to ensure these investors can make a return.
I prefer my writing and art without AI… but then I also prefer my movies without CGI and optical viewfinders in my cameras, and it’s hard to find either these days. It may be impossible in 5 years.
Aha! Because people having a career is not only NOT a goal, the objective is the exact opposite! The idea is to get rid of corporate reliance on those damn pesky creative types and their silly egos and unprofitable demands.
I would LOVE to build guitars. I adore guitars and think the idea of building them slowly with hand tools in my shed is the dream; the secret to my finding peace and bliss.
When we moved house, Mrs PF had a number of things she was looking for in a property. She asked if I had any. I had only one: I would like a garage or space for a shed I could build guitars in. Other than that, it was completely her choice.
Guess what the only one of our requirements was that didn’t get met! Yup!
Still, the very wonderful Mrs PF was probably acting out of love, given that I can’t chop an onion without adding extra unintended “protein” to the dish, nor refill my fountain pen without getting ink on the carpet. Who knows what would be left of me by now if I had access to chisels, sanders and routers! 
2 Likes
Tell that to the post I was responding to.
3 Likes
I was thinking that it could be a pretty good ai business model. Instead of Print-On-Demand the customer could just specify the book they wanted with their preferred biases and have the book AI generated and emailed to them. WOD. Write-On-Demand. No authors, publishers or bookstores. Everybody’s happy. Except maybe those who loose their jobs.
. . . . . . . . . . . .
5 Likes
I can’t think of anything worse. As a software engineer, I’m also pretty sure it would also make money. As a writer I do not want to be fed novels tailored specifically for my tastes.
I can fully imagine a world in the near future where people just stop thinking because they never consume media that ever challenges them. I don’t want to be a slave to that machine.
7 Likes
Sadly, both those perspectives (software writer; book engineer) fall under the category of pesky creative whose removal is the goal. Your opinion, like mine, is therefore not valued by the decision makers.
The two perspectives that count to those making the decisions are the corporations and the consumers. If they could find a way to completely remove consumer decision too, they would.
6 Likes
I was going to ask for a source on this, as someone who spends a lot of time with other writers and readers, because I can’t believe this to be true but then @pigfender said:
And it made me sad because he’s probably right and I have absolutely no desire to live in that world. I hope @kewms is right: