Judgements about what?
A major challenge for all models, both simple and sophisticated, is that they can only consider the data they have been given, which necessarily is limited to what is (a) measurable and (b) deemed important enough to measure.
1 Like
Specifically, it’s the process for all human judgements which is flawed and can make it a poor predictor of outcomes.
To be clear, the books don’t talk about AI / LLMs at all (at least, not by the point I’ve reached in the book), but rather mathematical or process driven predictors of outcomes as an alternative to human judgement such that Noise (the subject of the book) and Bias (an occasional guest star in the book) can be reduced.
As AI tech tries to deal with and move beyond the constraints of its current limitations, we may find it’s actually better than humans at reaching beyond the existing data set to extrapolate creatively.
EDITED TO ADD: or not? I made an extrapolating prediction, and I’m human (I promise) so the process (and therefore potentially the conclusion) would be flawed. 
Most contemporary neuroscience tells us biological intelligence ultimately reduces to bags of chemicals driving electrical activity in a mess of convoluted biological machinery. But it is also obvious that intelligence is not linearly reducible to sodium, to glutamate or GABA, to glial cells or mylin sheaths. This is all to say that emergence from complexity, whether it is an absurdly complex bag of molecules or an absurdly sohphisticated digital pattern-matcher, is still far from understood. It is possible that something greater than the parts “emerges” from 400+billion parameters neural networks across many layers of interacting elements. Positing emergence can be an easy cop out (we don’t understand a system well enough, and when it does something cool: emergence), yet the concept remains. Brains started off a simplistic input-output reflex managers, and after millions of years of evolution here we are sat with electronic devices discussing intelligence and emergence on the internet.
Intelligent agents like Hinton and others are arguing that more complex outputs than just pattern matching are “emerging” from these high-complexity artifacts, so just because their architecture and training fits a statistical pattern-matcher, that is not the only logical result we might expect.
I read a really negative review of that book that put it low on my reading list. I totally loved “Thinking, fast and slow”, so I probably should revisit its rank on what I try to read next… You are spot on, biological cognition / decision making relies on a bunch of flawed shortcuts, often not available to conscious interrogation. We also totally overvalue individual intelligence and undervalue collective intelligence. In fact this is where I think Hinton’s doomsayer attitudes about AI are more correct. Even if we argue that an LLM is not intelligent, Agentic AI (large networks of connected differentiated individual models) is in full development. Each AI can share parameters, train among each other (knowledge distillation), are starting to meta-cognise (evaluation among different chains of thought). We are constrained in how we share information with each other (podcasts, books, talking round a campfire), but AI swarms will be able to share information in a way we cannot.
1 Like