There's a growing understanding in the field that producing human-like texts does not imply human-like cognitive processes. Using the traditional terms like "Artificial Intelligence", "Neural Networks", etc. obscures that fact. (I wish I could up with a new term). We are developing and learning how to co-exist with new kinds of learning entities, the process that rhymes with biology, but is fundamentally different from it in the underlying substrate (what Deleuze would call "risome").
Mossing and others, both at OpenAI and at rival firms including Anthropic and Google DeepMind, are ... studying them [LLMs] as if they were doing biology or neuroscience on vast living creatures—city-size xenomorphs that have appeared in our midst.
Anthropic and others have developed tools to let them trace certain paths that activations follow, revealing mechanisms and pathways inside a model much as a brain scan can reveal patterns of activity inside a brain. Such an approach to studying the internal workings of a model is known as mechanistic interpretability. “This is very much a biological type of analysis,” says Batson. “It’s not like math or physics.”
Anthropic invented a way to make large language models easier to understand by building a special second model (using a type of neural network called a sparse autoencoder) that works in a more transparent way than normal LLMs. This second model is then trained to mimic the behavior of the model the researchers want to study.
Creating a model that behaves in predictable ways in specific scenarios requires making assumptions about what the inner state of that model might be in those scenarios. But that only works if large language models have something analogous to the mental coherence that most people do.
And that might not be the case.
...
Another possible solution ... Instead of relying on imperfect techniques for insight into what they’re doing, why not build an LLM that’s easier to understand in the first place?
https://www.technologyreview.com/2026/01/12/1129782/ai-large-language-models-biology-alien-autopsy/
The biological complexity issue is tricky because we don't want to confuse the complexity of structure with the complexity of behavior. For example, my dog is an extremely complex biological system, but getting/training her to sit is not a big deal. But as we crank up the complexity of behavior, our ability to understand and predict outcomes goes down dramatically.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
