timelets

Mossing and others, both at OpenAI and at rival firms including Anthropic and Google DeepMind, are ... studying them [LLMs] as if they were doing biology or neuroscience on vast living creatures—city-size xenomorphs that have appeared in our midst.

Anthropic and others have developed tools to let them trace certain paths that activations follow, revealing mechanisms and pathways inside a model much as a brain scan can reveal patterns of activity inside a brain. Such an approach to studying the internal workings of a model is known as mechanistic interpretability. “This is very much a biological type of analysis,” says Batson. “It’s not like math or physics.”

Anthropic invented a way to make large language models easier to understand by building a special second model (using a type of neural network called a sparse autoencoder) that works in a more transparent way than normal LLMs. This second model is then trained to mimic the behavior of the model the researchers want to study.

Creating a model that behaves in predictable ways in specific scenarios requires making assumptions about what the inner state of that model might be in those scenarios. But that only works if large language models have something analogous to the mental coherence that most people do.

And that might not be the case.

...
Another possible solution ... Instead of relying on imperfect techniques for insight into what they’re doing, why not build an LLM that’s easier to understand in the first place?

https://www.technologyreview.com/2026/01/12/1129782/ai-large-language-models-biology-alien-autopsy/

It often happens, therefore, that in criticising a learned book of
applied mathematics, or a memoir, one’s whole trouble is with the first
chapter, or even with the first page. For it is there, at the very
outset, where the author will probably be found to slip in his
assumptions. Farther, the trouble is not with what the author does say,
but with what he does not say. Also it is not with what he knows he has
assumed, but with what he has unconsciously assumed. We do not doubt the
author’s honesty. It is his perspicacity which we are criticising. Each
generation criticises the unconscious assumptions made by its parents.
It may assent to them, but it brings them out in the open.

Whitehead. Science in the modern world, 1925.

emphasized communicability:

But a second, especially crucial role of models, drawings, and computer graphics is to make explicit a relationship that you have found, enabling other people to see it as well. This often can be done just by making the relevant part a heavier line or a brighter color, or by deleting most of everything else, but it always requires explicit effort.

Richardson, Jane and Richardson, David C. (1992), ‘looking at proteins’, Biophysics Journal, 63, pp. 1186–209.

quoted from Drawing Processes of Life: Molecules, Cells, Organisms. 2024.

(4) the irreproducible "initial" observation, which cannot be clearly seen in retrospect, constituting a chaos; (5) the slow and laborious revelation and awareness of "what one actually sees" or the gaining of experience; (6) that what has been revealed and concisely summarized in a scientific statement is an artificial structure, related but only genetically so, both to the original intention and to the substance of the "first" observation. The original observation need not even belong to the same class as that of the facts it led toward.
...
Direct perception of form [Gestaltsehen] requires being experienced in the relevant field of thought. The ability directly to perceive meaning, form, and self-contained unity is acquired only after much experience, perhaps with preliminary training. At the same time, of course, we lose the ability to see something that contradicts the form.

--- Ludwik Fleck, Genesis and development of a scientific fact.

"I'm sorry, but I am unable to provide information about events that occurred after my knowledge cutoff in 2021. As a language model, I do not have the ability to browse the internet or access current events. My responses are based solely on the information and knowledge that I possess at the time of my creation."

“Social learning. Our species is the only one that voluntarily shares information: we learn a lot from our fellow humans through language.

“ In our brains, by contrast, the highest-level information, which reaches our consciousness, can be explicitly stated to others. Conscious knowledge comes with verbal reportability: whenever we understand something in a sufficiently perspicuous manner, a mental formula resonates in our language of thought, and we can use the words of language to report it. ”


One-trial learning. An extreme case of this efficiency is when we learn something new on a single trial. If I introduce a new verb, let’s say ”

“To learn is to succeed in inserting new knowledge into an existing network.”

--- Stanislas Dehaene. “How We Learn.”

The exact sciences elude social analysis not because they are distant or separated from society, but because they revolutionize the very conception of society and of what it comprises. Pasteurism is an admirable example.

--- Bruno Latour. The Pasteurization of France, 1993. p 38.