Replying to
@johntinker@hear-me.social
I appreciate the direction you are taking in thinking about the chatbot as a kind of “language experience.” Since I have built up a fairly large research corpus on AI hallucinations and failure modes, I wanted to add a perspective that may be useful here. What you are reading from the model is indeed fluent and often quite beautiful, but there are characteristic signs that this is linguistic synthesis rather than a research partner reasoning through ideas.
One thing I see repeatedly in my research is cross domain stitching. The model pulls metaphors from philosophy, quantum mechanics, media theory, cognitive science, and social theory, and places them next to each other in a way that flows stylistically but does not commit to the conceptual rigor that any one of those fields would require. The fluency creates the illusion of coherence, but there is no underlying disciplinary grounding. This is simply the model doing what a transformer does, matching and recombining statistically likely continuations of language.
You can see similar patterns in the way the commutator idea is handled. The model treats “the user” as a single type of entity. That kind of flattening is common in generative output because it reflects patterns in safety and general audience explanations, not because it arises from a real theoretical position. The same thing happens with AGI, which is presented as if it were a well defined and unified category, even though many researchers debate whether AGI is even a coherent construct. A human researcher would feel pressure to define terms before making claims. A model feels no such pressure. It is optimizing for plausible sounding language.
The algebra in the commutator metaphor is another example. It is invoked, but never defined. There is no domain, no codomain, no semantics of composition. Again, this is typical of chatbot output. It mirrors the surface structure of technical prose without making the commitments that actual technical work requires.
Perhaps the deepest point is this. The model can generate text that reads as if it has an intention to explain something, but it has no pragmatic aim to be understood. A human writer juggling abstractions would normally be guided by the internal coherence of the argument. A model is guided only by the statistical structure of its training distribution. Fluent language does not imply agency, reasoning, or conceptual responsibility.
None of this is meant as a criticism of the technology. In fact, recognizing these patterns can be comforting because it helps clarify what is actually happening. The real risk is that people mistake verbal smoothness for insight, coherence for grounding, confidence for comprehension. That misunderstanding is exactly the kind of thing that inflates hype cycles and feeds unrealistic expectations. In that sense, I agree completely that LLM is not AGI, and that the gap between the two is often obscured by the qualia of fluency.
PS.
I'd gladly share a portion of my corpus and a few token level language perturbation tools I've build to stress test models, if you'd like to analyze it on your own.