Ordinary observation: my two cents on using LLMs

Qian Wang · August 19, 2026

One day, out of curiosity, I sent Claude (Opus 4.8) a question: ‘Is your way of thinking purely reasoning, logical or mathematical?’ It gave me five paragraphs of answer as it usually does. In short, it says mechanically it’s mathematics, in character though it’s more like pattern completion. What caught my attention in the long answer is the sentence saying ‘…when I write out step-by-step reasoning, that text isn’t guaranteed to be a faithful trace of what’s actually driving my answer underneath. It might be closer to a plausible-sounding reconstruction than an honest readout…’

It sounds like the model generates answers, but it does not know why it reaches that answer. Even though we can ask it to give a reason, the reason would be just another answer-generation step rather than actual introspection. Claude called the problem the faithfulness gap, and it told me about two published research papers on this gap: Turpin et al., “Language Models Don’t Always Say What They Think” (2023), and Anthropic’s own follow-up on reasoning models (2025). These two studies did tests by inserting a subtle hint into a prompt that would measurably change the model’s answer, and then they checked whether the model mentions that hint in its stated reasoning. The studies show that the model often does not mention it; instead, it often constructs a different plausible-sounding justification.

However, I checked these two studies myself and found their focus is on the model’s chain of thought, which is the step-by-step reasoning shown in the chat box before it gives the final output answer. My understanding of the faithfulness gap before was about the answer the model gives when it’s asked with introspective questions by the user, such as why do you say so? I am not sure how much the faithfulness gap on the chain of thought matters for a regular user; at least I never read that part. What I would read and sometimes use is the output answer.

I didn’t realise the distinction between the chain of thought and output answer right away, though. When Claude pointed out this faithfulness gap and backed it with research evidence, my immediate reaction was: well then tell me ‘considering this bug, what kind of tasks or questions do you think you would handle better?’ It reframed my question to be what tasks this faithfulness gap matters less for. It’s a fair and more accurate reframe indeed. And below is the shortened version of its answer.

Regarding where it matters less, they are tasks that we don’t have to trust LLMs’ own narrative:

As for the tasks where the faithfulness gap matters more, they are those we are relying on the why it arrives at its answer, not what the answer is.

After reading it, I got excited and felt like I had found something new and useful, thinking I should share it with more people. What is more interesting is that Claude pointed out humans’ reasoning has this faithfulness gap too! It gave an example called confabulation, which is the creation of false memory without intent to lie. It is usually pathological and related to some brain damage. But in daily life, a healthy person may also misremember things that they thought had truly happened. Humans sometimes also struggle to give accurate explanations about their actions. I was intrigued by this human example, so I asked Claude if what happened with humans in the confabulation case was comparable or similar to what causes the faithfulness gap in its generated answers.

However, after sending the question, I just realised how pointless it was. The question can only lead to an answer concerning the faithfulness gap because I am asking it an introspective question about itself, which is exactly one of the mentioned tasks where the faithfulness gap matters more.

However, after checking the two studies: Turpin et al. (2023) and Anthropic (2025), I also realised something neither I nor Claude had noticed. The faithfulness gap concerns the chain of thought while it’s working on the output, as evidenced by Turpin et al. (2023) and Anthropic (2025). But the tasks where the faithfulness gap matters and the example of humans’ confabulation concern the output answers: whether those outputs are trustworthy. The outputs might also have the faithfulness gap, but this point is not established or evidenced in our conversation. This means the whole discussion with Claude was built on a false assumption about the premise.

Whether the chain of thought has a faithfulness gap and whether the model’s introspective self-report is unfaithful are two different questions, and they carry different weights. As an average regular user, I care more about the faithfulness of the output answer. I of course then asked Claude about the current research on the model’s self-report. A short answer to this is: scientists are still not sure about the mechanisms, and tests show mixed results. There is measurable introspective accuracy in some setups (improvable with training), but clear confabulation-style failures in others are also found. The involved factors can be the model, the task and the framing used in tests.

What is the moral of this story? LLMs make mistakes? This is not news. But the mistake here is indeed a different kind from hallucinating. All things Claude mentioned are true and verifiable; I explicitly asked it to provide me with the source of information. It assembled relevant information that fit the content of the conversation and laid those pieces of information one next to the other. It constructed plausible-looking answers to my questions. But the logic is a bit flawed, and I misunderstood it too. I conflated the chain of thought with the output answer, and it followed this confusion, using research on the former to back up an argument about the latter. Subtle distinctions between concepts get glossed over, an argument is formed and defended without second-guessing, and follow-up problems are identified and solved without ever questioning the premise. And yes, humans do this too; similar issues happen to human conversations all the time.

A language model that is trained on an enormous amount of text has absorbed statistical regularities about how words are assembled in language to connect ideas and form arguments. It is able to generate similar text based on the absorbed patterns and provide interpretation of input text through pattern matching. Language used to be the special human way of communicating, which distinguishes us from other creatures. Now it seems not the case anymore, which makes LLMs fascinating and, to some degree, ‘scary’, on the one hand. On the other, language is exactly what a large language model is made for; it is trained with language, it masters languages, and then it produces language. It then makes sense that conversations with LLMs highly resemble human conversations.

So there is no moral to the story. It’s just an ongoing ordinary observation about LLMs from a layman’s perspective and ongoing thinking on how to use this tool without either being too drawn to it or too sceptical about it. Most human cognitive tasks involve language; now we have a tool to assist with those tasks. How can we master the tool without handing our cognitive capacity over to it? How can we improve our cognitive capabilities instead? These are the ongoing questions.

Last edited August 21, 2026