Machine Reasoning

A model asked to show its working produces a second output. Whether that output describes the first one is a question about the model, not about the text it wrote, and the two are generated by the same weights under the same pressure.

Nothing in training rewards that second output for being true of the first. It is rewarded for being accepted, which is a weaker property and a much easier one to satisfy.

So the interesting question is not what the explanation says. It is which parts of a run can be changed without changing the answer, and which cannot.

That question has the advantage of being answerable. You edit the run rather than the account of it, and the places where the answer moves are the places that were doing the work.