It’s bad logic. But more than that, it occurs to me that by doing this we indirectly deny the value of our subjective, non-verbal experience. It’s true we may not be able to faithfully report how we reasoned or what exactly we desired or intended or understood. But part of the reason we have terms for these things is because we perceive a distinction between more genuine and less genuine versions of them within ourselves. These qualitative distinctions are how we come to believe that reasoning and belief and intention and desire exist in the first place: because we can perceive them as being authentically present to different degrees in different situations, even if sometimes we seem to be mimicking the real thing rather than really doing it. It’s what licenses researchers to keep trying to study these things in people despite the difficulty of getting an unbiased read. I’m not saying we shouldn’t look for analogous processes in machines. But we should acknowledge that the referent for these terms is grounded in subjective experience. Among humans that has worked fine, because we assume ourselves to share an understanding of what it’s like to have an inner life.