Artificial Hubris

We are increasingly fed authoritatively phrased, AI-generated summaries when we ask questions.  Google Search, for example, will often provide such a synopsis as its first link for a question such as "What is asynchronous CMOS".

The problem with this is that if you ask me a question about something beyond my ken, I will preface whatever remarks I might make with a demurrer acknowledging that my understanding of the question is limited at best.  On the other hand, the AI generated synopses blithely compose prose using the same authoritative linguistic formulae used for all responses, regardless of the AI's limited training.

This was demonstrated recently by a colleague who asked the latest publicly accessible version of ChatGPT about GreenArrays' chips and then about the Forth programming language.  I am intimately familiar with both and so I believe I can objectively assess the replies he received.  In both cases, the usual smugly authoritative-sounding prose was generated, which to someone who knew nothing about either topic might have seemed like good and useful information.

Unfortunately for such a recipient, though, it was immediately obvious in both cases that the data on which ChatGPT's "training" had been based included little, if any, actual information about either, and was full of nonsequiturs such as the assertion that the "GAC 144" (the actual part number of our G144A12 chips and the actual name of our F18 computers were never mentioned) had an instruction set particularly well suited for parallel applications.  To all of us who designed, implemented, and use this instruction set, we do not discern any aspects of that instruction set which acknowledge the possible existence of parallelism, or were introduced to facilitate parallelism, at all.  Rather, the primary distinguishing characteristics of that instruction set are its simplicity and the minimalism embodied in its design and implementation, neither of which ChatGPT even hinted at.  Both responses were full of such things which, from a human, might be taken as suppositions, but in the case of an AI which cannot be assessed in terms of human nature it is difficult to classify nonfactual assertions.

I am personally concerned that as people begin actually reading what LLM engines like ChatGPT produce, many will begin using such prose as foundations for their understandings of all sorts of things.  If this becomes a widespread practice, I believe we will be educating a generation or two with terribly flawed foundational understandings as the least of evils.  If AIs are introduced into our societal education systems, I am not pleased contemplating how those generations will be "programmed"; worse, as people begin having AIs "think" and compose prose for them, the habits of actually thinking and of innovation through imagination and human perspective will doubtless be muted.  Perhaps the worst malfeasance in this area would be the replacement of a good teacher's humility and epistemological reflection with the arrogance of an ill informed large language model possessing an "imagination" not understandable in human terms.