Nice grey life

As Answers Get Cheaper, Questions Grow Dearer

---

This opinion article tackles the much discussed issues of Large Language Models (LLMs) both endangering jobs and improving productivity.

The authors begin by making a comparison, likening the current understanding of the effects LLMs are currently having upon knowledge-intensive work to that of artists in the early XIX century, when photography was first invented: they explain that photography didn’t result in painting becoming obsolete, but undeniably changed in a fundamental way. Realism was no longer the goal of painters, as they could no longer compete in equal terms with photography. Painters then began experimenting with the subjective experiences of color and light: Impressionism no longer limits to copying reality, but adds elements of human feeling to creations.

The authors argue that LLMs make getting answers terribly cheap — not necessarily correct, but immediate and plausible. In order for the use of LLMs to be advantageous to users, a good working knowledge of the domain in which LLMs are queried is key. They cite as LLMs increasing productivity on average 14% at call centers, where questions have unambiguous answers and the knowledge domain is limited, but causing prejudice close to 10% to inexperience entrepreneurs following their advice in an environment where understanding of the situation and critical judgment are key. The problem, thus, becomes that LLMs are optimized to generate plausible answers. If the user is not a domain expert, “plausibility becomes a stand-in for truth”. They identify that, with this in mind, good questions become strategic: Questions that continue a line of inquiry, that expand the user’s field of awareness, that reveal where we must keep looking. They liken this to Clayton Christensen’s 2010 text on consulting¹: A consultant’s value is not in having all the answers, but in teaching clients how to think.

LLMs are already, and will likely become more so as they improve, game-changing for society. The authors argue that for much of the 20th century, an individual’s success was measured by domain mastery, but bring to the table that the defining factor is no longer knowledge accumulation, but the ability to formulate the right questions. Of course, the authors acknowledge (it’s even the literal title of one of the article’s sections) that good questions need strong theoretical foundations. Knowing a specific domain enables users to imagine what should happen if following a specific lead, anticipate second-order effects, and evaluate whether plausible answers are meaningful or misleading.

Shortly after I read the article I am reviewing, I came across a data point that quite validates its claims: A short, informally published paper on combinatorics and graph theory titled “Claude’s Cycles”² written by Donald Knuth (one of the most respected Computer Science professors and researchers and author of the very well known “The Art of Computer Programming” series of books). Knuth’s text, and particularly its “postscripts”, perfectly illustrate what the article of this review conveys: LLMs can help a skillful researcher “connect the dots” in very varied fields of knowledge, perform tiring and burdensome calculators, even try mixing together some ideas that will fail — or succeed. But guided by a true expert of the field, asking the right, insightful and informed questions will the answers prove to be of value — and, in this case, of immense value. Knuth writes of a particular piece of the solution, “I would have found this solution myself if I’d taken time to look carefully at all 760 of the generalizable solutions for m=3”, but having an LLM perform all the legwork it was surely a better use of his time.

¹ Christensen, C.M. How Will You Measure Your Life? Harvard Business Review Press (2017).

² Knuth, D. Claude’s Cycles. https://cs.stanford.edu/~knuth/papers/claude-cycles.pdf

---