Nice grey life

As Answers Get Cheaper, Questions Grow Dearer

---

This opinion article tackles two much-discussed issues: the impact of large language models (LLMs) on job displacement, and workplace productivity.

The authors begin by making a comparison, likening the effects of LLMs on knowledge-intensive work to that of photogrpahy on early 19th-century artists. While the invention of photography didn’t result in painting becoming obsolete, it undeniably changed the art in a fundamental way. Realism was no longer the goal of painters, as they could no longer compete with photography. Painters then “began experimenting with the subjective experiences of color and light.” For example, no longer limited to copying reality, impressionism adds elements of human feeling to creations.

The authors argue that LLMs make getting answers terribly cheap–not necessarily correct, but immediate and plausible. In order for the use of LLMs to be advantageous to users, a good working knowledge of the domain in which LLMs are queried is key. On average, LLMs increase productivity in call centers by 14 percent, where questions have unambiguous answers and the knowledge domain is limited. However, in environments where a thorough understanding of the situation and critical judgment are key, LLMs can cause prejudice close to 10 percent of the time. Thus, the problem: LLMs are optimized to generate “plausible” answers. If the user is not a domain expert, “plausibility becomes a stand-in for truth.” With this in mind, good questions become strategic, that is, questions that continue a line of inquiry, that expand the user’s field of awareness, that reveal where we must keep looking. They liken this to Clayton Christensen’s text¹ on consulting: “[a consultant’s] value is not in having all the answers, it is in teaching clients how to think.”

LLMs are already game changers (and will likely become more so as they improve). The authors argue that, for much of the 20th century, “[an individual’s] success was measured by domain mastery.” The defining factor is no longer knowledge accumulation, but the ability to formulate the right questions. Of course, the authors acknowledge (it’s even the literal title of one of the article’s sections) that “good questions need strong theoretical foundations.” Knowing a specific domain enables users to “imagine what should happen, anticipate second-order effects, and evaluate whether plausible answers are meaningful or misleading.”

Shortly after finishing this article, I came across a data point that quite validates its claims: a short, informally published paper² on combinatorics and graph theory by Donald Knuth (one of the most respected computer science professors and researchers, and author of the well-known “The Art of Computer Programming” series). Knuth’s text, and particularly its “postscripts,” perfectly illustrate what Choudary and Van Alstyne’s article conveys: LLMs can help a skillful researcher “connect the dots” in very varied fields of knowledge, perform tiring and burdensome calculators, or even try mixing together some ideas that will fail–or succeed. But guided by a true expert of the field, asking the right insightful and informed questions, the answers should prove to be of value–and, in this case, of immense value. Knuth writes of a particular piece of the solution–“I would have found this solution myself if I’d taken time to look carefully at all 760 of the generalizable solutions for m=3”–but having an LLM perform all of the legwork was surely a better use of his time.

¹ Christensen, C.M. How Will You Measure Your Life? Harvard Business Review Press (2017).

² Knuth, D. Claude’s Cycles. https://cs.stanford.edu/~knuth/papers/claude-cycles.pdf

---