Shortly after graduating from MIT, I went to Best Buy to buy a television. I was wearing an MIT hat as a new proud alum.

As the salesperson walked toward me, he noticed my hat and said, "Clearly there's nothing I can tell you about TVs you don't already know."

I hated to inform him, but I didn't take a single consumer electronics class at MIT. He'd seen the hat and decided I didn't need help because how could anyone that went to MIT not know everything about TVs?

I saw a more consequential version of that fallacy at Cisco.

The company would bring its top engineers (called Distinguished Engineers) together periodically and ask how Cisco should run certain parts of the business like marketing and support. These were technically brilliant people. They had developed many of the protocols behind the internet.

We knew how Cisco products worked. That didn't tell us why a customer bought them, what message would reach that customer, or how to spend a marketing budget. Our title got us invited. It didn't supply the experience to answer the questions.

The mistake was easy to make because the intelligence was real. Nobody had to invent our credentials. They just had to assume those credentials covered the question in front of us.

We do this constantly. We describe someone as smart and quietly drop the rest of the sentence: smart at what?

A person can spend decades developing exceptional judgment in one field and have ordinary, poorly informed opinions about another.

That's what bothers me about the authority we give Dario Amodei (CEO of Anthropic) and Sam Altman (CEO of OpenAI) in the AI conversation today.

Amodei's background is in scientific research and building AI models. His doctorate was in computational biology; his career took him through AI research at Baidu, Google, and OpenAI. That's a substantial foundation for understanding the technology. WIRED profile — https://www.wired.com/story/anthropic-benevolent-artificial-intelligence/

Then we ask how that technology should be introduced to millions of people, what it will do to society, and who should control its deployment. We carry the confidence we have in his technical knowledge straight into his answers.

Those questions require knowledge of complex human organizations. A process designed to protect the public can end up protecting the organizations best equipped to navigate it. Understanding the internals of a model doesn't teach you how those systems behave.

In my view, Amodei and Altman are underqualified to make that call for society. Running Anthropic gives Dario relevant experience with his own organization. It doesn't establish expertise in how a technology will spread through thousands of very different ones.

GPT-2 offers a small example of the leap. Amodei urged OpenAI to delay releasing the full model in 2019. Recalling his reaction, he told WIRED: "This is going to determine the fate of nations." Technical achievement and judgment about its social consequences were already getting mixed together.

His current proposal calls for outside evaluators embedded in frontier labs and government-backed coordination on the pace of development. Amodei's proposal — https://darioamodei.com/post/we-must-pace-the-frontier

To me, it reads as naive about how complex human systems work: the evaluators would have incentives and institutional pressures of their own. If he understands those dynamics and wants rules that favor the largest labs, I'd call that regulatory capture. I don't know his motive. Neither possibility is resolved by reminding me how good he is at building models.

I recognize the mistake because we made it at Cisco. Our technical judgment had earned trust. People extended that trust to questions we hadn't earned it on. Having an opinion about marketing was easy. Knowing whether it was any good required experience we didn't have.

And the mistake runs in both directions. The person being asked can start believing that every invitation reflects a qualification. The people asking can stop looking for someone who has actually done the work.

I had to draw that line myself at a client's all-hands meeting. I was giving an introductory AI talk to more than 200 people. When I opened it up for questions, the first hand went up eagerly. The question was about AI's impact on the water supply.

I know AI extremely well, and I drink quite a bit of water. That didn't make me qualified to answer.

So I begged off. I told them I didn't feel qualified to address it. The question had "AI" in it, but answering it required expertise I didn't have.

Before accepting someone's judgment, I want to know what in their background prepared them for this particular question. A brilliant answer to a different question doesn't count.

At Best Buy, I needed someone who knew the TVs.

Two things I'd genuinely like back from you. First, hit Reply and tell me which part of this matches what you're seeing. Second, if there's someone in your world who should be on this list, forward it to them.

- Robbie