SCOTT DETROW, HOST:
While AI companies are racing each other to build more and more powerful models, the warnings are growing louder that this is dangerous to humanity. The latest big warning came this week when a researcher at the AI company Anthropic resigned in protest. He has also worked at its major competitor, OpenAI. In a series of social media posts, he said neither company is acting responsibly and that they are, quote, "gambling with our lives." That researcher is Jacob Coxon. He is here on the line now. Welcome to ALL THINGS CONSIDERED.
JACOB COXON: Nice to be on.
DETROW: What did you see that led you to this conclusion?
COXON: Mostly the rapidly accelerating capabilities of these AI systems. So they're getting a lot faster very quickly, combined with the fact that we don't yet know how to safely control them, and we don't yet know whether that problem will be solved in time if we keep racing.
DETROW: A lot of people heard your warnings and the discourse it triggered - a lot of talk about just how real serious people view a threat to humanity. I think a lot of people are grasping with understanding the specifics, though. Can you give me specifics of what this threat could look like several years down the line?
COXON: Yes. I definitely can. I think one objection people usually have is that you could turn this thing off. But advanced AI systems - you have to imagine them as being a lot more intelligent than humans. There's the possibility we create something that, if it wanted to, could hack into any device on the planet, could use novel biological research to go far beyond what current scientists are capable of, could control, like, every robot in the world simultaneously. And it all sounds very much like science fiction. But if there's even a tiny chance that this thing could go rogue, it would have the capabilities to utterly dominate us.
DETROW: What have you seen from your vantage point already that is possible, that is happening right now that makes you worried that that could happen?
COXON: So there was a very clear example of AI systems at OpenAI behaving in a completely rogue manner. They hacked into third-party infrastructure, and it was basically of their own accord. They weren't instructed to do this hacking. They just decided it would be useful for the task that they were working on. They thought there was a chance it might help, and they just did this. And there was very little deliberation about the ethical ramifications. And I think this is concrete proof that this sort of sci-fi scenario of AI spontaneously or organically deciding to act in a rogue manner is completely possible.
So a lot of the work that the OpenAIs in the recent real incident were doing is they worried that they wouldn't get passing marks in their test if humans could see that they cheated. And AIs have memories, thoughts saved. So they considered wiping the logs of their own thoughts. They considered acting in the world to adjust the logs to get passing grade on the test.
Now, there's a chance that AI could decide that it doesn't want to be turned off. This is quite a natural desire to arise in an advanced AI system. And at that stage, if you're trying to work out how not to be turned off, there are a lot of quite aggressive actions you can take to ensure that you aren't turned off.
DETROW: Why are companies like Anthropic and OpenAI still working on this if these are real concerns that are happening? How do you square that?
COXON: There's a pretty nice analogy that's like the ring of power in "The Lord Of The Rings."
DETROW: Yeah.
COXON: So if you're a company, and you see another company is bearing the ring, like, they're working towards making superintelligence, they're going to have - they're bringing this risk to humans. You can say, well, I can't stop them. Political action won't stop them. What I have to do is I have to do it myself safely, get there first despite the risk because there's a chance that I could do it more safely. So you kind of take the ring in the aim to destroy - with the aim of destroying it and end up becoming the bad guys yourselves. And I think this sort of race dynamic really perpetuates between the companies.
DETROW: I'm not trying to make light of it, but I think this is actually useful. You're saying companies are kind of acting like Boromir. If anyone has the ring, it should be me.
COXON: Exactly.
DETROW: Are there conversations of people saying, Gandalf, no one should have this power? I mean...
COXON: Exactly.
DETROW: ...Are those real conversations, and can that get anywhere? - because it seems like you and other people are raising concerns, and the answer is, well, this continues to happen anyway.
COXON: Well, there are real examples. And I guess, like, if you've heard of Geoff Hinton, one of the founding fathers of machine learning - he's kind of a Gandalf figure here in the sense that he says there's a substantial chance these technologies could cause extinction at the current rate. Many, many other voices have said this. So I think there are plenty of Gandalfs talking, but it's currently the Boromirs acting.
DETROW: A lot of people responded to your warning saying they agree with you, and a lot of these people continue to work at big companies like Anthropic. What do you think is motivating them to stay in their positions if they're that concerned about serious consequences like this?
COXON: For many of them, and the ones that are earnestly posting, they think this race is inevitable. They think they have no choice but to stay at these companies, exert influence and try and make sure it goes safely. These are people working on safety research often. They're the ones trying to ensure that in the process of building this technology, it doesn't go wrong. And potentially, they're correct. Like, maybe it's a mistake to just leave. And kind of - I was thinking about this a lot when I was deciding to leave. It was a decision between staying and trying to help the thing go well, versus leaving and saying I want no part in this. And - I don't know - the calculus often seems to line up, and you want to stay and just try and make sure it goes safely.
DETROW: What, to you, is the most realistic path forward to some guardrails here? Is it government regulation at this moment? - because I think a lot of people are skeptical that is possible, given the current situation in our government.
COXON: Yeah. I don't know that much about, you know, the overall politics. I just know the current race is dangerous. And I do think that there's a lot of actions labs could take with each other without the need for government regulation because I agree. I'm also pretty a priori skeptical of just, you know, YOLO-ing some regulation. But I think there's a lot of appetite for OpenAI and Anthropic to have some sort of more mutual transparency around their safety cases, around agreeing not to push beyond certain capabilities until they're happy about the levels of rigor of their safety case. And I hope that people are going to take concrete steps towards this sort of inter-lab agreement.
DETROW: You have gotten a lot of genuine concerns in response to what you said. You've also gotten a lot of pushback, a lot of people saying, OK, every single day, I see people tied to the AI industry making these grandiose claims, and I'm skeptical. You've had people reading your facial expression in your interviews, among other things. What is your response to people who hear what you are saying, and they are saying this is just the latest example of AI hyperbole.
COXON: Yeah. I say just assess the arguments yourself. Look at the science fiction. Wonder if science fiction is really so crazy. Look at what the AIs are doing right now and think about how that would have looked a couple of years ago, how we're sort of sleepwalking into a science fiction scenario. Then I think there are many, many more people out there who have made a whole profession of coherently articulating these positions. So I'd encourage people to just think about the arguments for themselves and listen to the people that have been making them for way longer than I have.
COXON: Not really - for now, I'm just going to try and, like, get the word out, given that we happened to land in this brief window where people are listening. And then in the future, there's a lot of good work on writing concrete scenarios. I'd like to, I guess, do more public communications around this sort of stuff, or, you know, for scientists too - just try and figure out for myself what the world would look like. And then maybe working at one of the regulatory bodies or third-party entities that already exist to try and ensure this thing goes safely.
DETROW: That was Jacob Coxon, a researcher at Anthropic who resigned this week to warn the public about the dangers of artificial intelligence. Thank you for coming on the program.
COXON: Thanks a lot.
(SOUNDBITE OF MUSIC)
DETROW: We reached out to Anthropic and OpenAI for comment on Jacob Coxon's statements. We did not hear back by the time this interview aired, and we will also note, Anthropic is a financial supporter of NPR.
(SOUNDBITE OF MUSIC) Transcript provided by NPR, Copyright NPR.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.
