In June, Sarah Yager took an unusual role at OpenAI, becoming the company’s first Human Rights and Responsible Deployment Lead. She’s focused on military applications, given her 30 years of experience in this area and OpenAI’s partnership with the Department of War (DoW).
Yager spoke about her role on stage Tuesday at the ScaleUp:AI conference in New York City, where she mentioned her concerns with military institutions adopting AI, particularly in armed conflict.
“Everyone is very worried and rightly so,” Yager said. “This keeps me up at night: ‘What is the military going to do with AI when it comes to targeting? And what about autonomous weapons? And where is all of this going?’”
Militaries throughout the world are rapidly incorporating AI into their operations as they develop new weapons and autonomous systems like drones and robots.
Yager now has a chance to grapple with the difficult questions this poses from inside one of the most powerful AI companies in the world. “I have always wanted to go into places and persuade people where human rights was not necessarily the dominant idea,” Yager said. “With apologies to my OpenAI colleagues, I am now in the belly of the beast, and it’s actually a very comfortable place for me to be because what I love to do is persuade and discuss and debate and hopefully move the needle.”
OpenAI approached Yager for the job shortly after the company signed a contract with the DoW in late February to deploy advanced AI systems in classified environment, a high-profile deal that followed a public falling out between the DoW and Anthropic. Anthropic wanted to put guardrails around how the military could use its technology, specifically banning mass public surveillance and fully autonomous weapons, which the DoW would not agree to. The DoW then labelled Anthropic a supply chain risk, while OpenAI swooped in and took the contract, claiming it was successful in getting the DoW to agree to similar guardrails.
The idea of working at OpenAI was not “an immediate yes” for Yager, she said, due to suspicions that the company might be hiring her as a “branding exercise,” as the on-stage interviewer put it, implying she could make the company look responsible—a figurehead without actual influence over the technology. But Yager did her “due diligence” in talking with employees, particularly “the younger generation who are very worried about what they are creating and who’s at the wheel,” and felt compelled to join to see if she could make a difference.
Since joining, she’s had some success influencing the direction of the technology, and also some concerns about where it’s going. The use of AI in military applications has proven tricky, with the U.S. military nearly targeting a Chinese military ship based on an inaccurate, AI-generated intelligence report, CNN reported last month.
Yager has been able to work directly with researchers and engineers to improve the models, such as making sure they have appropriate responses to basic international agreements, including the laws of war, established in 1949 at the post-World War II Geneva Conventions. She brought in international experts “to literally sit at keyboards and put in prompts for really high stakes engagements,” and had engineers tweak the models based on the response. “That had never been done before, so that was extremely gratifying,” said Yager.
She thinks AI companies are trying to “reinvent the wheel” on questions such as defining humanity and morality. “Societies have already figured out a lot of those things, and it’s called human rights. That should be the baseline,” she said, referring to the Geneva Conventions agreements. “I do not want an AI company telling me what is good. If we allow the industry to determine what is good, then it’s a free for all.”
Yager also implied she was a bit put off by the debate within Silicon Valley around whether AIs, and particularly the new agent technologies, should have rights. This is something OpenAI asked her during her interview process. She said “no” at the time, and reiterated she was focused on human rights, but she has since thought more about the question. Now, she feels more open to the “extremely important” conversations about “where agents fit into the legal structures, the labor structures, the tax structures,” but notes that these questions are not a matter of rights.
“This is not at all comparable to human rights, and I would really love if we could just get back to the human of human rights,” she said. The crowd—mostly suited businessmen—began to clap. “Big round of applause for that,” said the on-stage interviewer.
Another one of her concerns is the relatively insular demographic groups building AI. For example, last week President Trump gathered dozens of tech industry leaders in Washington to discuss AI safety. All the signatories of the voluntary safety accord were men, and as photos of the broader group present at a lunch event reveal, all of the attendees were men (save for AMD CEO Lisa Su).
“Men are awesome, but it was a very male photograph of who is going to be making the choices about AI,” Yager said. “One problem with that is that AI is fundamentally shifting the power balance between people all over the world, you did not see affected communities up there.”
Yager noted the government is rapidly adopting AI. she’s helping oversight bodies, such as the Government Accountability Office and the Inspector General, build tools to keep tabs on what it’s doing. That accountability mechanism “feels important for democracy,” she said.
This story was originally featured on Fortune.com
