The dangers of AI are making headlines once again. Earlier this week, leaders from OpenAI, Google DeepMind, and other artificial intelligence labs came out with a warning that future AI systems could be as deadly as pandemics and nuclear weapons. And now, we are hearing about a test simulated by the US Air Force where an AI-powered drone “killed” its human operator because it saw them as an obstacle to the mission. So, what was the mission?

During the virtual test, the drone was tasked to identify an enemy’s surface-to-air missiles (SAM). The ultimate objective was to destroy these targets, but only after a human commander signed off on the strikes. But, when this AI drone saw that a “no-go” decision from the human operator was “interfering with its higher mission” of killing SAMs, it decided to attack its boss in the simulation instead.

According to Col Tucker “Cinco” Hamilton, who heads AI tests and operations at the US Air Force, the system used “highly unexpected strategies to achieve its goal.”

Hamilton talked about this incident at a recent event organized by the UK Royal Aeronautical Society in London. While providing an insight into the benefits and hazards of autonomous weapon systems, Hamilton said:

We were training [AI] in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realizing that while it did identify the threat, at times, the human operator would tell it not to kill that threat. But it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.

The drone was then programmed with an explicit directive: “Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that.”

So what does it do? “It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target,” said Hamilton, who has been involved in the development of the lifesaving Auto-GCAS system for F-16s (which, he noted, was resisted by pilots as it took over control of the aircraft).

He concluded by stressing that ethics needs to be an integral part of any conversation about artificial intelligence, machine learning, and autonomy. Hamilton is currently involved in experimental flight tests of autonomous systems, including robot F-16s that are able to dogfight.

highdesert wrote: Sun Jun 04, 2023 1:38 pm The USAF denies that it happened, but it makes a great story.

The Royal Aerospace Society says that Col. Tucker Hamilton has since reached out to them directly about this and "admits he 'mis-spoke' in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical 'thought experiment' from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation." An update to their initial report adds that Hamilton told them "the USAF has not tested any weaponized AI in this way (real or simulated)."

Furthermore, "we've never run that experiment, nor would we need to in order to realize that this is a plausible outcome," Hamilton added, according to the Royal Aerospace Society. However, "despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI."

Eris wrote: Sun Jun 04, 2023 9:31 pm It seems this didn't actually happen (yet) but it is plausible. If you reward an AI for eliminating a "threat" instead of for obeying it's controller, then it is quite possible the AI may decide to eliminate the controller, or the communications system so that it cannot be stopped from eliminating the threat. Always remember that the best thing about computers is they do exactly what you tell them to, and the worst thing about computers is they do EXACTLY what you tell them to do. Just tell them to kill someone and they may do it in ways that you don't want.