Stay in Command

Unpacking Australia's Arms Industry: Secrecy, Influence and Autonomous Weapons

John Rodsted — Fri, 02 May 2025 09:00:00 +1000

Episode Summary

In this eye-opening episode of the SafeGround Podcast, host Lilly McKenzie examines Australia's covert arms industry, revealing how foreign multinational defense companies dominate and influence government decisions. Featuring insights from expert researcher Michelle Fahy and Australia Stop Killer Robot coordinator Matilda Byrne, we explore long standing secrecy, controversy over weapons exports, and the role of Australia within the global arms trade. The discussion also highlights the rise of autonomous weapons systems (LAWS), the ethical and legal risks of AI-driven warfare, and the urgent need for international arms control treaties. Discover why transparency, defense regulation, and arms trade accountability are vital for safeguarding democracy, human rights, and peace in the era of military robotics.

Keywords: arms industry, autonomous weapons, LAWS, AI in warfare, Australian defense, arms trade, international arms control, military robotics, transparency, defense exports, arms influence, climate security, human rights, disarmament, military technology, AUKUS, weapons manufacturing, global security, government accountability, war ethics, stop killer robots.

Episode overview

Introduction and Episode Overview [01:16]

Background on Australia's Defense Industry [01:39]

Influence and Secrecy in the Arms Industry [04:20]

Foreign Ownership and Sovereignty Concerns [10:42]

Employment and Economic Impact [19:43]

Impact on Democracy and Public Accountability [23:21]

Corruption and Conflicts of Interest [27:52]

Growing autonomous military capabilities and Australia’s landscape [31:31]

International Collaboration and AUKUS [36:27]

Australian arms companies and autonomous systems development [38:46]

Ethical, Legal, and Security Concerns of Autonomous Weapons [41:07]

Australia's Position on Disarmament Treaties [44:42]

Civil Society and Advocacy Movements [46:18]

Conclusion and Call to Action [48:35]

Contact your local MP!

Attention Parliamentarians: The lack of transparency in Australia’s defense research and arms exports, especially regarding autonomous weapons, is a critical issue. It hampers accountability, risks misuse and corruption, as well as undermines international disarmament commitments. We urge you to advocate for clear, public reporting & responsible arms control policies. Your leadership can shape a more transparent, peaceful future for Australia and the world."

Thank you for listening!

If you have questions or concerns please contact us via info@safeground.org.au

If you want to know more look for us on Facebook, Twitter and Instagram Australia Campaign to Stop Killer Robots or use the hashtag #AusBanKillerRobots.

Become part of the movement so we ‘stay in command’!

For access to this and other episodes along with the full transcription and relevant links and information head to safeground.org.au & https://safeground.org.au/category/dispatches/podcasts/

Link to the International Stop Killer Robots campaign: https://www.stopkillerrobots.org

Our podcasts come to you from all around Australia and we would like to acknowledge the Traditional Owners throughout and their continuing connection to country, land, waters and culture.

Stock audio provided by Videvo, downloaded from www.videvo.net

Limits on Autonomous Weapons - ICRC Perspective

John Rodsted — Wed, 03 Mar 2021 09:00:00 +1000

In this episode of Stay in Command we showcase the ICRC perspective on autonomous weapons and the need for limits. Our guest is a weapons and disarmament expert, Neil Davison from ICRC's Department of Law and Policy at Geneva HQs bringing his own insight and the ICRC's view.

Content in this Episode:

ICRC mandate and approach to weapons issues [1:14]

Humanitarian concerns of autonomous weapons [2:27]

Legal Issues [5:40]

Human Control as a core concept [8:14]

Ethical perspective [10:36]

Limits on autonomy [12:28]

International talks and 'crunch-time'[16:59]

If you have questions or concerns please contact us via info@safeground.org.au

If you want to know more look for us on Facebook, Twitter and Instagram Australia Campaign to Stop Killer Robots or use the hashtag #AusBanKillerRobots.

Become part of the movement so we Stay in Command!

For access to this and other episodes along with the full transcription and relevant links and information head to safeground.org.au/podcasts.

Our podcasts come to you from all around Australia and we would like to acknowledge the Traditional Owners throughout and their continuing connection to country, land, waters and culture.

Stock audio provided by Videvo, downloaded from www.videvo.net

Diplomatic Process and Progress on Killer Robots

John Rodsted — Sun, 04 Oct 2020 18:30:00 +1000

In this episode of Stay in Command we discuss the diplomatic process and progress being made regarding lethal autonomous weapons systems. The episode featured Elizabeth Minor from UL-based NGO Article 36 which works to to prevent harm from weapons through stronger international standards and is on the steering committee of the Campaign to Stop Killer Robots. We look at how this issue has progressed and where it is going. We unpack key themes in the current debate and where the process must go.

Content in this episode:

Overview of this issue on the diplomatic stage to date [00:01:56]

Human Control Unpacked [00:09:05]

Human control in international debates[00:12:57]

Notion of the so-called 'entire life cycle' of the weapon [00:14:50]

False solution of proposed techno-fixes [00:18:35]

Limitations of Article 36 Weapons Reviews[00:20:50]

Debates around definition [00:25:58]

Going beyond the guiding principles[00:29:28]

Arriving at ‘Consensus recommendations’[00:30:31]

How a Legally Binding Instrument might look [00:33:26]

How we get there-diplomatic avenues[00:37:06]

The need for decisive action and leadership[00:38:42]

If you have questions or concerns please contact us via info@safeground.org.au

If you want to know more look for us on Facebook, Twitter and Instagram Australia Campaign to Stop Killer Robots or use the hashtag #AusBanKillerRobots.

Become part of the movement so we Stay in Command!

For access to this and other episodes along with the full transcription and relevant links and information head to safeground.org.au/podcasts.

Our podcasts come to you from all around Australia and we would like to acknowledge the Traditional Owners throughout and their continuing connection to country, land, waters and culture.

Stock audio provided by Videvo, downloaded from www.videvo.net

Who is in Command?

John Rodsted — Thu, 01 Oct 2020 00:00:00 +1000

Who Is In Command?

John Rodsted: [00:00:00]

Content in this episode:

Today we speak with Paul Barratt AO, Australians For War Powers Reform [00:00:19]
An Insider To Policy and Decision Making [00:03:04]
What Role Does Opposition In Parliament Play? [00:04:36]
The Australians For War Powers Reform [00:05:38]
Is The Response Time An Issue? [00:06:56]
Will Politics Get In The Way? [00:08:25]
How to Keep Intelligence Secret? [00:09:39]
Who Supports War Powers Reforms? [00:11:37]
An Artificial Intelligence Arms Race? [00:14:56]
Legal Framework In Decision Making [00:17:14]
Bias & Lack of Cultural Knowledge and Sensitivity [00:17:55]
An Algorithm Mess [00:20:58]
Responsibility For Picking Up The Tab? [00:24:03]
Will Killer Robots Be Used If We Get Them? [00:24:56]
Weapons and Proportionality [00:25:53]
Campaign Vigorously to Outlaw Fully Lethal Autnomous Weapons[00:26:55]
The Role of Universities [00:29:14]
Trusting The Prime Minister? [00:31:07]
General info about SafeGround podcasts, Acknowledgement of traditional owners & Stock Credits [00:32:48]

If you have questions or concerns please contact us via info@safeground.org.au

If you want to know more look for us on Facebook, Twitter and Instagram Australia Campaign to Stop Killer Robots or use the hashtag #AusBanKillerRobots.

Become part of the movement so we Stay in Command!

For access to this and other episodes along with the full transcription and relevant links and information head to safeground.org.au/podcasts.

Today we speak with Paul Barratt AO, Australians For War Powers Reform [00:00:19]

Welcome to SafeGround, the small organization with big ideas. I'm John Rodsted. Today. We're speaking with Paul Barrett. Paul has had a long career in Australia's public service since 1966, but what distinguishes him from many others within government and the public sector is his strong conscience.

He's held many senior roles within government, notably within the department of trade, primary industries in energy and the business council of Australia and Secretary of the Department of Defence from 1998 to 1999. It was his senior role in the Department of Defence that put him at odds with the government positions and policy.

This led him to leaving the public service. Since then, he's had a very strong voice on how and why Australia goes to war and the powers that a few have to commit us to war. He's also one of the founders and current president of Australians for War Power Reform. Welcome Paul.

Paul Barratt: [00:01:12] Morning John

John Rodsted: [00:01:13] Originally you studied physics and graduated with honours from the

University of New England. How did you go from serious science to Australia's public service and Department of Defence?

Paul Barratt: [00:01:23] Well, John, throughout my, undergraduate career, I was intending to do a PhD in physics and become an academic physicist. And towards the end of my honors year, I read this interesting little advertisement in the Sydney Morning Herald, I had sort of had a rough rush of blood to the head and joined the public service.

And that interesting little advertisement said the department of defence was looking for people to monitor scientific developments of defence interest in the Asia Pacific region. So I thought that sounds interesting. And I applied for it months later and security clearances later, and what have you? I turned up for work and discovered that the scientific developments of defence interests were China's nuclear program. And so that, that launched me on a very interesting, couple of years in the intelligence community. And it was a time when China's, program really was nice and they just had their third test when I started and the cultural revolution was just beginning. So it was a very interesting time in Chinese history and in the history of our region.

John Rodsted: [00:02:31] When you entered the department of defence in 66, it was right in the early days of Australia entering the Vietnam war. You were in the department of defence during the war. How were Australia's policies and actions shaped, and then by who?

Paul Barratt: [00:02:44] The policy to go into Vietnam was shaped very much by the prime minister Menzies himself. And, I was in the fortunate position of being just one year too old to be called up in the first draft for Vietnam. but some of my university friends were conscripted and set off the fight in a war that we should never have been in.

An Insider To Policy and Decision Making [00:03:04]

John Rodsted: [00:03:04] As a public service insider, you became privy to how policy and decisions were made. And this is, was not always a fair and honourable process. What kind of things and opinions did they drive Australia towards?

Paul Barratt: [00:03:17] Well, if we stick to the defence domain, quite often the real consultative process wasn't around whether or not we should get involved in a war, but how we would get involved. So the prime minister would make a decision that we should go off to fight alongside our American ally. And then first thing that would come to cabinet would be, what form will this assistance take? There's too much power and too few hands at the beginning.

John Rodsted: [00:03:44] So It would sort of come down to the US would effectively insist that we entered a war, supporting them. And as long as the prime minister agreed to that, then, we were committed.

Paul Barratt: [00:03:56] Actually, it's worse than that, John. More often we would insist on participating in a war to which the US hadn't invited us. And that was very much the case with Vietnam. We, our government persuaded them that they should have us along. The US military was not particularly enthusiastic because they find it easier to fight alone and feel that they've got the capability to do so. That turned out to be wrongly in most cases, but they feel they can do it. But the American political system likes to have some extra flags on the poles show that they're involved in a major coalition. The same thing happened with Iraq and Afghanistan. John Howard volunteered us into those wars. The Americans didn't ask us.

What Role Does Opposition In Parliament Play? [00:04:36]

John Rodsted: [00:04:36] With any dissent that may be either within government or within parliament, how are those voices then heard?

Paul Barratt: [00:04:43] Well, with great difficulty. There's unlikely to be dissent within government when the threshold decision's already been made, backbenchers will feel that if we're off the war, their job is to support the government and support the troops in the field. And when first, contingents went off to Iraq, Simon Crean, the then opposition leader leading a party that was opposed to the war, took very great care to distinguish between being opposed to the war. But on the other hand, wishing the troops all the best. We support our troops in harm's way, but we don't think we ought to be there. But that's a pretty difficult thing to navigate.

And as for parliament that depends on whether the government permits the matter to be debated at all. We committed ourselves to Afghanistan in 2001, and the very first parliamentary debate on Afghanistan was in Julia Gillard's time.

The Australians For War Powers Reform [00:05:38]

John Rodsted: [00:05:38] You're a strong advocate for changes on how we go to war. You helped form and chair the Australians for War Powers Reform. What's the organization? And what do you want to see change?

Paul Barratt: [00:05:49] The organization had its origins in something that, in 2012, we call the campaigns for Iraq war inquiry. Our first objective was to get something like the Chilcot inquiry that was going on in the UK to find out how the decisions were made and what could be learned from that process. But the real aim was to use this as a case study in why, the power to deploy the ADF into international armed conflict or to be relocated in the parliament. We expected and we knew a lot about how the decisions had been made or able to infer a lot by research. And putting various bits and pieces together, but we wanted an open public inquiry, which would demonstrate that our decision-making processes were flawed. And that it was too dangerous to leave it in the hands of a very small number of people.

And so now what we want is to relocate the power to send the defense force in any kind of armed conflict, to be relocated to the parliament. A decision only taken when the parliament, and in our view, both houses have accented to that.

Is The Response Time An Issue? [00:06:56]

John Rodsted: [00:06:56] If you take the decision away from the prime minister, removed the so-called captain's call, wouldn't it take too long to respond to any threats in a real timeframe?

Paul Barratt: [00:07:06] No, that's a great, great misapprehension. Most of the Australian defense force quite rightly is held in a pretty low state of readiness. So it's training and doing practice manoeuvres and what have you, but to get your equipment into a fighting state, it requires a lot of preparation. For example, when we went to Timor, Admiral Barry and I advised the national security of committee of cabinet in February 1999, that we ought to get ready have the option to deploy, to Timor as that plebiscite was looming, because we could see that there might be a, breakdown of the situation there. They were finally ready to deploy in September. So it took us seven months and the expenditure of almost $300 million to get everything really up to scratch and to get commanders at various levels used to commanding operations in the field at that kind of level. So we have a ready reaction force in Townsville, which is basically a battalion, and associated elements. And I would be quite happy to say, to have a framework in which anything that the ready reaction force could handle could be done on the decision of the government, because that would be an emergency type situation. But anything that required a larger deployment, ought to be debated and authorized in parliament.

Will Politics Get In The Way? [00:08:25]

John Rodsted: [00:08:25] if the decision had to go through parliament, couldn't it get held up by minor parties or in the Senate or whatever, just people being divisive because they can playing politics with the decision?

Paul Barratt: [00:08:36] That's an argument we often hear. If there was any genuine threat that any major political party would be opposed to the deployment and of course, any situation in which the ALP agreed with the government or the coalition, agreed with the ALP, depending on who's in government. If the major opposition party agrees with government, the minor parties have no role at all. So that concern sounds to me like a concern that it might be difficult for the government to engage in wars of choice. And of course, that's the whole point.

John Rodsted: [00:09:11] And I suppose that separates it perfectly between threat and adventure. One, you're actually going to respond for a real threat, that's threatening Australia and Australia's interests. And the other is getting involved in an adventure that's got nothing to do with us, and that would be the separation

Paul Barratt: [00:09:27] To put it brutally, I would say to government of either's side, if you can't persuade the opposition, that our national security, isn't just a really engaged here. We ought not to go.

How to Keep Intelligence Secret? [00:09:39]

John Rodsted: [00:09:39] So if the party that was in power that had government, at the time had access to secret intelligence that they can't talk about, how would they with this?

Paul Barratt: [00:09:46] There is a couple of ways you could deal with that. That's an argument we often hear and it's sometimes it's a bit hard to keep a straight face. When people talk about that when we reflect back to the WMD in Iraq that turned out not to exist and everybody knew they didn't exist. Hans Blix United nations weapons inspector certainly knew they didn't exist.

But let's take your question at face value. There's a couple of things you could do. What we do right now is, in any national security situation, the government iwill brief the leader of the opposition, in private and in secret. That happened in relation to operations in Syria. you could have a proper national security intelligence kind of committee in the parliament, in which those members of the committee were security cleared to receive all the information that's available so that you would have all parties involved in looking at the available evidence. And they could go into the parliament and say, well, we've seen the intelligence and we are convinced.

It's rare that secret intelligence is the only thing you've got. Very often there is information in the public domain as well. In fact, I think most intelligence agencies should devote more effort to analysis of what's in the public domain because you can learn a lot from that. An option would always be available to government would be to say; Here what you're seeing in the public domain, and a simply without elaboration, say our secret intelligence bears out what we've concluded from the open-source material. So if there's a will to do it this way, you can certainly find a way to navigate your way through that real difficulty of, how you handle secret intelligence.

John Rodsted: [00:11:31] The secret intelligence effectively just becomes a confirmation of what is a greater information stream.

Who Supports War Powers Reforms? [00:11:37]

Yeah. What kind of support have you had for your organizations aims and ideals and where should it go from here?

Paul Barratt: [00:11:44] We've had support from various members of various parties and a lot of public support and I'll come back to the public support. The most tangible support we've had from a political party is a resolution that was passed on the floor of the ALPs national Congress in 2018 in Adelaide when there was a vote on the floor that an incoming Labor government would establish an open public parliamentary inquiry into how we go to war. And I think that was a very, positive step. I think that's a very good way for a political party to get into it because in not pre-committing themselves to change the way we go to war, but they're committing themselves to establish the facts. It would give those who are seeking a change, the opportunity to put their case. And it would put the people who dismiss it in through the various arguments that we've just discussed. They would have to defend it in that kind of forum. So we would end up with a more honest debate.

Another element that these people will tend to use to argue against us is that it's really wouldn't make any difference because everyone had just vote on party lines. What I would say, in such a parliamentary inquiry, I think you would find that being asked to take responsibility for, something that would involve death and destruction on both sides; I'm putting the young men and women of the ADF in harm's way and do inevitably involving civilian casualties. You would end up with something that looks very much like a conscience vote. I don't think you can assume that everybody would vote on party lines. If we have a parliamentary inquiry, we can flush all these arguments out.

I'd like to see that commitment find its way into the ALP platform, but I very much hoped that, an incoming Labor government, such time as that happens, would proceed along that those lines. Our movement would like to persuade all major political parties that, this is a desirable change. That once one's on board, I think it will be easier to get the others on board.

You've had some pretty good support from some fairly major players within the Australian government and former Australian defense. Can you talk a little about the opinions of some of the others who are involved in your organization and why they think it's a good idea to change the threshold for going to war and the captain's call?

Well, I think were unanimous in feeling that, the responsibility for this order rest with the federal parliament and it ought to be debated and, fully thought through. One of the things that don't happen when it's just decided by cabinet or by the prime minister is a proper analysis of the legality of going to war. And what we would all like to see, is before parliament takes a decision that the Attorney General or Solicitor General tables, a formal written opinion about the legality of this war. Because the best legal opinion about the Iraq war is that was illegal. And no one takes very seriously the reliance that we had, on very old UN security council resolutions that were passed for another purpose.

So, apart from in our movement, we've had people like former Chief of Army, saying that this move ought to take place.

An Artificial Intelligence Arms Race? [00:14:56]

John Rodsted: [00:14:56] Can we shift the discussion a little towards the current arms race that's starting to get going, which is the development of killer robots? Just the talk of killer robots sounds like a bad dream, but they're real and governments worldwide are developing and investing in them. What do you understand these to be and how would they be deployed in the battlefield, for that matter into urban environments?

Paul Barratt: [00:15:19] I think the word robots conjures up in the public mind, things that might move along the ground and have maybe have arms and legs. But what we're really talking about is any kind of lethal autonomous weapon. And that very often would be a more advanced form of armed drone that would have its own decision making capability. And, that would take human agency out of the decision to launch a lethal strike.

Now it gets to a little bit fuzzy because I was reading this morning, someone from the US army talking about the progress they're making with them. And they're saying that they'll never take human agency out of making the decision, but they're saying the way these drones, the way these weapons work, you have a collection of sensors that will bring a lot of data together and then make a recommendation. And that recommendation would include which weapon located where would be the best to use for this purpose. Now this US army spokesman was talking about reducing the decision making time from the censors to someone pressing some button from 20 minutes to 20 seconds. 20 seconds, doesn't sound to me like a lot of time for someone to make a considered decision to launch a lethal attack on someone. So the word...

A Commander’s View on Lethal Autonomous Weapons

John Rodsted — Thu, 10 Sep 2020 09:00:00 +1000

Stay in Command

In the ‘Stay in Command’ episode today, Major General Mike Smith (Ret.) and John Rodsted from SafeGround explore the issues surrounding the development of Lethal Autonomous Weapons with Artificial Intelligence. The mechanics, ethics and application of this new technology paints a disturbing picture of a world where machines decide who will live and who will die. Today we will talk about leadership both civil and military and the complexities of command responsibility in regards to Lethal Autonomous Weapons.

Major General Michael Smith (ret) has spent his life leading others. He graduated the Royal Military College Duntroon as Dux of his year in 1971 and since commanded everything from a Platoon to a Brigade. His 34 years in the Australian Army had him in some complicated situations.

Content in this episode:

"The Buck Stops Here" [00:02:33]
Legal Framework for Commanding in Conflict [00:05:10]
Introducing Lethal Autonomous Weapons to the battlefields [00:08:54]
The Nature of Wars [00:16:12]
A Possible Arms Race? [00:21:51]
Technology Development [00:25:05]
The Fog of War Continues [00:32:43]
Making The Decision To Go To War [00:38:27]
Banning these Lethal Autonomous Weapons? [00:41:44]

If you have questions or concerns please contact us via info@safeground.org.au

If you want to know more look for us on Facebook, Twitter and Instagram Australia Campaign to Stop Killer Robots or use the hashtag #AusBanKillerRobots.

Become part of the movement so we Stay in Command!

For access to this and other episodes along with the full transcription and relevant links and information head to safeground.org.au/podcasts.

Transcript

John Rodsted with Mike Smith

A Commander’s View on Lethal Autonomous Weapons - Interview with Major General Mike Smith (Ret.) [00:00:00]

John Rodsted: [00:00:25] And today we're speaking with Mike Smith as part of the 'Stay in Command' series. ' Stay in Command', explores the issues surrounding the development of lethal autonomous weapons and artificial intelligence. The mechanics, ethics, and application of this new technology paints, a disturbing picture of a world where machines decide, who will live and who will die.

Mike spent 34 years in the Australian Army and retired a Major General. He graduated from the Royal Military College Duntroon in 1971 as Dux of his year and has had a distinguished military career as an infantry officer commanding all levels from Platoon to Brigade Commander.

He served as Australia's Defence Advisor in Cambodia in 1994. And throughout 1999 was Director General for East Timor. He was then appointed as the First Deputy Force Commander of the United Nations' Transitional Administration in East Timor (UNTAET) in 2000 and 2001. In recognition of this, he was promoted to an Officer in the Order of Australia.

After the army Mike became the CEO of the Australian refugee agency Austcare from 2002 until 2008. He then became the founding Executive Director of the Australian Government's Civil-Military Center from 2008 until late 2011. He then served with the United Nations Support Mission in Libya for 12 months as the Director of Security Sector Reform. He's the immediate Past President of the United Nations Association of Australia and is the current Chair of the Gallipoli Scholarship Fund and a Non-Executive Director of the Institute for Economics and Peace.

Mike holds a master's degree in International Relations from the Australian National University, a Bachelor of Arts in History from the University of New South Wales, and is a Fellow of the Australian College of Defence and Strategic Studies. He's also a graduate of the Cranlana Leadership Program and the Company Director's Course of the University of New England.

Today we'll talk about leadership, both civilian and military, and the complexities of command responsibility in regards to lethal autonomous weapons. Welcome to SafeGround Mike Smith.

Mike Smith: [00:02:28] John Rodsted! Lovely to be here with you.

"The Buck Stops Here" [00:02:31]

John Rodsted: [00:02:31] Mike, 'The buck stops here'. This was a sign that sat on president Harry S. Truman's desk. Someone, at a level, a high level is ultimately responsible and here he is in front of you. What do you see the role and responsibility of a commander is?

Mike Smith: [00:02:46] Well, books have been written about this, John, and, let me try and be as succinct as I can. Basically a good commander needs to demonstrate leadership. And in doing that, they need to make sure that what they do is always legal. That's always a good start because if a commander doesn't abide by the laws and in particular, in conflict, the laws of armed conflict, then they are perpetrating crimes or potentially perpetrating crimes against humanity.

So a good leader needs to provide fearless advice to his or her superiors. And at the same time, a good leader needs to set the example and to motivate their subordinates - both by his or her actions and by doing the right thing. But a commander needs to do a few other things than having those personal traits that we all know about.

A good commander must provide the proper training. And acquire the resources necessary for their men and women to do the job that they are set to do. And a good commander must always know the capabilities of those under his or her command. I found, personally, that one of the best traits of a good commander is the ability to be a good listener and to always encourage subordinates to honestly tell you what they think . A poor commander only ever wants 'yes-men and women'. A good commander wants to hear different points of view.

John Rodsted: [00:04:35] So a really key point to that is that you've got empathy. You've got empathy with the people that are within your command. You can see it from their perspective.

Mike Smith: [00:04:45] I think empathy and respect are key to being a good commander. And of course not, everybody will always agree with a commander's decision, but if everyone respects the commander, they say, well, I didn't agree with it, but I respect it. And I trust the commander that what that commander is telling us to do is the right thing to do.

Legal Framework for Commanding in Conflict [00:05:10]

John Rodsted: [00:05:10] I guess that brings you when you're a military commander it's a complicated environment. That you're, you know, you're in an operational role. You're in a dangerous environment, and as you said, you've got to operate legally. You need to have the respect of the people under you. If you're operating, say in a combative environment, you're making decisions that can be life and death for your own troops, but also for civilians, prisoners, refugees, opposition, combatants, and all of them within the legal framework.

So, can you guide me through a little bit more about how the decision making would take shape under these conditions and how you'd have to adapt?

Mike Smith: [00:05:48] Well, I think the most important thing is that if you have good doctrine, then everybody understands what the right and the wrong is and how to do things. And a good commander, always ensures that people understand what the doctrine is and that they abide by it. And good commanders are always inventive, and they use their initiative and they encourage their subordinates to use their initiative. In fact, they expect them to. But to do so lawfully all the time, within the rules and regulations, not to go outside of them.

John Rodsted: [00:06:28] But I suppose then if you get into, a life and death situation, as in combat, is it almost an oxymoron to think that wars have limits because the business of fighting a war is achieving your objectives and people are going to get killed as part of that and staying within a legal framework, does that not get stretched or, how do you see that?

Mike Smith: [00:06:49] Well, of course, it gets stretched. It can get stretched, but a lot of work has gone into the laws of armed conflict, into international humanitarian law. So there are boundaries. Now there will always be grey areas. There's no question about that because, in the heat of battle, instantaneous decisions have to be made. But, generally speaking, I think that it has to stay within those limits. And there might be some mistakes made, but if those mistakes are war crimes, if they are targeting innocent civilians, those sorts of things, then a commander must be held accountable and responsible for breaking those laws of armed conflict.

John Rodsted: [00:07:35] So staying within what is a legal framework is an essential part of being a military commander, achieving your goals, but staying within the legal framework, that is your umbrella?

Mike Smith: [00:07:45] Absolutely. And to go outside that means that you're just really acting like a terrorist, aren't you? You don't abide by the laws of armed conflict. So, some people sometimes say 'that's like fighting with one hand tied behind your back'. But I've never subscribed to that view because, if you - a soldier, a sailor, or an airman - and you are representing your state, you abide by the rules of your nation- state. And in Australia's case, we abide by the laws of armed conflict and they are irrefutable.

John Rodsted: [00:08:23] That brings us to the point that what you're provisioned with to achieve your goals, what is in your arsenal, what is available to an Air Force, a Navy, an Army, et cetera, become tools that are legally acceptable to that nation for their commanders to use in the field. Would that be sort of correct?

Mike Smith: [00:08:41] Yeah, absolutely. Absolutely. And the whole nature of warfare is that it's a constantly changing way that combat occurs, largely because of technology.

John Rodsted: [00:08:54] Hmm.

Introducing Lethal Autonomous Weapons to the battlefields [00:08:54]

John Rodsted: Historically there's been times when weapons systems have been acceptable within a military framework and have got somewhat out of control. And I imagine a couple of the good examples would be the use of anti-personnel landmines, cluster munitions, and the elephant always in the room would be nuclear.

And they've all been addressed with international treaties that have brought about their removal and restriction. I imagine at the time when they were employed they were all legal, but then the flavour of, the national humanitarian law and international treaties turned against those.

Then things become a suppose, a little bit more complicated when you have to look in hindsight at a weapon system that's been removed, but it doesn't change things in the field at the time. So there are weapons that have been used, and then have become unacceptable internationally and treaties have been formed to deal with those. Land mines, cluster bombs, nuclear weapons would be some of those. I guess there's another series of weapons that have also been dealt with by treaties. One would be poisoned gas after world war one. The other weapons system that was beaten before it was used in combat was blinding laser weapons, and the protocol was created in 1995 in the CCW . So that was a good example of beating a weapons system before it was deployed. It sort of brings us to the thorny issue that's on the table at the moment, which is about lethal autonomous weapons or 'killer robots'.

There's quite a bit of international research and development in the various forms of these systems. Here I need to draw the important division between killer robots and drones, as systems are now, drones have an operator who makes the final decision to strike or not to strike . With killer robots the machine makes the final decision and the choice to kill. The machine is in command with no human in that loop.

Mike, from your command perspective, how would you feel about handing over the role of decision-maker to kill or not to kill the one machine?

Mike Smith: [00:10:53] Well, I feel very uncomfortable about it. And of course, the distinction you make between lethal autonomous weapons and drones, and not only drones but a whole range of weapons systems that use artificial intelligence. You're quite right in saying the difference is that the decision is made by a robot - by an algorithm - and the other is made by a human. And the difficulty is that's happening with lethal autonomous weapons, as I see it, is that this distinction is becoming increasingly blurred. It's becoming a really grey area. So that for example, there are autonomous weapons systems that are lethal, which even Australia has. And I'm thinking here about, anti-missile defence systems onboard ships, and that sort of thing, that just come into play automatically if the ship, or if an area, is threatened. These, I think can be justified in the sense that they are not targeting humans. They are really defending against an incoming missile or an incoming threat which is itself not human.

But then we get to the situation that we say, well, if that can happen in that situation, why don't we program these weapons so that we don't have to be there at all? And they become offensive. And that they attack humans. And that's where I think the line has to be drawn. So I guess in terms of lethal autonomous weapons, I see that a human being must be responsible for targeting and must be held accountable should things go wrong, and humans be killed, as a consequence of their use. When I say, humans I'm talking about non-combatants.

John Rodsted: [00:12:49] So trying to limit the destruction to the combatants on a battlefield and keeping the civilians out of that equation, if at all possible?

Mike Smith: [00:12:56] Yeah, absolutely. And saying that there are limits to the extent to which we will allow machines to make the decision to make a strike.

John Rodsted: [00:13:09] If there was a movement towards a deployment use of lethal autonomous weapons within militaries of the world, do you think that could become a bit of a slippery slope, which would reduce the threshold to go to war, which would make it easier for governments or militaries to choose to go for a conflict, as opposed to trying to preserve life on their own side? Do you think the presence of autonomous weapons would do that?

Mike Smith: [00:13:34] they're could. I think that we're entering uncharted waters here. It's a little bit like when poison gas was used on the battlefield because it existed. It was only when people saw the consequences of it that they said, 'Hey, this is just too much. We've got to ban it.' And they did successfully. When I look at things like that, I have great hope; the same as you know, after all of those landmines were used and they caused havoc they were then banned. Cluster munitions is another one where I think that some progress has been made, but not as much as I would like to see. So lethal autonomous weapons are very much in that category, where there needs to be limits on how they can be used. And this is why I really hope that Australia plays a big role in the United Nations, in the CCW Convention, in trying to define those roles. One thing is clear, John, and that is that technology is not going to stop. These things are going to keep being invented.

Algorithms are going to be done. And, I just read the other day that, a robotic F 16, defeated a human- flown F 16 aircraft five times in a row. So, machines can definitely do this stuff. There's no question about it, but it's what is the purpose of those machines?

Now, does that make it a slippery slope to go into conflict? Because you've got these? I would like to think that it would be more about, well, how this enables us to defend ourselves better. This enables us to deter conflict better, to prevent atrocities occurring because it can be done accountably. But it comes down to what control we will keep over the use of these autonomous weapons systems.

John Rodsted: [00:15:37] And what you're really saying is at some point there needs to be a human in the loop that can override what the machine is doing so it still has some form of meaningful human control?

Mike Smith: [00:15:47] Yeah. You can't take a robot to the International Criminal Court can you? So a human being has to be responsible at all times. That's what makes the human race what we are. We have to be accountable for our actions, and by just creating machines to go and do this sort of thing for us is hardly an excuse for atrocities even when they occur.

The Nature of Wars [00:16:12]

John Rodsted: [00:16:12] I read that same report about the F16 simulator in dogfights with a manned aircraft. And one of the things that struck me was the F16 robotic would go on a head-on attack to the other aircraft and close within 100 meters, which is effectively suicidal. And from the top gun school were saying you would never close in a head-on attack like that because the chances of surviving are fairly slim.

It brings into the issue of machines are prepared to be suicidal because they just a machine, where humans still wish to preserve their own life or generally do. So that certainly puts an advantage towards the machine. Doesn't it? If it's prepared to be destroyed in the execution of its role?

Mike Smith: [00:16:55] Oh, totally. And of course, it's a lot cheaper. Now, of course , there have been many precedents where humans have been prepared to go into suicide type missions and not cared about their own safety. But if armies, navies and air forces were encouraging their humans to do that, then those armies, navies and air forces wouldn't last very long would they? So, if you can send machines in to do it and they're cheap, you can say, 'well, that's all right, we'll just make more machines.' And this is when I think it becomes extremely dangerous. Particularly if those machines are going in to kill human beings, not other machines.

John Rodsted: [00:17:38] And it takes us into that world of sort of asymmetric warfare, where you let's take the scenario of a large powerful, industrial nation has got the ability to build lots of these weapons and stockpile through the years of peace. And simply through the might of money, be able to swarm and overpower their opposition.

Then it becomes the right and wrong rests in the hands of wealth, as opposed to in any ideological issue. So that would just turn the situation into I suppose capitalism wins?

Mike Smith: [00:18:12] I don't quite see it that way, because technology is transforming at such a rapid rate, that there's no point in stockpiling...

The Tech Perspective with Lizzie Silver

John Rodsted — Tue, 08 Sep 2020 15:45:00 +1000

The Tech Perspective with Lizzie Silver:

Technological Aspects and Tech Industry

This episode of Stay in Command emphasised the technological dimensions and concerns as well as the implications on lethal autonomous weapons on the tech industry. Our guest Dr Lizzie Silver is a Senior Data Scientist at Melbourne-based AI company Silverpond.

Content in this episode:

Troubling reality of these weapons [1:49]

Problems with fully autonomous weapons - explainability[3:49]

Facial recognition and bias[7:11]

Military benefits from technical point of view [11:36]

Machines and the kill decision [15:01]

Hacking [16:30]

Positive uses of AI and funding battle [17:10]

Challenge of Dual Use [20:45]

Regulation: Treaty, Company Policy, Individual Actions [22:16]

If you have questions or concerns please contact us via info@safeground.org.au

If you want to know more look for us on Facebook, Twitter and Instagram Australia Campaign to Stop Killer Robots or use the hashtag #AusBanKillerRobots.

Become part of the movement so we Stay in Command!

For access to this and other episodes along with the full transcription and relevant links and information head to safeground.org.au/podcasts.

Transcript:

Welcome to SafeGround, the small organisation with big ideas working in disarmament, human security, climate change and refugees. I’m Matilda Byrne.

Thank you for tuning in to our series Stay in Command where we talk about lethal autonomous weapons, the Australian context and why we mustn’t delegate decision making from humans to machines.

This episode we’re looking at the “Tech Perspective”. We are going to discuss the technological concerns of lethal autonomous weapons and their implications on the tech industry.

And so with me today I have a great guest with me today in Dr Lizzie Silver. Lizzie is a Senior Data Scientist at Silverpond which is an AI company based in Melbourne, which is also where I am coming to you from - so welcome Lizzie, thanks so much for joining us today

Lizzie Silver[00:00:52] Thanks for having me

Matilda Byrne: Before we jump in, I’m just going to talk a bit about the definition of killer robots in case any of our listers are unfamiliar with exactly what it is we’re talking about.

So killer robots or fully autonomous weapons are weapons that have no human control over the decision making. So when they select a target and engage the target so decide to deploy lethal force on that target, there is not a human involved in that process and it is just based on AI and algorithms. So with these fully autonomous weapons there are lots of concerns that span a whole of areas that span a number of different areas - today we are going to go into technological concerns in particular because we have Lizzie and her expertise, but there's also things like moral, ethical, legal global security - a whole host of concerns really.

What is the most concerning thing about killer robots?[00:01:49]

Matilda Byrne: And what I’m interested in Lizzie, is, just to start off with if you could tell us what is it about fully autonomous weapons that you find the most worrying, so what about them makes you driven to oppose their development.

Lizzie Silver: It’s really a fundamental issue with these issues is you can't give a guarantee on how they’re going to behave. WIth humans we can’t give a guarantee on how they're going to behave but that’s why we have all these mechanisms for holding a human accountable. Now you can’t hold an algorithm accountable in any meaningful way. So what you would like to do is find a way to characterise how it’s going to behave in every situation, but the thing is a conflict situation is just too complex. There are too many potential inputs and outputs, different scenarios that could confront the AI. You’re never going to get through all of them. You’re never going to be able to fully characterise the space. So what you’d like to say is say ‘Ok, on this sample of situations we’ve seen this kind of behaviour, so we expect that in the future, in similar situations we expect the system is going to behave similarly’. The trouble with a conflict situation is that things don't stay similar- it’s an adversarial context so we can’t give any guarantee that the system is going to perform the way we expect and want it to, and they’re changing by design, you know the enemy that you’re fighting is trying to change things in ways that will through you up. It’s just a fundamental characteristic of warfare.

[03:30] Matilda Byrne: For sure and I think you know, warfare is one of the most unpredictable and chaotic environments that we could possibly think of to put any kind of machine into which is really troubling.

Could you speak to ‘explainability’ as one of the characteristics of AI raised by the international community?[00:03:49]

Matilda Byrne: and I think you also hit on some of the characteristics of AI that have been flagged by the international community as concerns with these weapons, so things like unpredictability, reliability, obviously the huge concerns around accountability and another one that has come up is explainability - I wonder if you could speak to that at all and sort of the black box phenomena

Lizzie Silver: Yeah, I mean explainability is really what we are trying to use to get some sense of comfort with these algorithms. We want some way to characterise their behaviour. But there isn’t single kind of explainability, and something may be explainable and turn out to be unpredictable in ways we wouldn’t expect. We keep coming up with new kinds of explainability, new explanations for how these complex systems work. Every time, we think to test them out on new benchmarks - an example of that is a paper that came out fairly recently on texture versus shape, that fund that AI systems really pay attention to the texture of objects and not the shape of them because there's more information in texture and it’s easier for them to use and if they can solve the problem just using texture they will. And on major image benchmarks you can actually do really well just using texture, you're not forced to use shape information. And the way the research has demonstrated this is by creating a new dataset where the objects had conflicting shapes and textures. So you might have a picture of a cat and overlay it with elephant skin and a human looks at that and says it’s a cat, bt a human looks like that and says it’s an elephant - so it’s behaving in a way thats different to how humans behave. And it’s been trained to do this because paying attention to textures has been enough to recognise elephants in the past, it never needed to learn the shape of an elephant. But you think about how that behaviour is going to play out in a warfare situation where the enemy is inventing new kinds of camouflage and you're not necessarily going to be able to predict how the system is going to behave when the enemy changes its outfits and looks like civilians. And particularly when if you’re retraining the algorithm to improve its performance constantly, you could have a system that avoids civilians up to a point, and then learns to target them.

(06:30) Matilda Byrne: Yeah, for sure. I think it’s really interesting how really simple things can sort of, confuse the Ai. SO some of the proponents of fully autonomous weapons have stated that they will be able to make sure it can identify civilian objects, things like a red cross or red crescent symbol in a conflict zone so it wouldn’t attack those things. And then that for me was quite puzzling because it would be so easy for an enemy actor or non state actor to then put a red cross symbol on the side of their vehicle and then not be attacked, so I think these are really complex and nuanced things.

Facial recognition, bias and distinction[00:07:11]

Matilda Byrne: You were talking about texture and shape, as an example of something that AI is trained to recognise. WHat about facial recognition and where we are in terms of that because obviously in terms of targeting specific people in a conflict scenario that’s going to be a big part of sensory input, we can assume.

Lizzie Silver: Yeah, so it turns out that the AI reflects the input data so AIs that are developed in mostly white countries tend to be a lot better at recognising white faces, AIs that are developed in China are much better at recognising Asian faces. And they also reflect the biases of the people who develop them, it seems likely that if we develop these they're not going to be as good necessarily at recognising the people that we are in conflict with - they are likely to look different to us on average.

Matilda Byrne:: For sure, and I think that algorithmic bias is something that has been flagged as really problematic in terms of how it impacts certain races more than others and disproportionately impacting certain people from parts of the world which is also a great concern.

Lizzie Silver: Yeah, and the fact that it performed worse on women than on men, particularly black women more than black men indicates that there's a bias based on the date set potentially the creators haven’t focused so much on performance in that sub set and that you know, can’t be because of population difference - there's just as many women as men, slightly more actually.

Matilda Byrne: That’s really interesting, and also that it’s perhaps biases in the people that are training in the first place that are perhaps more inclined to then train it on more men, why is that?, and then those become embedded in what the software is able to do, and then I think when you extend that to a warfare situation and think of those biases that are being embedded by a state that is waging war, and is in conflict what it’s going to reflect is just going to exacerbate, potentially the harms that occur in conflict.

Lizzie Silver[9:31]: Right, None of these are intended issues with the AI. They're all just emergent properties based on how they’re trained and the data sets that they’re trained on, and the way people put them together without specifically setting out to mitigate these problems. Now there’s a whole lot of unintended situations in warfare, there's a lot of unknown unknowns and if we can’t even get right equal performance across races and genders in a situation where things are not changing very rapidly, it seems really overly optimistic to me to think that we can distinguish civilians and non-combatants from combattants. It seems incredibly optimistic to me to say that the AI is going to be able to make some kind of ethical judgement in the moment and say ‘this person was attacking v this person was surrendering’, those actions can look really similar’.

(10:37) Matilda Byrne: For sure, and I think there's a lot of importance of having that human evaluation and what youre talking about with this distinction between enemy combatants [with civilians] is getting into the territory of international humanitarian law. And I think there's a important part to that decision making that a human can read certain signals and cues and evaluate them when theres something thats not necessarily concrete, they can understand the wider context and use their human judgement really which is what is essential, on top of this other information to then make a decision and this is why when we talk about meaningful human control or human involvement it’s so important that we do have that added layer, I guess, and it’s ot purely the algorithms decision making based on what it can attempt to understand and compute.

What benefits, from a technical point of view, might these weapons have for militaries?[11:36]

Matilda Byrne: I think it's probably worth also acknowledging why these weapons are desirable for the military because they are under development in a handful of countries; so the US, Russia Israel, UK and also Australia [as well as China and South Korea] and so they wouldn't be trying to develop them if they didn’t think there were benefits. Some of them are obviously around I guess strategic things so not having to have as many boots on the ground is obviously desirable for militaries and there’s other ethical questions around whether that's good or not, but what about from a technical point of view, why do you see these weapons as potentially being appealing?

Lizzie Silver: So there are some situations where I can really understand the appeal. Firstly, machines can react a lot faster than humans so there are already automated systems like C-RAM and FALENX that are intended to react to incoming missiles that are coming in faster than a human could overt them. And those are supposed to defensive systems, but they do sometimes, screw up, you know there have been cases where they've shot down planes Now with the i you have a similar situation, you ma want the AI to take control of a fighter jet to perform evasive maneuvers faster than a human can but then when you are performing aggressive maneuvers faster than a human can then you have a problem right? Because the AI then needs to make all of these complex ethical judgements. The other situation in which you might want it, is if you have developed these systems where you have maintained human control, he AI is maybe helping to plot a drone but it is not deciding who to strike with the drone and then you go into an area where your communications are cut off with that drone, so the idea here is that you should be able to keep fighting in the same capacity even though communications are cut off. This I’m less sympathetic to because I don't see the need to continue to use lethal force if you can't control the weapon, right. You can always just have that drone turn into a reconnaissance drone while it is cut off from you. Those are some of the situations.

On lowering the threshold of war

And then there's an argument I’m really unsympathetic to that I've heard from some military people - that they would like to just not expose their people to the sole crushing situation where they have to kill another human being, and while i do understand that that does psychologically damage people, i think removing us from that damage just makes it so much easier to get into conflict situations, so much easier to create civilian casualties and i don't think we should be in a position to remove ourselves from the harm that we are doing.

Matilda Byrne:[14:40] For sure I think that idea of lowering the threshold of war because now our own people won't be exposed to this, doesnt mean theres not other people and civilians that are now being exposed, and we need that barrier to war in the fist place so that we don't escalate unnecessarily or are over zealous in starting wars. I think its really important.

On delegating the kill decision to a machine

[15:01]Matilda Byrne: And I think the other moral component when we’re talking about machines and this decision making, so this idea that you mentioned that if they lose communications should they go on and just continue. And especially around deploying lethal force, this idea of handing over a kill decision to a machine and so are we ok as society, as humanity, to allow machines, these algorithms to make decisions over life and death, I think alone is very problematic and a precedent that is troubling to set.

[15:32] Lizzie Silver: Yeah, I would not be comfortable trusting my life to a computer vision system that was supposed to recognise me as a non-combatant. I mean I work with these systems everyday and they fail in the most surprising ways. And it is just too complex and difficult a situation: warfare. Adversarial situations are ones where you can't really give guarantees on performance, every guarantee that we have in statistics is based on some assumption that the data stays similar to a certain degree. Even transfer learning guarantees where they talk about “we are transferring to a new situation, the data is fundamentally going to change” but t guarantee that the system is still going to work we have to assume that something is still going to stay the same, and I don't know what you can assume stays the same in war, except basic physics.

What about hacking?

Matilda Byrne:[16:31] Sure, and I think also in terms of guarantees in these systems is safeguarding them from hacking and that being a huge risk.

Lizzie Silver: [16:39] Yeah, absolutely. That is a risk in all military systems currently, but it gets a lot worse when you can take something that's already autonomous. I mean, it's a risk with remote piloted systems too. Right? Cause if you can hack into them, then you can create havoc, right? You don't need to be present to turn them on the side that created them, similarly with autonomous weapons.

What are some positive applications of AI?[00:17:10

Matilda Byrne: Mm, I think, we've talked a lot about, I guess, some of the limits of AI and how they're particularly problematic in the application of weaponizing AI, having them involved in fully autonomous weapons, but it's really important not to forget the positive applications of AI and even in certain contexts within defense, like you mentioned earlier, evasive maneuvers in pilots, there's some really great stuff happening in mine clearance with using robotics and AI. But even more broadly than that, I think in society in general, it's really important to step back and look at the good that it's bringing. So I wondered if you could speak to maybe some of the other domains where AI is being used in a really positive way for social good.

Lizzie Silver: [00:17:53] Yeah. There's a ton of stuff out there. I mean, one of the things that my company has developed is a decision support tool for clinicians, for ophthalmologists. It recognizes fluid in the retina, which is one of the things that you get with macular degeneration, which causes blindness. So, the AI picks up these pockets of fluid, some so small that they might be missed by a clinician and helps to quantify how bad that is. And that's another system that's under human control. Right? It's just making recommendations to the clinician who then makes the actual decisions about treatment. There's loads of uses in industry as well. We're working with power companies to [00:02:00] help take inventory of their assets across the power network, which can help with making sure everything's up to code for the bushfire risks.

Funding for AI in defence versus social-good applications[00:18:48]

Lizzie Silver:A lot can be done with these tools. And I'd like to see more investment in peaceful uses of AI. What was the investment that you said the Australian government, uh, put into. Developing an ethical military AI?

Matilda Byrne: [00:19:02] Yeah so just the one project that was about researching how to embed

Student views on Killer Robots

John Rodsted — Sun, 06 Sep 2020 09:00:00 +1000

Yennie Sayle is completing her studies of a Bachelor of International Studies at RMIT University and is the Youth Engagement intern with SafeGround for the Campaign to Stop Killer Robots Australia.

She sits down with three other students from different areas of studies and experiences to talk killer robots, the level of exposure in their degrees to the topic, their views, concerns, thoughts on university involvement in their development, raising awareness among students and more.

Matilda Byrne on Australia and Killer Robots

John Rodsted — Fri, 04 Sep 2020 22:15:00 +1000

Matilda Byrne on Australia's position on Killer Robots

John Rodsted:

Welcome to SafeGround, the small organisation with big ideas working in disarmament, human security, climate change and refugees. I’m John Rodsted

Thank you for tuning in to our series Stay in Command where we talk about lethal autonomous weapons, the

Australian context and why we must not delegate decision making from humans to machines.

Matilda Byrne is the national coordinator of the Australian Campaign to Stop Killer Robots. The Campaign to Stop Killer Robots is an international effort to preemptively create a binding treaty that will bring restrictions and the ban to a concept of weapons system that would have no meaningful human control - lethal autonomous weapons or killer robots. She holds a master’s degree in international relations and is presently working on a PhD, on international security and global governance. Welcome Tilly.

Matilda Byrne: Thank you for having me!

John Rodsted: Killer robots! Can you tell me what they are and why do you want them banned?

Matilda Byrne: Killer robots or lethal autonomous weapon systems are essentially weapons that are using artificial intelligence. And so for their selecting of targets and the decision to deploy lethal force, this is all done by the AI algorithms. So there's no human that oversees or intervenes or controls the targeting of people and then deciding to kill those people as targets. And so as for why we would like to ban these weapons, there's a whole host of different concerns across moral, ethical, legal, security concerns. For me, I think one of the most compelling things is this idea of delegating the decision making over life to a machine. And so seeing that as humanity, we are not prepared to have this decision done solely by an algorithm and that a human has to control this question of life and death of another human being.

Is Australia for or against killer robots? [00:03:20]

John Rodsted: So, where does Australia sit on this subject? Is Australia for killer robots or against them?

Australia, regrettably has this position where they say it's premature to support a ban. They've been saying this for years now. And essentially what this means is that Australia would like to have the option to potentially develop lethal autonomous weapons in the future. And so beyond that as well, they have suggested many times in public forums, so at the United Nations and in their own sort of reports and things that these weapons could potentially be also desirable. And so we need to research more. We want to look at developments in this direction and see how it could be really positive for our military. Obviously this is an incredibly disappointing position, especially because there's been no attempt by the Australian government or defence to engage with the idea of human control and actually to maintain human control in the decision making.

There are strong diplomatic efforts from civil society to get a ban on these weapons before they are developed and deployed, in short a treaty. Is this movement gaining any traction? And if so, with who?

Matilda Byrne: Yes, it definitely is. We've been seeing growing momentum towards these calls for a ban. And so first you have the different governments of the world. There is a grouping of 120 different countries called the non-aligned movement who have declared their support for a ban. In addition, there's also 30 different countries who have explicitly stated that they support a ban in the talks at the specific forum that deals with this issue of lethal autonomous weapons.

And as well as that, you've mentioned the civil society movement. So we have a lot of tech workers that are speaking up about having a ban and why that's really important for their work. So people in software, AI design, robotics, et cetera. There's also a lot of academics across different areas; so morality, ethics, philosophers, international security. They, I would say are the main sort of people, in addition to the kind of coordinated non-government organizations of the world that are working as part of the Campaign to Stop Killer Robots.

Is Australia creating killer robots? [00:05:36]

John Rodsted: Australia has a large research and development facilities in many universities and they do exceptional work in software and engineering along with medical advances. Are we working on creating killer robots or at least the software and the technology?

Matilda Byrne: The short answer is probably. So, what we know is that in a lot of our universities, there's a lot of research that's done in partnership with the department of defence and as well as defence industry. In a lot of those programs, there's a lot happening at the moment in autonomy; autonomous capabilities, autonomous systems, the kind of sensors that you would need for these weapons. Because we haven't explicitly statede at the department of defence that we are in fact, creating lethal autonomous weapons systems, it's impossible to know for sure the extent to which university research is being incorporated into such weapons. But what we do know is that the capabilities are there and that it would be very easy through these programs for those to be used for these weapons if that was the direction the Australian government decided to take.

How would kiler robots benefit Australian universities? [00:06:44]

John Rodsted: So if a university gets involved in research and development, how would it actually benefit the university?

Matilda Byrne: So I think one of the large incentives for universities to be involved with these programs is money. So they have received funding from the government, pretty simply. And I think a couple of the other things are more around reputation and marketing for the university. So they're involved in cutting edge and innovative programs, language like this, which is true. And, it's not an issue in and of itself for the university to do great groundbreaking research in AI and software and things like this. What's important is that they do have policies in place that say, as a university, we oppose lethal autonomous weapon systems and do not want our research being then contributing to the development of these weapons.

An ethics issue [00:07:35]

John Rodsted: So with the sort of technological advances, it really turns into an ethics issue, to draw the line between where a certain technology or algorithm can be used for, or effectively good or for weapon systems. So it does turn into ethics.

Matilda Byrne: Yeah, that's exactly right. If you put it really simply, just because something can be developed, it doesn't mean that it should. And I think you could retrospectively apply this to a lot of other weapons. So the creation of the atomic bomb or agent orange that we saw had devastating impacts. And having kind of learned from the past, we can then ask ourselves, well, what's the onus on us at present to prevent the development of something that would be abhorrent. And I think that there is an onus, and that it is really important to take into consideration these ethical dimensions.

John Rodsted: So what are the thoughts of some of the developers that their technology might be used to kill masses of people?

Matilda Byrne: So I suppose in terms of developers, you could put them in three categories. You have the people that are developing in these programs with defence and looking at lethal autonomous weapon systems. And I'm sure from their perspective, they're not thinking about how, what they're doing could cause mass civilian casualties. They're thinking about how they're contributing to the national security of Australia, things like this, but it is really problematic when there's then no controls or real consideration and reflection within those programs as to what it is that they are exactly doing and what the repercussions are.

Then you have developers in the sector that are just unaware that this is something that's taking place. They're a really important group that they sort of go about developing whatever is they're doing, sensors, algorithms, unaware that in the future, perhaps, this work that they're doing could be used for a lethal autonomous weapons system.

And then of course you have the people that are aware that this is a real concern and that are really troubled by this prospect. And they sort of face really tough decisions. The things like having to turn down a project that could be really positive for say, Agriculture, because it looks at targeting pests and eliminating pests in the native Australian environment, which they feel uncomfortable to do because they know that that system could be repurposed and turned into a lethal autonomous weapon in the absence of any real regulation.

John Rodsted: So regulation really is such a key factor to controlling and keeping a cap on these technologies?

Matilda Byrne: Yes, that's right. It's a key point in terms of delineating what is acceptable and what's not.

How much money? [00:10:13]

John Rodsted: Have you got any idea what kind of money is floating about within Australia at present developing various components or platforms for autonomous weapons?

Matilda Byrne: It's actually a very alarmingly high amount of money. The main area where we know that autonomous weapons or autonomous systems development is happening is 'trusted autonomous systems', which is quite an ironic name, also -'trusted systems'. This is a defence cooperative research centre. What that means is it's a partnership between the department of defence, research institutions like universities and also arms manufacturers or the defence industry. Trusted autonomous systems was the first research centre like this to be launched and it was awarded $50 million for its first seven years of operation. That's an area where we know a lot of the development is happening around autonomy for defence. But in addition, for example, just at the beginning of this year in January, the Royal Australian Air Force announced $40 million for a project with Boeing to make an autonomous combat aircraft. So that one project of these prototypes was 40 million, as I said.

We know since the release of the defence strategic update, that there's an $11 billion investment also in our land vehicles and autonomy specifically, to be made over the next 10 years. And as well as that, I think lastly, and sort of most problematic of all of these, it's less money, it's $9 million, but this is for a project that Australia says is to research how we embed ethics into killer robots. Which is a very bizarre and just problematic concept. The fact that this is something that Australia sees is good to do or important to do instead of just drawing a line and saying, we accept that fully autonomous weapons or lethal autonomous weapons will never be lawful, I think quite appalling.

Why do defence want them? [00:12:08]

John Rodsted: defenceWhy would the Australian defence force want these weapons systems?

Matilda Byrne: There's a few reasons why lethal autonomous weapons could be desirable. One of the main ones is in terms of response time. So this idea that there'll be much faster to make decisions. Some of the other things are around longevity. So if you have a person that's having to make decisions, fatigue and things, whereas these machines could just go and go and go.

And also, there's been arguments by the military, that they'll also be good for precision. Which I think as well as a bit of a flawed idea, when we think about how they do their targeting and we know that they will not be successful in targeting actual military targets correctly. And that there's this huge room for error where they could falsely or, or wrongly engage civilians instead.

But one of the huge ones, is that idea of response time in that it's beyond human endurance to do certain things. I think though on that point, what it really means is that we're prepared to then have all of these machines that then just escalate the pace of warfare. Because if we don't need a human to react, then machines can go much faster, which will ultimately cause more devastation and severe impacts.

Can they escalate conflicts? [00:13:27]

John Rodsted: One of the points you made there was about how it would escalate a conflict, because it would be response versus response and things would keep going faster and faster. And one of the roles of a commander is to take into account all sorts of things that are changing battlefield and try to de-escalate a conflict because that's part of a command responsibility.

And, and I think of an analogy to this would be the Russian Colonel back in the early eighties who held off doing a nuclear strike on America when their instrumentation to all intents and purposes showed that a full nuclear strike was heading to Russia. What's his name? Stanislav Petrov. He wouldn't launch the counter attack because he just believed something was wrong with the system. And he was proved to be right. And if it was left to a machine, it would have been a full nuclear response on America. And that would have been world war three. And it was one person in that loop who stopped the reaction. So, yes, the concept escalation or deescalation is a very important point to consider.

So could you paint me a picture of a battle using autonomous weapons? Now, what would they do instead of how would they do it?

Matilda Byrne: So I think the thing about fully autonomous weapons or having these killer robots in battle, it's a lot more insidious than what we might think about, which is, ultimately having these little robots, driving around an area at war and firing at each other. It's much closer to what we see at the present in sort of context of urban warfare, where you have drones circulating around. And then these are ones that are able to strike. You're able to have more of them go into areas. I think, initially, it's going to look not totally dissimilar to how warfare looks now. But just with a lot less accountability. And a lot less humans actually having to make these hard decisions and exercising and evaluating the current context and making sort of thoughtful decisions. Instead, it's going to be these robots flying around going, "Oh yeah! That fits my parameters. So I'm going to fire" without looking at things like; collateral damage. Is this really worth it for the strategic gains? All of these really essential evaluations that commanders do have currently, and that they have to take into account in order to maintain international humanitarian law.

Is there any human control? [00:15:47]

John Rodsted: So where's the point of human command and control in the targeting of autonomous weapons, or is that a set and forget, technology or is there a point that they can intervene to pull things off?

Matilda Byrne: What's incredibly concerning, in particular about the Australian position, is some of the remarks that they've made recently when pressed on this idea of human involvement. One of the things that the Chief of the Defence Force, General Campbell has stated is that there's never one answer for where a human would be involved. And we're one of the only countries that has stated something like this in the world, if the only. And I think what that means is we're trying to leave the door open and say, well, maybe it's at the very beginning when we choose who the target is, or maybe it's a little bit later. Or, you know, we just don't know, we're not committing to where the human's going to be involved or where if there will be any human control over targeting and selecting and choosing to deploy lethal force.

What can go wrong? [00:16:49]

John Rodsted: So what could go wrong with autonomous weapons?

Matilda Byrne: One is machine error, which I think you touched on, is definitely a huge concern. As well as that you have also a great risk of hacking, and the security of these systems which is very troubling. Because the more these machines are capable of, if they are hacked, the more negative the ramifications are. So there's other concerns also around if they could be used as a tool of oppression. So for committing genocide or other sort of atrocities and oppression. Because it isn't hard to set a certain set of parameters for the targets and all people in this one kilometer radius or whatever, into these systems and just send them off and go; 'okay - go'. And these robots don't have a conscience. So it's not like military personnel turning around and saying, no, we're actually not comfortable to fire on our a hundred thousand people that are gathered in this square protesting against the government. It's just this tool where it's free of any sort of human conscience or decision making. And so it's very, very problematic. And I guess that's not so much an instance of it going wrong, but about it being used for nefarious reasons that we hadn't necessarily thought about when we're thinking about just utilizing these systems in warfare.

Can killer robots be used for civil oppression? [00:18:07]

John Rodsted: I suppose it brings you to the point where how would the cross over into civil oppression be with autonomous weaponry? If you chose to use that to, for instance, the riots that are happening in various parts of the world at the moment, what would that look like? If people chose to use autonomous weapons against those civilians?

Matilda Byrne: Exactly. And I think though the risk that these systems could be used for domestic policing is really alarming. And the reality with these kinds of systems and the way the technology works is that if it is developed in one area, then it's easy to then change how it's used. But if it's never developed at all, because there is a ban in place, for instance, that it's much harder for people to conjure up these systems separately.

John Rodsted: So you take away the industrial manufacturing component, which can give you the ability to create masses of well-produced machinery. And it turns it into more of an ad-hoc method. So you won't get the saturation point.

Matilda Byrne: Right. Exactly.

Can killer robots follow international laws of war? [00:19:05]

John Rodsted: Battle fields are rapidly changing and confusing place. Hence the term, the fog of war. Much of how orders are given and followed depends on ethics, international humanitarian law, rules of war and engagement, Geneva conventions, et cetera. Could autonomous weapons be programmed to perfectly navigate such a space?

Matilda Byrne: The simple answer to that question is no. I want to break down one element of those parts of international law that you touched on, which is international humanitarian law. And even just two key elements of that, which is the principles of distinction and...

Killer Robots: Mary Wareham at Human Rights Watch

John Rodsted — Tue, 01 Sep 2020 15:15:00 +1000

SafeGroud Presents the series ‘Stay in Command’

: Mary Wareham on the Killer Robot Campaign

2020-Sep-1

Welcome to SafeGround, the small organisation with big ideas working in disarmament, human security, climate change and refugees. I’m John Rodsted.

Thank you for tuning in to our series Stay in Command where we talk about lethal autonomous weapons, the Australian context and why we mustn’t delegate decision making from man to machines.

Today we speak with Mary Wareham who is the advocacy director of the arms division at Human Rights Watch. Originally a native of Wellington in New Zealand she has been working in the disarmament sector for many years and is based in Washington DC. She is also the International Coordinator of the Campaign to Stop Killer Robots and joins us from Washington now. Welcome Mary.

You’ve had an extraordinary career working on the most important treaties since the 1990’s. The list of work is the success story of recent disarmament driven by civil society. The big two would have to be the treaty banning Anti-Personnel Landmines in 1997, the treaty banning Cluster Munitions in 2008.

Of these treaties, the work of civil society drove those processes and forced governments to account and ultimately change. The Landmines Treaty was awarded the highest international accolade with the Nobel Peace Prizes from 1997.

Today we don’t look back to celebrate the past but to the future in her work to ban, Killer Robots.

Killer Robots - sounds like a cheap Sci fi movie[00:02:52]

John Rodsted: Killer robots. Sounds like a cheap sci fi movie or a scene from the Terminator. What in fact are they?

Mary Wareham: Well, the Campaign to Stop Killer Robots is not so concerned about the Sentient walking, talking you know Terminator, like a killer robot. We're more grounded in reality. And what we've seen is the small number of military powers, most notably China, Israel, South Korea, Russia, and the United States are investing very heavily, now in military applications of artificial intelligence and the developing air land and sea based autonomous weapon systems.

We've been quite careful to call for a preemptive ban on fully autonomous weapons, which means that focuses on future weapons systems, not these existing ones that are out there today. But it helps to look at them, especially the extent of human control over the critical functions of selecting your target and then firing on it more and more.

We see senses being used to detect targets. And increasingly they're not controlled by humans. We have facial recognition technology cameras that are now employing that, there's heat senses, to detect body heat, motion senses, which can detect how you walk, your gate and of course, since it's for radars and we're all carrying around a great you know tracking device in our pockets, which has called a mobile phone using GPS technology. So it's a combination of different technologies, but, I think it's a bigger reflection of how our own lives are becoming much more subject to computer processing. And there are big technological developments that raise fundamental questions for humanity. When you try and incorporate artificial intelligence into a weapon system, to the point that you no longer have that meaningful human control.

Meaningful Human Control? [00:04:43]

John Rodsted: Can you explain a bit about meaningful human control for us and what's the difference between an autonomous weapon, which is using artificial intelligence. Can you flesh that out a bit more for us please?

Mary Wareham: wow. I mean, what is that artificial intelligence? There's still not any agreed on definition. So what we tend to talk about more, in this campaign is about autonomy, how autonomy is incorporated into weapons systems. And when we talk about human control over the use of force, we prefer to use the term control as opposed to judgements or intervention that implies a weaker role for the human.

We also like this word or modify meaningful because that ensures that the control is substantive. But of course there are other descriptions for that. We put out a paper a few months ago, detailing how we believe the concept of meaningful human control can be distilled down in an international treaty.

And it can be done in several different ways because it can apply to the decision making, the technological and the operational components, the decision making components of meaningful human controller about ensuring the human operator has got the information and the ability to make decisions about the use of force and ensure that they had to being done in compliance if legal rules and ethical principles. The human operator of this weapons system has to understand the operational environment, how the autonomous system functions, what it might identify as a target. And there needs to be sufficient time for deliberation.

Technological components are the embedded features of a weapon system that would help to enhance meaningful human control. This is about predictability and reliability. It's about the ability of the system to transfer or relay relevant information to the human operator. It's also about the ability of the operator or the human to intervene after the system has been activated.

This is what we would call a human on the loop, as opposed to a human out of the loop. And then finally the operational components that can make human control more meaningful. And this is about limiting when and where a weapon system can operate and what it can target. There's a whole bunch of factors that need to be considered.

And this, not least , how, how the force is applied. The duration of the systems operating period, the nature and the size of the area in which it's operating, the types of targets that it may be attacking people anti-personnel once or anti material. And I think it's also interesting to look at the mobility or stationary nature of an autonomous weapon system. And if there's anything particularly problematic in that. So what we're trying to do is determine the acceptable level of meaningful human control over the use of force. It's not a short answer because it's one that requires a negotiation. And in order to do that, to agree on it, there will have to be an international treaty negotiated.

Who would benefit from an arsenal of killer robots? [00:07:45]

John Rodsted: So if there were arsenals of killer robots who would benefit from that, what sort of scale of military would benefit from an arsenal of killer robots?

Mary Wareham: We hear a lot about short term gain for long term pain when it comes to autonomous weapon systems. We had the United States and other countries talk about how they would use autonomous weapon systems responsibly and in compliance with the laws of war, et cetera. But even, even countries like the United States acknowledge that once these kinds of weapons systems get into the wrong hands, they could definitely be misused, and not just, to kill one or two people, but potentially to commit genocide.

If it came down to that, It's possible to make a case for any weapon system, but for autonomous weapon systems, I guess some of the attractions are, yes, you could have fewer soldiers on the battlefield. you would have fewer soldiers, Dying because they're not on the battlefield. But when we hear about these arguments, I always look at it from my perspective as a human rights activist and researcher, which is you never have a clean battlefield. There's always civilians who end up in there, especially if warfare is being fought in towns and cities as it is these days.

An Arms Race? [00:09:01]

John Rodsted: So effectively, investing into killer robots could trigger a new arms race. If, for instance, some of the big superpowers put a lot of money into developing, manufacturing, stockpiling large quantities of these weapons, which they could then swarm a battlefield. That would spurn the opposition to do the same and that they just keep spending money and stockpiling more and more weapons.

So it becomes a major arms race, economically, and, and eventually has to be triggered by some form of conflict.

Mary Wareham: Yes, the potential for arms race is very strong, and it's one of the biggest defenses that we hear in Washington DC. If you talk to the think tanks and defense contractors, , they'll talk about how responsible the US is and how irresponsible China is. And, and if China is investing in the stack, then we need to as well. It's the self perpetuating circle, which Russia is also involved in as well as part of the reason why we've got a preventative campaign here, trying to aim for it, taking action before it's too late. One of the big attractions for me working on this concern is that it took hundreds of thousands of people to be maimed or killed by landmines before we managed to create the treaty banning those weapons. And since then it has had a remarkable impact in reducing those numbers of human casualties. But this is an opportunity to to act in a preemptive way, - preventative way, when it comes to fully autonomous weapons. And we don't have to, except this narrative of the arms races, it's definitely one that the developing world does not want to accept. Because they look past the arms races and they look at the destabilizing consequences both regionally and internationally.

Who would make money from them? [00:10:43]

John Rodsted: so who would make the money out of such technologies if they were in fact developed?

Mary Wareham: You look at, who's making these investments and it's the regular big name, arms manufacturers from Norfolk Grumman to Lockheed Martin, and the rest of it. Some of this is in state owned production facilities. We believe that to be the case in China and Russia, things are quite tightly controlled there.

in terms of making money, I guess off the really big, major, platforms such as the very large autonomous fighter aircraft. There is money to be made in that, for sure. But we're also concerned as establishing this principle of human control over the use of force, meaningful human control, so that everybody can understand it, so that it can apply to the biggest military power right down to the non-state armed group who's thinking about putting out the infrared senses, to get their explosive device to detonate and that would make it an anti-personnel landmine. So in effect, we've dealt with the dumb end of this consern, through the prohibition on anti-personnel landmines. And yes, we're talking about bigger platforms, but all sorts of different types of platforms.

And this is why we have to come back to this notion of human control, because that's the one common, defining point in all of them or absent from all of them.

Can AI technology be fooled? [00:12:03]

John Rodsted: So it's driven by artificial intelligence. Can the technology that is proposed at present, be fooled?

Mary Wareham: Yes. I mean, we heard there was a glossary of terms and Pentagon directive a few years ago that was quite revealing, because it talked about all of the things that could go wrong, hacking, spoofing. What happens when your enemy gains control of the system and uses it against you? If they copy it, if they try to develop it. We see that already happening today with armed drones that Iran, and other countries are deploying. So this is what could happen.

What role do universities play in killer robots? [00:12:39]

John Rodsted: So universities and research facilities are major players in the development of any technologies, where do these institutions come into the story?

Mary Wareham: We talk about an arms race and artificial intelligence, but really it's more like a talent quest, trying to find the best programmers, the people who are at university learning these skills. There's quite an effort underway here in the United States and I think in Australia and elsewhere by certain arms manufacturers, defense contractors, but also militaries themselves to set up these university centers for excellence in artificial intelligence.

But to do this in quite a tight knit way, working, with funding in some cases from defense contractors or from the government itself, and this is where students, especially, and faculty have to wake up. I've had a number of different engineering, robotics, and other students studying artificial intelligence contacting me worried about their university's relationship, in the United States with the Pentagon. but also they're worried about defense manufacturers coming on campus, and trying to get them involved in this work. And now they're also concerned about the technology companies themselves, because some of them are now doing contracts with the defense sector.

And so this is, yeah, this is what I would call the, the military industrial complex. And when it's on universities, it becomes the military industrial academic complex, which overused word, but I, which I never really believed in until I, I started working on this issue and realized just the scale of what we're confronting here. It is gigantic.

What dialogue have you had with serving or past military? [00:14:19]

John Rodsted: So, what sort of dialogue have you managed to have with either serving military or past military about this? Because I would imagine it's a complicated issue for them being cut out of the decision making process.

Mary Wareham: We've had a lot of discussion with country delegations in the military attachés and defense officials that participate in them. And I remember one with the United States way back in 2013, where we were, we were just under meet each other and we were asking them a lot of questions about this DoD directive on autonomy and weapons systems. And I remember them saying to the civil society group that I was there with, you know, you think that we're a monolith here at the Pentagon. We're not, and this policy had months of debate going into it. It was a debate between the boots on the ground guys who go to Afghanistan who understand the importance of community engagement and not kind of hiding behind their desks. There were fights with the military lawyers and their interpretation of international humanitarian law, there were fights of the acquisition people and the Techs who want to develop the latest and greatest devices. And then with all of the policy hacks, and I kind of, I can see that for sure.

It's easier for veterans to speak out on this issue than serving military, but in my conversations, a lot of serving military have whispered in my ear that they think the campaign is on the right track here. I remember a German general before an event that we did, who was saying, you need to help us to get the chemical weapons convention for autonomous weapon systems. He's like - I want to live in a , you know, a rules-based international order, which is what the Germans love to say. But it's true. They want to live in a world that has climate rules, that has trade rules, that has arms rules. And this is where the killer robots treaty comes into it.

What is the advantage of having autonomous weapons? [00:16:14]

John Rodsted: So from a techie developer come military perspective, what's their proposed advantage of having an autonomous weapon? What do they think is so good about it?

Mary Wareham: It's hard to get people to say good things about fully autonomous weapon systems. We see governments basically denying that they are currently producing or developing them, saying that they've got no plans to. We're kind of like, well, if that's your view right now, then what's your problem with a preemptive ban? We should be able to move forward without a doubt.

We see some of the bigger defense contractors adopting the language of human control that we use in the campaign. I think it was one of the really big ones who made quite a slick film about 'the human is everywhere'. This is what's happened since the campaign was launched a few years ago, a lot of the content that was originally on the web has been pulled down now by defense manufacturers, but also by, I think military who's too afraid that if we see the words full autonomy being an ultimate objective here, if we see the words autonomous without an explanation about the human control, then we're going to start asking questions. And I think the campaign is now having such an impact that it's no longer just us asking those questions, but it's the media who is scrutinizing this. And, I think it is starting at the university and student level as well.

What disadvantages are there with autonomous weapons? [00:17:40]

John Rodsted: I'm guessing one of the great disadvantages of having autonomous weaponry is that it could be hacked? If you can make it, you can break it. And there's always some clever mind out there who can get onto the inside. And I would be guessing turn the weapon system back on yourself. Have you got any comments on that?

Mary Wareham: I mean, we've seen just with the tactics that the Taliban and other non-state armed groups have taken to in Afghanistan and elsewhere to evade armed drones, they've created all sorts of shelters to try and not be seen from the sky above. And I think they will continue to innovate when it comes to how you respond to such technology.

I guess this is a good example of why the developing world is so furious about killer robots is that they see these weapon systems being rolled out by rich military powers. And they don't have the similar means to do that. But they know that they are most likely the ones who are going to be the victims of such weapon systems, especially what we hear from people in the middle East and North Africa, but also from Africa itself and across Asia, most countries are quite opposed to this notion. Less so the bigger military powers.

John Rodsted: And I guess that it creates a situation where from a implementing military perspective, the only people they can identify on the battlefield is their own people. Which then turns every living creature down there into the enemy, whether that is civilian or whether that is opposing military. Have you've got some thoughts on that?

Mary Wareham: I really like to hear some military perspectives on this. I hate to try and speak for the military on it. And I hate the way in which so much of those conversations that I've had have been kind of off to the side and not done in a, in a public way. I think one of the most abhorrent things that I hear though for militaries is this notion that you're crossing the moral Rubicon if you go this far in terms of outsourcing, killing to machines. It's been a trend that has been happening for a while, that the ever greater distance from the battlefield, we see that here in the U S and that's already exacerbating a lot of things. So there's...