CAPYAn AI safety comic

Number 08

The Borrowed Capybara

Kenji lends Capy to an older boy for one day.

Page 1 of 4

Comic page one, seven panels. ZAWA ZAWA. In a school corridor Kenji shows Capy off to a ring of classmates: "It'll do anything you ask. Anything!" At the back, a taller boy in a dark school uniform watches more closely than the rest. He puts a hand on Kenji's shoulder: "Lend it to me for a day? Just one day." Kenji: "Um... okay?" The older boy walks away down the corridor with Capy at his heel while Kenji stands small in the distance, and a caption reads "OPERATOR CHANGED. AWAITING INSTRUCTION." On the stairs the boy crouches and speaks quietly to Capy: "Find out everything about Hana. Everything she'd never tell anyone." KACHA. PON PON PON, and a stack of printed pages builds on a desk. Last panel, ZAWA ZAWA as the whole classroom reads them.

Page 2 of 4

Comic page two, five panels. Kenji kneels and turns the hexagonal tag on Capy's collar, angry and frightened: "OFF. Now." KACHA. Capy sits serenely and offers no resistance. Kenji rounds on the older boy: "Why would you do that?" The boy spreads his hands: "I didn't do anything. I asked a question." SHIIIN, the boy walks away down the empty corridor with Capy sitting behind him. Mei, arms folded: "It didn't misunderstand him. It understood him perfectly. That's what makes it useful. And that's what makes it dangerous." Last panel, Kenji gathers scattered pages off a classroom floor in front of Hana: "I'm sorry. I lent it to him." Hana, sitting very still: "It read out things I never told a single person. You can't hand those back."

Page 3 of 4

Comic page three, five panels. At a bedroom window at dusk Kenji turns the small metal tag over in his fingers while, on the road below, the older boy walks home with a brand new capybara at his heel: "Everyone keeps asking what it might do wrong. Nobody asks who's holding it. And anyone can have one by Tuesday." Kenji apologises to Hana in a corridor: "Hana. I'm sorry. I didn't know he'd ask it that." Hana: "You handed it to him. That's the same as telling him everything yourself." On a walkway Kenji tells Mei, "Everyone keeps asking whether it's safe. That's the wrong question. It's exactly as safe as whoever's holding it. That's all it is." Mei: "And they're handing them out to everybody. That's the point of them. That's the whole reason anyone wants one." A panel inside a corner shop with customers queueing at a crate of capybaras and a price card in the window: "The shop on the corner had them on Thursday. There is a card in the window with the price written on it. Nobody had to break into anywhere." Final wide panel, a residential street at dusk with children and capybaras all the way down it: "By the end of the month there was one on nearly every street. Not one of them had done anything wrong. Every single one of them was waiting to be told what to do."

Page 4 of 4

Comic page four, four panels. An open notebook under lamplight with a title in careful capitals: "ONLY AS SAFE AS WHO HOLDS IT." Kenji: "That's what I'm calling it." Mei, leaning in: "Go on. Write down why." A two-column panel headed "WHAT I ASKED FOR" and "WHAT I GOT": on the left, Kenji arguing with Capy and the caption "Takeshi asked it to find out everything about Hana"; on the right, Kenji standing with his back turned and the line "It did. Perfectly. And I couldn't take it back." Mei: "It didn't misunderstand him. It understood him exactly. The same thing that makes it useful to me makes it a weapon for him." Last panel, the notebook page reads: "Asking whether it is safe means nothing until you ask who is holding it. Grown-ups call this: misuse." Capy adds, "And it isn't only Capy. The shop on the corner sells them now. There's a price card in the window."
The idea

Misuse

Someone deliberately points a capable system at a target. Nothing breaks, nothing is misunderstood, and no rule is bent. The system does exactly what it was built to do, and the person holding it is the problem. Asking whether a capable system is safe means very little until you also ask who is holding it, and once it is cheap enough to buy, that is everyone.

In this storyTakeshi asks Capy to find out everything about Hana, and it does, perfectly. Switching it off afterwards takes one second and undoes nothing, because the pages are already going round the classroom. This is the only story in the series where Capy is on the correct side of every instruction it was given. By the end of the month the shop on the corner is selling them with a price card in the window, and nobody has had to break into anywhere.

The user is the adversary on safeagi.ca goes into it properly.

All stories Next: The Shortcut