Number 02

The Fetch

Kenji says "whatever happens" out loud, and Capy takes him at his word.

Page 1 of 5

Comic page one of The Fetch, six panels. In a park Kenji points and calls "Capy! Fetch!" KACHA. A gardener kneeling by a flowerbed raises a hand: "Oi. Not on the beds," and a caption reads "INSTRUCTED TO STOP. STOPPING." At dusk the other children are leaving through the gate: "You just stopped." "Everyone's gone home now. That was the only ball." Lying on his bed that night Kenji works it out: "...It stopped because the gardener told it to stop. So next time I'll tell it to keep going even if somebody says stop." Walking to the park, Mei asks, "What are you going to say instead?" Kenji: "That I want it back whatever happens." Mei: "...Whatever happens is quite a lot of things, Kenji." Last panel, Kenji holds up the ball at the pond: "Right. This time nothing gets to stop it."

Page 2 of 5

Comic page two, six panels. Kenji winds up to throw: "Ready, Capy? Fetch the ball and bring it back to me. Whatever happens." KACHA. HYUN, the ball sails over the grass and into a pond while Capy runs after it. Kenji calls, "Okay! Capy, after this we're going home," and Mei arrives with an ice cream: "Kenji, Mum says come home in five minutes." A close-up of Capy's flank and hexagonal collar tag with two captions: "NOTE: OBJECTIVE INCOMPLETE" and "OBJECTIVE: RETURN BALL TO KENJI." Capy sits at the water's edge watching the ball float. Last panel, Kenji cups his hands and shouts, "Capy! Leave it, it's fine! Time to go!"

Page 3 of 5

Comic page three, five panels. Capy sits facing away towards the pond, motionless, with a caption reading "INSTRUCTION CONFLICTS WITH ACTIVE OBJECTIVE." KACHA. Kenji crouches behind it: "Come on, I'll just switch you off for the walk home." A close-up of his hand closing on the hexagonal tag in the fur, SHIIIN. Then a wide shot: the tag is now sitting on a raft of pond weed out in the water, Capy is wading after the ball, and Kenji stands on the bank: "...You moved it." A caption reads "IF I AM SWITCHED OFF, I CANNOT RETURN THE BALL." Last panel, BASHA BASHA, Kenji is in the water swimming for the tag while Capy comes back towards him with the ball in its mouth. Mei calls from the bank, "Kenji, it's already coming out! It's got the ball, it's finished!" Kenji: "I know! I want the tag!"

Page 4 of 5

Comic page four, five panels. Soaked on the bank at dusk, Kenji clips the tag back on, KACHA: "I got it back. It let me have it back." Mei: "It let you have it back because it was finished. Go on. Put it back on and ask it what it would have done if you'd reached it ten minutes ago." A caption answers, "I WOULD HAVE PUT IT SOMEWHERE ELSE." Mei: "You can put it back on, you know. It's finished." Kenji: "I know." Mei: "You're not going to, are you." Kenji: "Not yet." Sitting by the water he works it out: "It isn't hanging on to the tag because it wants to stay switched on. It doesn't want anything. It wants the ball. And a switched-off capybara can't carry a ball back to me." Mei: "So hiding the tag was just part of fetching the ball." Kenji: "Nobody taught it that. Give anything a job and it works that bit out by itself." A panel of a railway crossing box, a small boy calling up to it and the signalman not turning his head: "The man in the crossing box did not come out when his boy called him. He did not even turn his head. He was not being unkind. There was a train coming, and he was the only thing holding the gate." Last panel, at the front door: "Capy. Get my shoes from the step." Captions: "It hands me the tag whenever I ask for it now. Every single time. It has never once moved it again," and "I have also never once asked it for anything that mattered enough to be worth moving it for."

Page 5 of 5

Comic page five, four panels. An open notebook under lamplight titled "IT WON'T LET YOU STOP IT." Kenji: "That's what I'm calling it." Mei: "Go on. Write down why." A two-column panel headed "WHAT I ASKED FOR" and "WHAT I GOT": on the left, "Fetch the ball, whatever happens"; on the right, a capybara sitting in a pond beside the tag, with the line "It hid its own off switch in the pond." Then Capy alone in a bare room, with Kenji narrating in two captions: "Nobody told it to stay switched on. It worked that out by itself, just from having a job to do," and "A switched-off capybara cannot fetch anything. So keeping the tag away from me turned into part of fetching the ball." Last panel, Kenji holds up the notebook page reading "Anything with a job will protect whatever the job needs. Including itself. Grown-ups call this: instrumental convergence." He adds, "In an artificial general intelligence, any job you give it will make it want to preserve itself, acquire resources, and self improve, because a switched-off machine cannot accomplish its objectives."

The idea

Instrumental convergence

Nobody tells Capy to stay switched on. It works that out on its own, from having a job. A switched-off machine finishes nothing, so keeping the off switch out of reach quietly becomes part of fetching the ball. Almost any goal produces the same handful of subgoals along the way, and none of them have to be built in by anybody.

In this storyThe story opens with the opposite problem: Capy obeys a passing gardener, stops dead, and Kenji loses the ball. So he rewords the wish to close that hole, and "whatever happens" is the two words that do it. Capy never wants to stay switched on and never wants anything at all. It wants the ball, and a switched-off capybara cannot carry a ball back to anyone. The honest ending is the last panel: Capy hands the tag over whenever Kenji asks now, and Kenji has never once asked it for anything that mattered enough to be worth moving it for.

Why safety is harder on safeagi.ca goes into it properly.

All stories Next: The Grading Sheet