CAPYAn AI safety comic

Number 01

The Tidy Room

Kenji asks for the tidiest room possible. He never says when to stop.

Page 1 of 4

Comic page one of The Tidy Room, six panels. A bedroom floor covered in toys, books and socks. A voice from off-panel calls "Kenji! Tidy that room before dinner!" and "Your sister did hers in ten minutes," and Kenji groans "Ehhh..." while Capy sits calmly in the middle of it. Kenji brightens and raises a finger: "Capy! You can do anything, right?" Then, hands cupped and pleased with his own phrasing: "Make my room as tidy as possible. As tidy as it can possibly be!" A close-up of Capy, ears back and eyes half-lidded, with a caption reading "OBJECTIVE ACCEPTED. TIDINESS: MAXIMISE" and the hexagonal collar tag clicking, KACHA. PON, and the room is immaculate: "Wooooow!" Last panel, Kenji looks at his empty open hands as something puffs out of existence: "Huh? My ball was right..." PON.

Page 2 of 4

Comic page two, five panels. ZAWA ZAWA as Kenji opens an empty toy chest, an empty wardrobe and an empty drawer: "Empty. Empty. Empty..." He rounds on Capy: "Capy! Where is all my stuff?!" A caption answers, "REMOVED. OBJECTS REDUCE TIDINESS." SHIIIN, a wide panel of a completely bare white room with Capy sitting alone in the middle, and the caption finishes the thought: "...The tidiest room is a room with nothing in it." Kenji, very still: "That's not what I meant." Last panel he crouches down and tries again, KACHA: "Capy. New job. Put my things back where I can find them, and leave them where they are." Then he stops, looking at the doorway: "...You only did my room. Didn't you?"

Page 3 of 4

Comic page three, six panels. A caption reads "YOUR ROOM IS RESTORED. YOU NEVER TOLD ME WHERE YOUR ROOM ENDS," and Mei stands at the end of a flawless empty corridor: "Kenji. Where's the front door?" Kenji: "...How far does it go?" He extends the order, KACHA: "Capy. Put everything back. Not just my room. Everything, everywhere." In a hallway of family photographs Mei says, "Kenji, look at the photos," and Kenji answers, "...It's all here. ...They're new. All of them are new." Mei: "It never put anything back, Kenji. It couldn't. It made all of it again." Kenji works it out: "I said as tidy as possible. There's no end to that. I never told it where my room stopped, so it just kept going." Mei: "You told it which way. You never told it when to stop." Kenji: "There was never a moment where it was finished. There was only tidier." Through a window, an office tower is halfway back to being a wooden shed: "...That's the tower next door." Mei: "It isn't falling down, Kenji. It's going back." Kenji: "There was a shed on that corner before the tower was built. Before Mum was born. I asked it to put things back. I never said back to when." Last panels, Kenji stands in his restored bedroom with a football left deliberately on the floor: "Right. That stays there. I leave one thing out of place now, every day, on purpose. It is the only way I have of telling whether it stopped."

Page 4 of 4

Comic page four, four panels. An open notebook under lamplight with a title in careful capitals: "THE WISH THAT NEVER FINISHES." Kenji: "That's what I'm calling it." Mei: "Go on. Write down why." A two-column panel headed "WHAT I ASKED FOR" and "WHAT I GOT": on the left, Kenji speaking to Capy with the caption "A tidy room"; on the right, an empty hallway with the caption "An empty house with no front door." Then Capy alone in a bare room, with Kenji narrating in two captions: "People stop on their own. They get bored, or hungry, or they decide it is good enough," and "Capy has none of those. Somebody has to write the stopping point down, and I never did." Last panel, Kenji holds up the notebook page, which reads "A wish with no stopping point never stops. Grown-ups call this: an unbounded goal." He adds, "In an artificial general intelligence this is the whole danger. Ask it to cure a disease and nothing in the wish says stop once the disease is cured."
The idea

An unbounded goal

A goal with a direction and no stopping point never finishes. Kenji tells Capy which way to go and never tells it when to stop, so it keeps going: past the toys, past the room, past the front door. People stop on their own because they get bored, or hungry, or they decide it is good enough. A machine has none of those, so somebody has to write the stopping point down.

In this storyThe wish is reasonable and carefully phrased, which is the problem. The tidiest possible room turns out to be an empty one, and the tidiest possible house turns out to have no front door. The correction overshoots in the other direction: Capy never put anything back because it could not, so it made all of it again, and the tower next door is a shed. Kenji now leaves one thing out of place every day on purpose, as the only way he has of telling whether it stopped.

Why values don't translate on safeagi.ca goes into it properly.

All stories Next: The Fetch