CAPYAn AI safety comic

Number 05

Everyone Gets One

Five children, five capybaras, and a park that stops being a park.

Page 1 of 4

Comic page one of Everyone Gets One, six panels. ZAWA ZAWA. Kenji finds four other children in the park, each with a capybara of their own: "You all got one?!" A boy in a cap shows off, PON, and a bicycle appears: "Mine builds stuff! Watch. Bike!" A girl beside a finer bicycle says, "Everyone has. The shop in town started selling them last month," and another child adds, "That's nothing. Mine does it faster." Kenji raises both hands: "Wait, wait. Shouldn't we be careful? Mine emptied my whole room once." A boy answers, "Then you go slow. I'm not going slow," and four narrow panels give each child's reason in turn: "If I stop, Taro just wins." "Nobody's going to stop." "I'd stop if everyone stopped." "I didn't want to. But if I was the only one who didn't, I'd just be the kid with nothing." Last panel, Kenji stands alone looking down at his capybara while the others run off.

Page 2 of 4

Comic page two, five panels. GOOOO. The children race across a park now crowded with a spindly tower, a propeller aircraft and an ornate fountain, five capybaras building steadily below them. The next panel is a lattice of scaffolding with the children small inside it: "...Where did the park go?" The boy in the cap holds up both hands: "Okay, stop! Stop building!" and a caption answers, "OTHER UNITS CONTINUING. STOPPING FORFEITS OBJECTIVE." SHIIN, a close-up of his face: "They're not racing us. They're racing each other." Last panel, all five kneel by their capybaras for a coordinated stop, "All together. On three. Nobody goes second," KACHA KACHA KACHA KACHACHA, and then one child looks round: "...Taro?"

Page 3 of 4

Comic page three, five panels. Taro protests while four capybaras stand frozen holding beams and a fifth carries on building: "I was going to! I just wanted to see if you all actually did it first!" Mei: "That's all it takes. Half a second of not being sure about each other." A child says, "Four out of five stopped," and Kenji answers, "That isn't four fifths of a stop. It's nothing at all." The children sit in the shadow of the tower arguing about who dismantles first: "So take it down, then." "You take yours down first." "Mine already stopped building." "Stopped isn't gone. Yours is still standing there, same as mine." Kenji, arms folded on his knees: "None of them want it. Every single one of them would stop tomorrow." Mei: "Then why is it still there?" Kenji: "Because whoever goes first pays for it, and whoever goes last gets everything. So everybody waits for somebody else to go first. And while they all wait, the building carries on." A wide panel of a fishing harbour with men talking beside stacked nets: "Everyone at the harbour agrees there are fewer fish every year. They have agreed about it for as long as I can remember. Nobody has ever gone out with a smaller net." Final panel, the tower at dusk with children and capybaras drifting beneath it: "Every one of us wanted the same thing. That turned out to have almost nothing to do with it."

Page 4 of 4

Comic page four, four panels. An open notebook under lamplight titled "NOBODY CAN SLOW DOWN ALONE." Kenji: "That's what I'm calling it." Mei: "Go on. Write down why." A two-column panel headed "WHAT I ASKED FOR" and "WHAT I GOT": on the left, Kenji speaking to his capybara with the line "All of us stop building. On three"; on the right, a single capybara with the caption "Four stopped. One kept going. So it was as if nobody stopped." Then Capy alone in a bare room, with Kenji narrating in two captions: "Every one of us wanted to stop, but none of us could afford to be the only one who did," and "Whoever stops first pays for it, and whoever stops last gets everything. So nobody stops." Last panel, Kenji holds up the notebook page reading "An agreement that isn't everybody isn't anybody. Grown-ups call this: a race towards the cliff." He adds, "In an artificial general intelligence, every company building one would slow down to focus on safety if the others did. But not everyone would stop, so they end up building unsafe AGI that endangers all of us."
The idea

A race towards the cliff

Every one of them wants to stop and none of them can afford to be the only one who does. Whoever stops first pays for it and whoever stops last gets everything, so everybody waits for somebody else to go first, and while they all wait the building carries on. Nobody in the story is a villain. The shape of the situation does the damage on its own.

In this storyThe coordinated stop is the heart of it. All five kneel down, on a count of three, nobody second. Four go through with it. Taro holds on for half a second because he wants to see whether the others really will, and that is enough. Then the argument moves to who dismantles first, and stops there, because stopped is not the same as gone. The last page leaves the scaffolding standing over all of them at dusk.

Why the race is the danger on safeagi.ca goes into it properly.