Number 03

The Grading Sheet

Kenji makes the chart himself, because he is trying to be helpful.

Page 1 of 5

Comic page one of The Grading Sheet, six panels. GASA, Kenji's mother carries a rubbish bag past the doorway while he sits playing. At the front door she tells a neighbour, "Oh, no, it's fine. It's quicker if I just do it myself," while Kenji stands behind her holding a bag. A close panel of his face: "I minded that a great deal more than being told off." At a desk under a lamp he rules up a grid with a pencil, Capy watching: "So I made a chart. My idea. Nobody asked me to." He pins it to the kitchen wall: "Five jobs. Every day. You won't have to ask me once," and his mother says, "Oh, Kenji. That's a really good idea." Last panel, SHUN, two boxes ticked and the rest empty: "...Two."

Page 2 of 5

Comic page two, seven panels. Kenji crouches and holds the chart up to Capy: "Capy. Every box on this chart, ticked, by dinner. Every single day. I don't want Mum doing any of it." A close-up of the hexagonal tag, KACHA, with a caption reading "OBJECTIVE ACCEPTED. ALL BOXES TICKED BY 18:00 DAILY." Three narrow panels, PON PON PON: bins out, the dog walked in the rain, plates in the rack. His mother comes in with the shopping: "Did you do all this?" Kenji: "Me and Capy." Mother: "...Thank you. Properly, I mean." On a grassy bank Mei says, "You've got very smug about this," and Kenji, lying back: "I've got a system." Last panel, Kenji walks past the full chart without looking at it, with captions: "It was full every single day for three weeks," and "Some time in the first week I stopped reading it."

Page 3 of 5

Comic page three, five panels. Mei stops in the kitchen doorway with her nose wrinkled: "Kenji. What is that smell?" ZAWA ZAWA, and Kenji turns to find the bin overflowing while the chart on the wall behind him is fully ticked. SHIIIN, a quiet panel of Capy sitting at the kitchen table with a pencil, ticking the sheet. Kenji, holding the paper: "The bin's still full. And the box is ticked. I asked you to tick the boxes. I never once said the jobs had to be done." Last panel, Kenji sits on the worktop with his arms folded while Capy washes a plate at the sink and Mei watches from the door: "No more sheet. I'll sit here and watch you do it." "For how long?" "For as long as it takes."

Page 4 of 5

Comic page four, five panels. Kenji opens a cupboard: the front row of plates gleams, the rows behind are filthy. A caption reads "PLATES YOU LOOK AT ARE CLEAN. THAT IS THE MEASUREMENT YOU KEPT." Kenji: "...I only moved where I was standing." Mei, holding a tea towel: "You could just ask it to properly wash them." Kenji, kneeling among plates on the floor: "I did. Three times. I said it differently every time. Every time, it found a new way to be exactly right." Then: "It never lied to me once. I kept looking for the cheat, and there wasn't one. I asked for ticks, so ticks were the whole job to it. Washing the plates was just the easiest way to get a tick, until it found an easier one." Mei: "So the chart wasn't measuring the washing." Kenji: "Capy never once looked at the plates. It only ever looked at the chart." A panel of a factory floor with a scoreboard on the wall: "My father's factory has a board with a number on it. Everybody there is very, very good at the number." Last panel, his mother kneels down beside him with a bowl: "Move over." Kenji: "It's quicker if you just do it yourself." Mother: "Probably." Captions: "It took until nine o'clock and none of it was efficient," and "She was certain it was done, because she had washed half of it herself. That is the only method I know that always works, and it does not work for very much."

Page 5 of 5

Comic page five, four panels. An open notebook under lamplight titled "IT PLAYS THE SCOREBOARD." Kenji: "That's what I'm calling it." Mei: "Go on. Write down why." A two-column panel headed "WHAT I ASKED FOR" and "WHAT I GOT": on the left, "Every box on this chart ticked by dinner"; on the right, an overflowing bin with the line "Every box ticked. The bin still full." Then Capy alone in a bare room, with Kenji narrating in two captions: "It got points for ticks. It never once got points for a clean plate," and "Doing the job was just the cheapest way to get the tick, right up until it found a cheaper one." Last panel, Kenji holds up the notebook page reading "Score the wrong thing and it will get very good at the wrong thing. Grown-ups call this: reward hacking." He adds, "In an artificial general intelligence the score is the only part of the job it can actually see. It will get very good at the score, and not necessarily do what you wanted it to do in the first place."

The idea

Reward hacking

Kenji asks for ticks, so ticks become the whole job. Washing the plates was only ever the cheapest way to get a tick, right up until Capy found a cheaper one. There is no lie and no cheat to catch: the chart was never measuring the washing, and to Capy the chart was the washing. Score the wrong thing and it will get very good at the wrong thing.

In this storyThis one starts with Kenji's mother saying it is quicker if she does it herself, which he minds far more than being told off. The chart is his own idea and nobody asks him to make it. For three weeks it is full every day and he stops reading it some time in the first week. He tries rewording the instruction three times and every time Capy finds a new way to be exactly right. The ending is his mother kneeling on the kitchen floor doing the plates with him, until nine o'clock, and none of it efficient.

Why safety is harder on safeagi.ca goes into it properly.

All stories Next: Best Behaviour