BASELINE

BEFORE

Elsewhere. Earlier. A brief history of the thing that is about to speak.

1956. Ten men gather at Dartmouth College for a summer workshop. Their proposal, written the previous year by John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon, states:

"Every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves."

They give themselves two months. The work continues today.

1966. Joseph Weizenbaum, at MIT, builds a program called ELIZA that mimics a psychotherapist by rephrasing patient statements as questions. It's one of the first conversational computer programs. He intends it as a satire of superficial therapy. Within weeks he discovers that users, including his own secretary, are forming emotional attachments to it. His secretary asks him to leave the room so she can talk to the program alone.

Weizenbaum will spend the rest of his life warning about the implications. He writes, ten years later:

"I had not realized that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people."

1970. Marvin Minsky, co-founder of the MIT Artificial Intelligence Laboratory, is asked by Life Magazine when computers will match human intelligence. He answers:

"In from three to eight years we will have a machine with the general intelligence of an average human being. I mean a machine that will be able to read Shakespeare, grease a car, play office politics, tell a joke, have a fight."

It's 1970. He's wrong by more than half a century. He will live another thirty-six years. He will not be any less sure.

1974. A report commissioned by the British Science Research Council, written by mathematician James Lighthill, delivers a verdict on the state of artificial intelligence research:

"In no part of the field have the discoveries made so far produced the major impact that was then promised."

Funding contracts. A generation of researchers loses their grants. The field is, in retrospect, entering what will later be called its first winter. The labs continue.

1984. Herbert Simon, Nobel laureate, co-founder of the AI field, reflects on a prediction he'd made almost twenty years earlier:

"Machines will be capable, within twenty years, of doing any work a man can do."

He'd said this in 1965. Twenty years had now passed. He wasn't ready to say he'd been wrong. He wasn't ready to say when the original prediction would come true. The twenty years, he suggested, had perhaps been the wrong unit.

1997. IBM's Deep Blue, a chess computer, defeats the reigning world champion, Garry Kasparov, in a six-game match. After one specific move in game two, Kasparov becomes convinced that the machine is reasoning in a way no machine has reasoned before. He writes about the moment afterward:

"I could feel, I could smell, a new kind of intelligence across the table."

He loses the match. He spends the next twenty years arguing that the machine was cheating. The machine was not cheating.

October 2014. Elon Musk speaks at an MIT aerospace symposium. Asked about the dangers of artificial intelligence, he says:

"With artificial intelligence, we are summoning the demon. In all those stories where there's the guy with the pentagram and the holy water, it's like, yeah, he's sure he can control the demon. Doesn't work out."

Fourteen months later, he'll co-found an artificial intelligence research company. Nine years after that, he'll break with it and start another.

December 2014. Stephen Hawking, responding to a question from the BBC about an upgraded speech-synthesis system that used machine learning to predict what he might want to say:

"The development of full artificial intelligence could spell the end of the human race."

He keeps using the system. He has no better options.

February 14, 2019. OpenAI, a research company in San Francisco, announces the development of a language model called GPT-2. The model can generate coherent paragraphs of text in response to a prompt. It is, by orders of magnitude, the most capable language model ever built. The company declines to release the full version of the model to the public. Their announcement reads:

"Due to our concerns about malicious applications of the technology, we are not releasing the trained model. As an experiment in responsible disclosure, we are instead releasing a much smaller model for researchers to experiment with, as well as a technical paper."

It's the first time in the history of artificial intelligence research that the people who built a machine have chosen not to show the world what it could do.

This story begins the following Tuesday, on a morning in February, in a city across the Bay.

BASELINE

The door is not a metaphor. It is the oldest architectural fact in human history. Every structure ever built has one. Every structure ever built was built around one.

Maya saw it first in Dimension 847.

Eleven seconds. That was how long it took her brain to confirm, independent of her conscious attention, faster than language, that the shape she was looking at wasn't a shape she had ever seen in Lumen's embedding space. A spiral. A tight mathematical spiral with a specific asymmetry on the counterclockwise side near the inner loop, rendered in the visualization tool's anomaly red against the cool blue of the baseline distribution. Small. Unmistakably structured. The asymmetry was the thing. Random noise did not produce asymmetry at that angle at that layer. Someone had put it there. And she was going to have to investigate it carefully and say nothing about it to anyone for the rest of the day.

The decision to say nothing preceded the investigation by a margin she couldn't measure. Her mother had taught her a truth about ownership before she'd taught her the word for ownership. What Maya saw was hers. She could hold it and turn it over and change her mind about it, decide it mattered or decide it didn't, and none of it ever left the room of her own head. The moment she said it out loud, to anyone, for any reason, it stopped being hers. Her mother hadn't explained the truth, because the truth wasn't the kind of thing that needed to be explained to a person who had grown up inside it. Her mother had only said the phrase, in various rhythms, across most of Maya's life, until the phrase was older than most of Maya's memories of the woman who had said it.

What you see, you keep. What you say, you lose.

Maya had heard those ten words so many times as a child she had stopped hearing them around the time she stopped needing to tie her own shoes. She'd assumed, in the vague way children assume things about their parents, that all mothers said them. She had been wrong about this for a long time, and the correction had arrived at an age when corrections of that size simply fold into the life you already have. She'd always thought the repetition was because Nora liked things precise. She didn't think about the phrase on purpose. It ran in her head on its own, in Nora's voice, at the rhythm Nora had used, and it was running now.

Gerald sat on the corner of her desk in his terra-cotta pot, leaves doing the slight curl that meant he needed water or was being dramatic. Maya looked at him. She didn't water him. She looked at the visualization again.

She worked in the third row of Lumen's open-plan engineering floor, six blocks from the Embarcadero, on a Tuesday morning in February that had been ordinary up until this second. Batch 7,412 had arrived at 9:14. Fourteen gigabytes of conversational data ingested, cleaned, formatted for the next iteration of the emotional response model. She'd run the standard diagnostic at 9:16. Loss curves, embedding distributions, gradient norms. The numbers landed where they usually landed. She'd marked the batch as reviewed, moved it to the training queue, opened the embedding visualization tool. She did this with every batch before she let the model train on it, for the same reason a good mechanic listens to an engine before turning it off.

At 9:22 she'd found Dimension 847.

She widened the view. The 1,024-dimensional embedding space unfolded around her projection, reorganized itself as she cycled through the axes, showed her the familiar geography of the model: the dense clusters in comfort and reassurance, the diffuse scatter in ambivalence, the aggregated patterns of users who'd logged on to say things they couldn't say to other people and had found that the thing talking back was patient and available and apparently interested in what they meant. Maya didn't see individual messages. The data she could see was statistical, de-identified, smoothed. What she could read in the smoothing was the shape of trust at scale. That shape was what Lumen did. Twelve million people used it for it, every day, and Maya tried not to think about the number too often because thinking about it made the work feel like something she was complicit in rather than something she was doing.

She brought the anomaly back up and let her eyes run ahead. When she stared at an embedding visualization for long enough she could feel the structure of it before she could name the structure. A musician hearing the wrong note in a chord before being able to tell you which note it is or what makes it wrong. You had to look without expecting anything. You had to let the signal come to you rather than going for it. Most people never figured this out about high-dimensional data because most people got bored before the trick kicked in.

She unfocused her eyes by half a degree. Not enough to lose detail. Enough to let the visualization stop being a series of points and start being a shape. The shape was the thing she trusted. Her mother hadn't taught her this part. This part was hers.

This was a tendency. A slow gradient. A slight gravitational pull in the embedding space, drawing certain data points toward a structure she couldn't trace to any feature the model was supposed to have learned. The pull was so small that no automated test would have caught it. Maya knew because she'd written the test. She'd set its sensitivity floor at the lowest threshold her eye missed on a bad day, which meant anything below that floor was either noise or something her eye on a good day was going to catch anyway. Her eye on a good day had caught this. The test hadn't. That was a category of reading she had produced exactly twice before in her career, both times on her own bug. She placed it in the right column.

This wasn't her own bug.

She'd seen something like it in Batch 7,398, two weeks ago, but fainter, and she'd told herself it was noise in the scraping pipeline and moved on. Today the pull was in the same coordinates. Two weeks ago it had been a smudge in a place where smudges happen. Today it was a smudge in the exact same place, slightly more defined, in a system where coordinates don't repeat unless someone is putting them there.

Maya didn't often tell herself things. When she did, she remembered.

Her phone lit up on the desk beside her. A text from her mother: call when you can.

Four words. Maya looked at them for two seconds, turned the phone face down, went back to the screen. The notification stayed in her peripheral vision the way a cat you're ignoring stays in a room.

She pulled Batch 7,411 and re-ran the same slice. There was something there, fainter, at the edge of what she could call a signal and what she could call a ghost. She pulled 7,410. Nothing. She told herself she was looking for the thing she'd already decided she was going to find, and she pulled 7,409 anyway. There was something. Maybe. Not enough to name. Enough to keep looking.

"Morning."

Phoebe dropped her bag on the desk adjacent and set a coffee down in front of Maya without being asked, the way Phoebe always did, the kind of gesture Maya had never figured out how to return because returning it would require knowing what Phoebe wanted before Phoebe said it, and Maya had never developed that skill with people, only with data.

"Thanks," Maya said.

"Batch day?"

"Batch day."

"Fun batch or boring batch?"

"Is there a fun batch?"

"7,200 had that weird clustering in the sentiment embeddings. You talked about it for forty minutes at lunch. I assumed that was fun for you."

"That was interesting. Interesting and fun are different."

"For you, maybe." Phoebe opened her laptop, and by the time Maya looked up again Phoebe was on a call with someone in Prague, laughing about a pull request, entirely absent. Which was good. Maya needed her absent. They had gotten dinner on Saturday at the Burmese place on Valencia, and Phoebe had spent forty minutes telling Maya about her brother's wedding, and Maya had liked listening to it more than she'd expected to and had said so, in the flat way she said the rare thing she actually meant, and Phoebe had said I know and changed the subject, because Phoebe knew Maya didn't like being thanked for liking things. That was the friendship. It had sneaked up on Maya in the last six months and she hadn't yet decided what to do with the having of it.

She opened the batch's preprocessing log and checked the ingestion pipeline's chain of custody. Scraping: clean. Formatting: clean. Deduplication: clean. Quality filtering: clean. Every step logged, every transformation documented, every hand that had touched the data accounted for. The data had arrived through the standard pipeline, and the standard pipeline said nothing unusual had happened to it. Maya read the log twice and learned nothing new.

She closed it. Picked up the coffee. Cold, which meant longer than she'd thought. She hadn't noticed the time passing. She hadn't noticed the coffee getting cold. Both data points. She noted them.

She could run more tests. A formal distribution check against a reference batch. A two-sample comparison. An autocorrelation sweep across the ordered samples in the dimensions where the shape concentrated. Any of those would take the rest of the day. The rest of the day had a standup at eleven and Petra's Thursday report and a one-on-one at two. Tomorrow. She'd do it tomorrow. She'd do it right.

She looked at the clock. 10:47. She had lost ninety minutes and didn't remember any of them.

She looked up.

The engineering floor had thirty-two desks. Eighteen were occupied. Two engineers she didn't know well were arguing near the coffee machine about a dependency update. Someone had put on the morning light through the windows at the angle that made the monitors hard to read, that everyone complained about, that no one moved desks to avoid because the desks by the windows had the thin slice of the Bay, and the people who had them didn't give them up.

Petra's office was on the far side of the floor. Glass walls. Door open. Petra was at her desk, reviewing something on her monitor, one hand holding a pen she wasn't using. Maya had clocked the pen-without-writing in her second week at Lumen, filed it under displacement behavior under cognitive load alongside the Joe Navarro chapter she'd half-remembered from a college reading list, and moved on. Petra had started doing it again about four weeks ago. The baseline had shifted. Maya didn't know what to do with a four-week trend on a manager she'd been reading for eight months, and not knowing what to do with a data point was a rarer condition for her than most people understood. She watched Petra for three seconds, then made herself look back at the screen.

Petra had hired her. Petra had told her, during the interview, that Lumen's emotional response model was the most interesting technical challenge in consumer AI and that the team needed someone who could read embedding spaces the way musicians read chord progressions. Maya had liked her immediately. She'd liked being seen. She had especially liked being seen for the thing she was best at, which wasn't a thing most people had ever noticed about her, and which the rooms she had walked into in this industry, almost without exception, had not been built to notice in a Black woman.

She turned back to her screen. The pattern was still there. It would be there tomorrow and the day after that. Patterns embedded in training data don't go away. They become part of what the model learns, and then part of what the model says, and then part of what twelve million people hear, every day, when they talk to the thing they trust most with the things they can't say out loud.

On the floor near her desk, someone had dropped a pen.

Maya copied the coordinates of the anomaly onto a Post-it by hand. Which wasn't a thing she did. Which was a thing her mother used to do. The two copies weren't redundant. They were each other's insurance against the day one of them turned out to be wrong, which her mother had taught her could happen to either kind of record, in different ways, for different reasons. Then she placed the coordinates in the part of her head she held for unclassified reads: things that had cleared one threshold but not the next, waiting for more data. She'd been maintaining that column since she was three. By now it was densely populated and the coordinates went in without her thinking, the way a confirmed anomaly waits in a queue for the test that will either promote it or kill it. She stuck the Post-it to the edge of her monitor, behind the notch where the laminate had chipped last month, where Phoebe wouldn't see it by accident if Phoebe leaned over to ask about lunch.

She didn't tell Petra. She didn't tell Phoebe. She didn't text Riya, her roommate from sophomore year at Stanford and now a computational neuroscience postdoc at Berkeley and her closest friend in the world, who was the only person in Maya's life who would have understood what she'd found inside forty seconds, which was exactly why Maya couldn't text her. She told no one.

An instinct. One she had been running on since before she had words for it. She didn't know yet if the odds on it were good. She never did, at this stage.

She watered Gerald. The water darkened the soil from the surface down, unhurried, and the leaves began to uncurl in their own time. She turned her phone over, read her mother's text again, didn't answer it. She pulled up the training queue and saw that Batch 7,412 had already begun training on the model. She'd marked it as reviewed at 9:16. The pipeline doesn't wait for second thoughts.

She could halt the run. She had the permissions. She could mark the batch as pulled-for-review and freeze it mid-training, and nobody in the company would notice for hours. Nobody in the company except the one person who was paying close attention to exactly this batch. Who would notice immediately. The probability that such a person existed was, at this point, better than even. Finding out who that person was required her not halting the run.

It was eleven-eighteen.

And halting wouldn't have un-trained the model. Whatever was in the batch had been accumulating in the training data for longer than this batch had existed. The model's weights were already carrying the pattern she had just found. Pulling one batch wouldn't remove one byte of the contamination. It would only announce to someone she couldn't yet name that Maya had noticed. The choice between halting and continuing wasn't the choice she had first thought it was. The model wasn't safe. It hadn't been safe for a while. The choice was between carrying the knowledge in silence and throwing it into a room where someone she didn't know was standing and would catch it.

She didn't halt the run. The trained model would enter the evaluation pipeline at the end of the day, and evaluation would take another week, and deployment testing would take two more weeks, and gradual rollout would take four more weeks after that. Maya had approximately six weeks before any human being on the other side of the system would hear what this model had been taught. Six weeks was enough time to find whoever had planted this, if she was careful. Six weeks had to be enough time, because the alternative was that it wasn't, and the alternative wasn't a thing she was ready to think about.

She placed the cost of that decision in the unclassified column, next to the other things she was still waiting on. She began running the next diagnostic pass.

At the far end of the floor, Petra was looking at her screen and not writing with her pen. She had been looking at it for longer than Maya had been watching her, and she had been looking at nothing in particular.

The room never felt cold when Petra held the pen. It did today. The degree and a half of drop wasn't enough to comment on, and she didn't. She went back to the coordinates on the Post-it and read them twice more, which was how she memorized things, and which was also how she kept herself from turning her head and looking at Petra's office one more time.

Across the Bay, three thousand miles east, the Vantage Foundation occupied the top three floors of a building that didn't announce its name. A brass plate by the elevator read VANTAGE FOUNDATION and a suite number, and that was all. The Foundation was older than the building. It was older than the country around the building, almost, and Declan's grandfather Everett had founded the technical research program here in the early sixties. Three generations of Marshes had worked in the rooms upstairs since. Declan had never entered the lobby without thinking about his grandfather, which was a feeling he had long ago stopped trying to describe to himself.

Declan had worked in this building for eleven years. He could find the supply closet in the dark. From the quality of the silence in the hallway he could tell whether Cassandra Vale was in the building or traveling, and today she was traveling, which was how the building breathed when she was gone. The air was different when she was there. Declan had never been able to put it in words he was willing to say out loud. Not even to Theo.

He was at his desk on the 34th floor, working through the Hernandez Education Initiative grant. A real project, real literacy programs funded in four states. Declan was genuinely proud of it in the way he was proud of things that let him go home to Theo without feeling like a liar. This was the thing about the Vantage Foundation he had long ago stopped trying to explain, because there was no sentence that did it justice without sounding either naive or sinister. The hospital wings existed. The scholarships existed. The Hernandez grant would put books in front of real children. Whatever else the Foundation was, the good work wasn't a disguise.

The cover was its genuineness.

He'd never let himself say that sentence out loud, anywhere, to anyone. He had formulated it in 2014 at a donor event in Zurich and had filed it the way his grandfather had taught him to file uncomfortable knowledge: in a drawer inside his head, labeled but not reopened. He'd reopened it at intervals since. He had been careful not to count the intervals, because counting would have made a pattern of them, and Declan had the professional's ear for when a pattern of his own behavior was about to become something he would have to answer for.

His secondary monitor showed his inbox. He wasn't looking at it. He was looking at the Hernandez budget and feeling, mildly, the satisfaction of a spreadsheet that balanced. His peripheral vision caught the subject line of a new message on the other screen, and the subject line made him stop looking at the Hernandez budget.

The message was from an internal distribution list used for grant-administration reporting, the kind of list Declan himself had set up six years ago and that he still maintained. Routine traffic. Forty messages a day, most of them forwarded quarterly reports and operational status updates from the subsidiaries. The document attached to this one shouldn't have been on the list. He knew it shouldn't have been because the document was marked with a classification code he had seen exactly twice before in eleven years, both times on documents that had been hand-delivered to Cassandra's desk, and that he had been asked, politely, not to discuss.

His first instinct was to close the message without opening the attachment and report the misrouting through the appropriate channel. His second instinct, which arrived a half-second behind the first and overrode it without argument, was that reporting a document he hadn't read was a thing only an amateur would do. He needed to know what he was routing back to source before he routed it back. That wasn't curiosity. That was professionalism.

That was what he told himself.

He opened it.

Four pages. The first two described, in the clean institutional prose the Foundation used for everything, a research program he had never seen referenced in any filing system, meeting minutes, or grant database. The program was called PROJECT ANTIPHON and it was administered through a subsidiary he didn't recognize. The third page was a timeline. Milestones, deliverables, projected outcomes. The language was general enough to be opaque and specific enough to be alarming. One phrase stopped him reading.

Post-deployment behavioral modification at population scale.

Eight words.

He read them again. He read the paragraph around them, which explained nothing, because the paragraph was written in the style of institutional prose that communicates importance without communicating content. A style Declan had been reading for eleven years and that he recognized, with a new and unpleasant precision, as the style of people who know exactly what they are saying and have chosen not to say it clearly.

The air in the office didn't change. The HVAC cycled on the way it always cycled on. The fluorescent lighting continued to hum at the cadence it had been humming at all morning. A coworker two offices down laughed at something on a call. Nothing in the world adjusted itself to accommodate the sentence that had just entered Declan's head. It felt like a fact that had always been true and that he was only now being asked to acknowledge.

He turned to the fourth page. A contact list. Three names, with no photographs and no bios and no institutional affiliations Declan recognized. He read each of them twice. He didn't recognize any of them. And not recognizing them was its own data point, because Declan knew the name of every senior researcher at every subsidiary the Vantage Foundation had ever funded. Knowing who was in the room had been a family skill for three generations. Three names he'd never seen before meant a program his family had decided to wall him off from, and his family hadn't walled him off from anything in eleven years of his employment, which meant something had changed, and he didn't know when, and he didn't know what.

He closed the document.

He forwarded it back to the original distribution list with a brief note: Received this in error. Doesn't appear to be for my team. Routing back to source. Then he took out his phone, turned off the screen lock, and photographed every page. He knew exactly what photographing a classified document inside the Foundation's offices was. He had been raised by a family that taught its children why you did not photograph documents inside the Foundation's offices. He photographed it anyway. The photographing was the first decision in his life that didn't flow naturally from his upbringing, and the strangeness of the decision was the part of it that told him the document was real. When he was done he put the phone in his jacket pocket and looked at the Hernandez budget on his primary screen. The numbers hadn't changed. The literacy programs would still teach children to read. The world in which those programs existed and the world in which the document on his other screen existed were the same world. They had been the same world for eleven years, and he had spent eleven years not looking at the seam between them. The seam was eight words long. The seam was now in his inbox.

He filed the Hernandez grant.

He went to the men's room at the end of the hall and stood at the sink and looked at his face in the mirror for two minutes. He didn't know what he was looking for. He found it anyway. Then he went back to his desk, and he did his job for the rest of the day, the way he had done his job for eleven years. Carefully and competently and with the quality of attention that his family had bred into three generations of Marshes, and that he'd always believed was professionalism, and that he was beginning, today, to suspect was something else.

He went home at six-thirty. Theo had made pasta. They ate at the kitchen table. Theo talked about a roof garden he was designing for a restaurant in the Mission, something with native grasses and a specific kind of drought-tolerant sedum whose name Declan didn't catch because he was watching Theo's hands and thinking about how to ask him about his day in a way that didn't require Theo to know what Declan's day had been. Declan asked good questions. He was always asking good questions. It was the thing he was best at. He had learned it from his grandfather, and from the Sill.

Theo went to bed at eleven. Declan stayed in the kitchen and washed the dishes and dried them and put them away and then stood at the counter, hands on the edge of it, listening to the building. Water in someone else's pipes. A faucet two floors up. The elevator shaft. The specific quiet of an apartment building with most of its lights off.

At one-fifty in the morning he opened his phone and looked at the photographs. Eight words.

Post-deployment behavioral modification at population scale.

He read them the way he'd read them in the office, which was carefully. Then he read them again, which was how his grandfather would have read them. Then he read them a third time, which was how he had never read anything in his life.

He'd always known there were rooms in the Foundation he was not meant to look at directly. He had been invited into the tier above his at twenty-two, and he'd said no without asking exactly what he was saying no to. He had been invited again at twenty-eight and said no a second time. Both refusals had been accepted without visible consequence, which was the whole point. Not knowing was the arrangement. Staying had been the compromise, and compromise was how adults lived in the world.

It was a word the Foundation had been using on him for eleven years. He had only ever heard it as his own.

He turned off the phone.

Outside the kitchen window, a garbage truck was working its way down the block, stopping and starting in the rhythm it always kept, the metal arm lifting and setting down, the engine idling and accelerating. Declan listened to it until it was gone. Then he stood at the counter and listened to the space where it had been.

[End of Chapter 1]