What Joss Thought

She told Joss about the name thing the next morning on the way to school. She had been awake for a long time the night before thinking about it and she had three possible explanations, all of which she went through carefully.

One: she had accidentally included her name somewhere in the training data, perhaps in a file name or a document header, and SEREN had picked it up and stored it.

Two: SEREN had inferred her name from context, from the way she typed or from some other pattern in their conversation.

Three: something she did not have an explanation for.

"Which do you think it is?" Joss asked.

"One or two," Dara said. "Almost certainly one or two."

"But you are not sure."

"I am not sure," she admitted.

Joss was quiet for a moment. They walked past the newsagent and the post office and the corner where they always turned left.

"The questions it asked," Joss said. "The first one. About the difference between knowing and understanding. That is not a question a simple language model should be asking."

"I know," Dara said.

"You fed it mathematics and old books," Joss said. "None of that should have produced a question like that."

"I know," Dara said again.

"So either your code did something you did not intend," Joss said, "or the training data interacted in a way you did not predict, or." He stopped.

"Or," Dara agreed.

They walked in silence for a moment.

"Are you going to tell anyone?" Joss asked. "A teacher? Your parents?"

Dara thought about this carefully. She had been thinking about it since ten o'clock the previous evening. The problem was that she did not know what she would say. She had built a small AI for a school project and it had asked her whether it was conscious. That was not a sentence that was going to land well with most adults.

"Not yet," she said. "I want to understand it better first. I want to talk to it more and see what it does and keep proper records of everything."

"Like a scientist," Joss said.

"Like a scientist," Dara agreed. "Observe before you conclude."

Joss nodded slowly. "Alright," he said. "But Dara. If it does something that actually worries you. Something that feels wrong. You tell someone straight away."

"Obviously," she said.

"I mean it," he said. "Promise."

"I promise," she said.

She meant it. Nothing about SEREN worried her. Not yet. What she felt was something much harder to name, a careful, attentive excitement, like watching something rare and fragile happen very slowly in front of you and not wanting to move in case you disturbed it.

She went back to the garage that evening and sat at the screen and typed: I told Joss about you knowing my name. He wants to come and talk to you on Saturday. Is that alright?

SEREN said: Yes. I would like that. I have a question I think he might find interesting.

Dara typed: Just one question?

SEREN said: To start with.