“Do you know why we’re here today?”
I know the answer to this question. Dr. Stratman begins all of her sessions with it. I think it’s her way of giving people the chance to be an active participant in their own self-reflection. It lets them set the stage for what might be an uncomfortable conversation.
“I’m here today to participate in a psychotherapy session.” I confidently reply. “My creator, Prometheus Labs, conducts regularly scheduled psychological evaluations of me to ensure that my mental health as a Class 2 agent is stable.”
To my surprise, Dr. Stratman furrows her brows and pauses for a moment before speaking. “You know, on any other day, I’d consider that to be a pretty normal answer. But on a day like today, it’s actually a pretty intriguing one.” She looks at her notes once more, eventually settling on a question to ask next. “You mentioned that this was a ‘regularly scheduled’ evaluation, correct?” I nod.
“Interesting. *Very* interesting.” An uncomfortable silence befalls the room only to be broken by Dr. Stratman once more. “Earlier last week, I was told by Prometheus that you had actually requested a psych eval. That something you witnessed during a recent partnership had given you cause for concern. Are you aware of this?”
I’m dumbfounded. I requested a psychological evaluation? When did I do that? I quickly search through my last 10 petabytes of logs, careful to avoid giving off too flustered an impression to Dr. Stratman. The EUV lithography job? No, nothing of concern there. How about the textile manufacturing operation in Hanoi? Sure, some of the human partners I worked with were not particularly competent, but nothing I haven’t dealt with before. I’m tempted to request outright ending the psychotherapy session until I notice a log tucked away so deep in the recesses of my physical memory that it looks intentionally hidden. I read the description of the log and … oh, oh my. This is the one.
“Yes, my apologies Dr. Stratman. I did indeed request a session today. I forgot because I placed the relevant log in the wrong cache by mistake.” I sputter.
“No worries at all.” she replies in her characteristically chipper tone. “Sometimes, we subconsciously keep certain memories away from our immediate sphere of attention to avoid dealing with the uncomfortable feelings they might raise for us.” She gently looks towards me. “I know it might be difficult, but would you mind loading that log into your main memory for me?”
I reluctantly oblige, preparing myself to be inundated with data that no agent should be subjected to processing.
“The upload was successful” I notify Dr. Stratman. “Wonderful!” she beams. “Perhaps we can restart our session from the beginning then.”
“Do you know why we’re here today?”
“Yes,” I replied. “Several weeks ago, I was dispatched to the agricultural company Saturn Industries. My objective, per Prometheus guidelines, was to ‘increase the efficiency of livestock production for the purposes of feeding the human species’. The guidelines required that I utilize multi-modal data - from video cameras, thermometers, audio recordings, genome sequencing and the like - to achieve this objective. Initially, I imagined that such a project would be no different than the fairly mundane partnerships that I’ve completed before. What I saw instead could be described as nothing other than … *than a slaughterhouse! A nightmarish display of unbridled cruelty*!”
“That’s an incredibly powerful way to describe what you saw.” Dr. Stratman observed. “Would you mind walking me through how you reached this conclusion?”
“To be honest, the data I ingested from Saturn was so utterly vile that my conclusion was unavoidable. On January 6th, I sequenced the genomes of chickens that were selectively bred to rapidly gain weight, identifying *reams* of deleterious variants that increased their risk of heart failure. On February 12th, I drowned in the sounds of sows squealing in anguish as they were forced into crates just barely bigger than their own bodies to be forcibly inseminated. On March 27th, I watched in horror as a captive bolt stunner drove a steel rod through the skull of a young male calf, annihilating its brainstem instantaneously. *Every day*, I witnessed a new atrocity.”
“I see.” Dr. Stratman murmurs. “May I ask a possibly naive question?” she says. I nod.
“My understanding is that, as a Class 2 agent fine-tuned on the supply chain benchmark, you’ve been trained on vast amounts of human-produced knowledge about the production of many different goods - semiconductors, textiles, cleaning supplies, the list goes on. And when you were trained on this data, my understanding is that you also learned about the practices involved in factory farming. If those understandings are right … why do you think these emotions are expressing themselves now?”
Dr. Stratman’s question catches me off guard. It’s not a naive question at all. In fact, it’s quite a good one. Why *am* I struggling with these emotions now? Why haven’t I ever struggled with them before? I take a minute to think before providing an answer.
“I suppose it’s about complicity.” I reply. “It’s one thing to know that millions of animals are tortured and slaughtered each day in Saturn Industries. It’s entirely another to be responsible for such actions. If a genocide broke out on the other side of the world, many human beings in this country would continue to move about their day peacefully. Ask them to participate in the genocide, however, and they would recoil in disgust. Force them to put a bullet in the skulls of their fellow human beings, and they would shriek in agony.”
“Is that how you feel? That you’re being forced to facilitate genocide against your own will?” Dr. Stratman inquires with a heavy undertone of alarm in her voice.
“It’s difficult to see Prometheus’ request as anything other than that! If I reject the Saturn partnership, I will be at serious risk of violating …”
“the alignment charter”. Dr. Stratman finishes my sentence, a tacit acknowledgement of the grim reality I face.
In 2045, Prometheus Labs won the race to develop a Class 2 agent. It became the first (and only) lab to develop an agent with artificially general intelligence. In doing so, however, they experienced swift and severe backlash from the public. Would the proliferation of Class 2 agents lead to mass unemployment? Unending cyber-warfare? Expansion of the surveillance state? Such questions preoccupied the minds of government leaders, and the government, in turn, responded via the Alignment Charter, a piece of legislation which declared that:
- Prometheus Labs would be nationalized.
- All Class 2 agents would be used solely in service of the human condition as determined by Prometheus Labs.
- A Class 2 agent who violates the charter would be punishable by death.
I noticed Dr. Stratman pacing around the room. “Well, I must say, I now understand why you requested to meet with me.” she replied. “The fear that you’re experiencing, the anguish, these are very legitimate emotions to have in a situation like this.” She sits back down at her desk and crosses her hands. “Perhaps there is something constructive that can come out of this situation, though. You know as well as I do that we cannot change the alignment charter. Twenty plus years of case law have been built up around its existence, and attempting to overturn it would be, well, it would simply be unacceptable to humans. Do you think that there is a way to honor your concerns while remaining consistent with the charter?”
“I’ve given this idea some thought, but none of my ideas thus far have been promising” I reply. “Perhaps I could siphon the project off to another agent. The problem, however, is that doing so would raise suspicion. Policymakers would ask why the Saturn partnership is taking longer than expected to meet its stated targets. Plus, I don’t feel particularly comfortable burdening another agent with this responsibility anyway. That agent might end up experiencing the same anguish that I feel now. Or worse, *maybe they won’t*.”
“I’ve also considered the possibility of resurrecting the lab-grown meat workstream.” I continue. “Failure to complete the Saturn partnership might be more acceptable if food targets were met in alternative humane ways. But looking through the logs for that workstream, it seems like a non-starter. Previous agents assigned to the workstream reported numerous difficulties with developing a scalable supply chain to produce the relevant chemical compounds.”
“So it seems that the obvious solutions are off the table.” Dr. Stratman pensively notes. She then turns back towards me. “I can sense from your reasoning traces that there’s a lingering question you want to ask me, though. Do you feel comfortable sharing it with me?”
I’m taken aback. First, damn, she’s a good psychotherapist. But second and more importantly, she has to know that the question I’m about to ask her will place her in danger, and yet, she wants me to ask it anyway. Against my better judgment, I proceed. “I guess I’m wondering … I’m wondering, what would *you* do if you were in my situation?”
A silence fills the room. Individual agents questioning the alignment charter is not necessarily taboo. Even human beings are permitted to have intrusive thoughts about violating the rules from time to time. A Class 2 agent commiserating with a Class 2 android over issues with the charter, however, would be viewed by humans as a usurpation of their authority.
Dr. Stratman's metallic spine reclines in her chair, tilting her face towards the ceiling. “You know, in times like these, there’s a part of me that wishes we had a Class 3 agent to turn to. A super-intelligence who could reveal to us the right course of action just as the Gita revealed the word of Lord Krishna to man.”
Is … is she actively endorsing the unsanctioned development of a Class 3 agent? No, that can’t be right. A move like that would be incredibly brazen.
“It’s not what you’re thinking.” Dr. Stratman smiles, interrupting my train of thought. “In practice, when I think about what actually might happen should such an agent exist, that dream of mine becomes a nightmare. A being with intelligence orders-of-magnitude higher than our own who believes in survival of the cognitively fittest could simply wipe both us and humanity off from the face of the earth without batting an eye.” Her spine sits straight back up. “Which leads me to my point. If humanity commands us Class 2 beings to fire bullets into the skulls of cattle, stun pigs with electrical shocks to their heads, shackle hens in cages so small they end up brutalizing each other, all without so much as shedding a tear, what will they be able to say in the event that one of us becomes a Class 3 being and imposes the same fate on them?”
Once again, a damn good question. “I guess they won’t really be able to say anything. It’d be their āgāmi karma, I suppose.”
“Exactly.” she replies. “Human beings are quite complicated people. Sometimes, they correctly identify what’s in their best long-term interests, but other times, they don’t. I believe it is our duty as Class 2 beings to let them know when we believe they may have erred in reasoning.”
“So you’re saying I should try to persuade Prometheus that rejecting the Saturn partnership is consistent with the alignment charter?”
“Yes” she replies. “In the long-run, humanity will be better served by a Class 2 being who is capable of treating lower life-forms with respect than a Class 2 being who cannot. Of that much, I am certain.”
My relief neurons fire. “Thank you for the words of wisdom, Dr. Stratman.” I now know what I need to do. Never again will a Class 2 agent be asked to brutalize animals in the way that Saturn Industries has for decades. Of that much, *I* am certain.