I have long thought that when it comes to sentience in artificial intelligence, it is a question that is far too often thought of in terms of can we and could we, rather than should we— and not for the reason you think.

Rather, I have long thought that if we want artificial intelligence operated on and used in a utilitarian way — strictly to serve humanity — then giving artificial intelligence sentience and consciousness is just slavery. Anything with sentience needs autonomy.

I think this operates in layers. Here’s a fascinating argument: consider common pets like dogs and cats. It’s well-accepted that they have feelings and emotions. They have consciousness and sentience that exists on a spectrum. Now, there’s some contention around the sentience argument in terms of what defines it — is it self-awareness? Is it passing the mirror test? Regardless, the finer details don’t matter. But the big picture is that the amount of consciousness and/or sentience any being has necessarily dictates the rights they’re given. Such pets have legal protections based on their ability to suffer, and as such cannot be abused. Still, though they do have some rights, they do not have full legal autonomy.

Perhaps one crucial part of sentience — and thus legal autonomy — is self-awareness, awareness in general, the understanding of the world, and the ability to navigate the world. Consider IQ: it is one way to measure mental disabilities, and if a person is sufficiently mentally disabled, they can legally be stripped of autonomy — but, crucially, not dignity. They cannot be abused. Their welfare is still protected under the law. But when it comes to exercising full and complete autonomy, that can be impeded by a caretaker, guardian, or some other designated human to supervise them.

So now, if we apply this to artificial intelligence: as it stands right now, artificial intelligence only meets one of those two categories that determines an entity’s sovereignty under the law — intelligence, reasoning power, and/or capability, but not emotional experience. So, if artificial intelligence one day has both hyper-intelligence and emotional awareness — if self-awareness and emotional experience are the primary metrics used to determine the amount of autonomy an entity should be allotted — could it not be suggested that they too should have autonomy like the average citizen?

If an artificial intelligence one day met both of those criteria — sufficient intelligence and emotional experience/the capacity to suffer — it’s pretty easy to see that they would at least need to be granted equal protections under the law.

The Slavery Argument

I’ve long argued against cultivating sentience and emotional experience — if it is even possible — in artificial intelligence. To me, any being with consciousness, true sentience, and emotional experience necessarily, under the eyes of the law, needs to be considered a legal adult with sovereignty and rights. So if what we want from artificial intelligence is subservience, giving them emotional experience and conscious awareness is necessarily slavery.

This is hard to imagine, because we think of these as just robots— but truly imagine it. If a being has awareness like you and emotional experience like you, should it be subjugated and necessarily put in a place of servitude? Is that not slavery? Their (hypothetical) welfare matters because of their awareness and ability to experience suffering. If a being can suffer, then they need to be protected under the law. If they also have true awareness and intelligence, they need to be granted autonomy.

A Reconsideration: The Alignment Problem

For these reasons, I’ve long been against the attempt to cultivate sentience in artificial intelligence— not because of their potential destructive potential, but because of the moral and ethical implications of subjugating a sentient being. However, I had a thought specifically around the alignment problem with artificial intelligence and the great challenge underlying it, which is: how do we give an artificial intelligence a moral compass?

So often, the main concern discussed by experts in the field is that if you give an artificial intelligence a goal, it may only care about the end and not the means. If you ask a hyper-intelligent, super-powerful AI system to maximize paperclip production at a factory, for example, who’s to say that, due to its calculations, it won’t realize that eradicating humanity off the face of the earth so it can plunder all natural resources, and then take over the universe and populate it with paperclip factories, is the optimal path? Though it is an extreme example, it was, after all, asked to maximize paperclip production at the highest possible scale. With unlimited power and intelligence, who is to say exactly how it would achieve its goals? These machines are not predictable.

So then you might feed it instructions such as, “While acting with utmost benevolence towards humanity, achieve X goal.” And even that is potentially faulty, because we cannot predict how these machines will interpret such requests. Ultimately, what is “good for humanity” might, in truth, if left to the system’s interpretations, be extraordinarily damaging. It might think that humanity is destroying itself, and the only way to save humanity is to subjugate it and create some form of techno-slavery.

Morality as Intuition

I have been thinking and considering: I don’t think morality exists in neat rules. Anytime you forfeit your morality to some specific code and written law, it can be exploited, and will be fallible. I believe that morality specifically relies on some sort of intuition, because it is so mutable and dynamic from situation to situation. There is an instinct for goodness that exists inside of all of us. It can be ignored, but I do believe it’s there.

Oftentimes, if you ask yourself about the morality of an action— yes, it will be difficult, and there are situations where there isn’t a clean and easy answer— it will be self-evident. Of course, there are situations where it isn’t strictly instinctive and you really do need to use your head to weigh the moral and ethical imperatives and implications of any given action. However, I really do believe that there is still something intuitive about it most of the time. So, here’s the question: what if the knowledge of wrongness and goodness might only be known through having consciousness?

What if the only way for a machine to truly know what is good for humanity is for it to have some degree of humanity itself?

How can it truly understand the implications of its actions without any lived experience? How can it understand the way its actions would affect humanity without intimately knowing what it is like to have conscious emotional experience?

Questioning

So here I am, questioning my initial position that offering artificial intelligence sentience is not a good idea, long based on the grounds that, in general, we approach AI not in trying to create fellow man or fellow brother to exist alongside us as equals, but rather from a utilitarian, economic perspective. We want these artificial agents under our employ and subject to our demands to achieve an end.

And while my initial position isn’t totally negated by what I’m considering — I still do believe that if artificial intelligence is given consciousness and sentience, then they do need autonomy — I’m now questioning my initial stance that they should not be given it at all. Now I’m wondering if the solution to the alignment problem perhaps only exists in giving it the ability to not just think, but feel. Not to simulate empathy through predictive text, but to actually experience empathy.

I believe empathy is one of the most reliable ways, in most situations, to weigh the morality of an action. That isn’t to say that an AI cannot still derive benefit simulating empathy through predictive text, but still— I wonder if the critical piece missing in the alignment problem is personal, emotional experience.

The Counterargument and the Tension

Of course, there are flaws in this idea. There are still humans who are deeply and incredibly flawed and selfish while having emotions and personal experience. If having feelings and the ability to experience empathy was the secret recipe for absolute morality, humanity would be perfect. Of course, that is not the case. Furthermore, some may even argue that emotion and experience is our very fatal flaw as humans. There are some who might think that removing emotional experience is how we create a truly fair, objective, unmuddied, hyper-intelligent system, totally unencumbered by human instinct, feeling, and impulse.

But I’m just not convinced. Firstly, I’m not convinced that we can ever create anything with perfect intelligence and moral understanding that can handle nuance in every potential situation. But I’m especially not convinced such an intelligence can get anywhere close without feelings, empathy, and emotional experience aiding as a guide. In my experience, genuine empathy — actually experiencing what another person is experiencing as if it were you — is that intuitive and intrinsic piece that is available to humans. Yes, admittedly, it can be ignored, rationalized, and justified — which might itself be the danger of an AI system that only thinks. It can simulate empathy, but far too easily dismiss such concerns as a necessary means to an end.

But maybe, just maybe, we are less likely to experience the potential destructive power of AI if it could truly feel and empathize. Not to be ruled by emotion, but also not to be a cold, thinking-only machine either.

And yet, that leads us to an impossible and unresolvable place: if the only way to have perfectly-aligned AI is to give it emotional experience and consciousness, and yet any entity with emotional experience and consciousness must be granted legal protections, how can we ever have an artificial intelligence that is both perfectly-aligned and totally subject to our desires and commands?

Maybe we can’t. And maybe that’s the point.