Caliban

“Aside from judgment, there is the problem of interpretation. Imagine a situation where a criminal is firing a blaster at a police officer. It is a given that the police officer should defend him- or herself, even to the use of deadly force. Society entitles-even expects-the police officer to subdue or even kill his attacker, because society values its own protection, and the officer’s life, over the criminal’s life. Now imagine that the officer is accompanied by a robot. The robot will of course attempt to shield the policeman from the criminal-but will likewise attempt to protect the criminal from the policeman. It will almost certainly attempt to prevent the police officer from firing back at the criminal. The robot will attempt to prevent harm to either human. The robot might step into the police officer’s field of fire, or let the criminal escape, or attempt to disarmboth combatants. It might attempt to shield each from the other’s fire, even if that results in its own destruction and the immediate resumption of the gun battle.

“Indeed, we have run any number of simulations of such encounters. Without the robot present, the police officer can most often defeat the criminal. With a robot along, here are the outcomes more likely than the police winning: death of police officer and criminal with destruction of robot; death of police officer and with destruction of robot; destruction of robot coupled with escape of criminal; death of criminal and/or police officer with robot surviving just long enough to malfunction due to massive First Law/First Law and First Law/Second Law conflicts. In short, inject a robot into such a situation, and the odds are superb you will end up with a disaster.

“Theoretically itis possible for a robot to judge the situation properly, and not mindlock over being complicit in the death of the criminal. It must be able to decide that both the immediate and long-term general good are served if the police officer wins, and that coming to the assistance or defense of a criminal prepared to take the life of a peace officer is ultimately self-defeating, because the offender will almost certainly attack society again in other ways, if he or she is permitted to survive. However, in practice, all but the most sophisticated robots, with the most finely tuned and balanced potentials of First Law, will have no hope at all of dealing appropriately with such a situation.

“All the laws and rules we live by are subject to such intricacies of interpretation. It is just that we humans are so skilled, so practiced, in threading our ways through these intricacies that we are unaware of them. The proper way to enter a room when a party is in progress in midafternoon, the correct mode of address to the remarried widow of one’s grandfather, the circumstances under which one mayor may not cite a source in scholarly research-we all know such things so well we are not even aware that we know them. Nor is such practiced knowledge limited to such trivial issues.

“For example, it is a universal of human law that murder is a crime. Yet self -defense is in all places a legitimate defense against the accusation of murder, negating the crime and condoning the act. Diminished capacity, insanity defenses, mitigating circumstances, the gradations of the crime of murder from manslaughter to premeditated murder-all these are so many shades of grey drawn on the black and white of the law against murder. As we have seen with my example of the policeman and the criminal, no such gradations appear in the rigidity of the First Law. There is no room for judgment, no way to account for circumstances or allow for flexibility. The closest substitute for flexibility a robot may have is an adjustment in the potential between the First, Second, and Third Laws, and even this is only possible over a limited range.

“What are the Three Laws for? To answer my own question, then, the Three Laws are intended to provide a workable simulation of an idealized moral code, modified to ensure the docility and subservience of robots. The Three Laws werenot written with the intention of modifying human behavior. But they have done just that, rather drastically.

“Having touched on the intent of the Laws, let us now look at their history.

“We all know the Three Laws by heart. We accept them the way we accept gravity, or thunderstorms, or the light of the stars. We see the Three Laws as a force of nature, beyond our control, immutable. We think it is pointless to do anything but accept them, deal with the world that includes them.

“But this is not our only choice. I say again, the Three Laws are a human invention. They are based in human thought and human experience, grounded in the human past. The Laws are, in theory at least, no less susceptible to examination and no more immutable in form than any other human invention-the wheel, the spaceship, the computer. All of these have been changed-or supplanted-by new acts of creativity, new inventions.