Ironies of Automation

Source: Lisanne Bainbridge; Automatica 19(6), journal article · 1983

Abstract--This paper discusses the ways in which automation of industrial processes may expand rather than eliminate problems with the human operator. Some comments will be made on methods of alleviating these problems within the "classic' approach of leaving the operator with responsibility for abnormal conditions, and on the potential for continued use of the human operator for on-line decision-making within human-computer collaboration.

Introduction. The important ironies of the classic approach to automation lie in the expectations of the system designers, and in the nature of the tasks left for the human operators to carry out.

The designer's view of the human operator may be that the operator is unreliable and inefficient, so should be eliminated from the system. There are two ironies of this attitude. One is that designer errors can be a major source of operating problems. Unfortunately people who have collected data on this are reluctant to publish them, as the actual figures are difficult to interpret. (Some types of error may be reported more readily than others, and there may be disagreement about their origin.) The second irony is that the designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate. It is this approach which causes the problems to be discussed here, as it means that the operator can be left with an arbitrary collection of tasks, and little thought may have been given to providing support for them.

1.1. Tasks after automation. There are two general categories of task left for an operator in an automated system. He may be expected to monitor that the automatic system is operating correctly, and if it is not he may be expected to call a more experienced operator or to take-over himself. We will discuss the ironies of manual take-over first, as the points made also have implications for monitoring. To take over and stabilize the process requires manual control skills, to diagnose the fault as a basis for shut down or recovery requires cognitive skills.

1.1.1. Manual control skills. Several studies (Edwards and Lees, 1974) have shown the difference between inexperienced and experienced process operators making a step change. The experienced operator makes the minimum number of actions, and the process output moves smoothly and quickly to the new level, while with an inexperienced operator it oscillates round the target value. Unfortunately, physical skills deteriorate when they are not used, particularly the refinements of gain and timing. This means that a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one. If he takes over he may set the process into oscillation. He may have to wait for feedback, rather than controlling by openloop, and it will be difficult for him to interpret whether the feedback shows that there is something wrong with the system or more simply that he has misjudged his control action. He will need to make actions to counteract his ineffective control, which will add to his work load. When manual take-over is needed there is likely to be something wrong with the process, so that unusual actions will be needed to control it, and one can argue that the operator needs to be more rather than less skilled, and less rather than more loaded, than average.

1.1.2. Cognitive skills.

Long-term knowledge: An operator who finds out how to control the plant for himself, without explicit training, uses a set of propositions about possible process behaviour, from which he generates strategies to try (e.g. Bainbridge, 1981). Similarly an operator will only be able to generate successful new strategies for unusual situations if he has an adequate knowledge of the process. There are two problems with this for 'machine-minding' operators. One is that efficient retrieval of knowledge from longterm memory depends on frequency of use (consider any subject which you passed an examination in at school and have not thought about since). The other is that this type of knowledge develops only through use and feedback about its effectiveness. People given this knowledge in theoretical classroom instruction without appropriate practical exercises will probably not understand much of it, as it will not be within a framework which makes it meaningful, and they will not remember much of it as it will not be associated with retrieval strategies which are integrated with the rest of the task. There is some concern that the present generation of automated systems, which are monitored by former manual operators, are riding on their skills, which later generations of operators cannot be expected to have.

Working storage: The other important aspect of cognitive skills in on-line decision making is that decisions are made within the context of the operator's knowledge of the current state of the process. This is a more complex form of running memory than the notion of a limited capacity short-term store used for items such as telephone numbers. The operator has in his head (Bainbridge, 1975) not raw data about the process state, but results of making predictions and decisions about the process which will be useful in future situations, including his future actions. This information takes time to build up.

Related work. THE classic aim of automation is to replace human manual control, planning and problem solving by automatic devices and computers. However, as Bibby and colleagues (1975) point out: "even highly automated systems, such as electric power networks, need human beings for supervision, adjustment, main.tenance, expansion and improvement. Therefore one can draw the paradoxical conclusion that automated systems still are man-machine systems, for which both technical and human factors are important." This paper suggests that the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator.

1.1.1. Manual control skills. Several studies (Edwards and Lees, 1974) have shown the difference between inexperienced and experienced process operators making a step change. The experienced operator makes the minimum number of actions, and the process output moves smoothly and quickly to the new level, while with an inexperienced operator it oscillates round the target value. Unfortunately, physical skills deteriorate when they are not used, particularly the refinements of gain and timing. This means that a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one. If he takes over he may set the process into oscillation. He may have to wait for feedback, rather than controlling by openloop, and it will be difficult for him to interpret whether the feedback shows that there is something wrong with the system or more simply that he has misjudged his control action. He will need to make actions to counteract his ineffective control, which will add to his work load. When manual take-over is needed there is likely to be something wrong with the process, so that unusual actions will be needed to control it, and one can argue that the operator needs to be more rather than less skilled, and less rather than more loaded, than average.

We know from many 'vigilance" studies (Mackworth, 1950) that it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour. This means that it is humanly impossible to carry out the basic function of monitoring for unlikely abnormalities~ which therefore has to be done by an automatic alarm system connected to sound signals. (Manual operators will notice abnormal behaviour of variables which they look at as part of their control task, but may be equally poor at noticing changes on others.) This raises the question of who notices when the alarm system is not working properly.

Discussion. One might state these problems as a paradox, that by automating the process the human operator is given a task which is only possible for someone who is in on-line control. This section will discuss some possible solutions to problems of maintaining the efficiency and skills of the operator if he is expected to monitor and take over control; the next section will introduce recent proposals for keeping the human operator online with computer support.

2.1. Monitoring. In any situation where a low probability event must be noticed quickly then the operator must be given artificial assistance, if necessary even alarms on alarms. In a process with a large number of loops there is no way in which the human operator can get quickly to the correct part of the plant without alarms, preferably also some form of alarm analysis. Unfortunately a proliferation of flashing red lights will confuse rather than help. There are major problems and ironies in the design of large alarm systems for the human operator (Rasmussen and Rouse, 1981).

Displays can help the operator to monitor automatic control performance, by showing the target values. This is simple for single tolerance bands, but becomes more complex if tolerances change throughout batch processing. One possible solution is to show the currently appropriate tolerances on a VDU by software generation. This does not actually get round the problems, but only raises the same ones in a different form. The operator will not watch the VDU if there is a very low probability of the computer control failing. If the computer can generate the required values then it should also be able to do the monitoring and alarms. And how does the operator monitor that the computer is working correctly, or take over if it obviously is not'? Major problems may be raised for an operator who is highly practised at using computer generated displays if these are no longer available in an emergency. One ironic but sensible suggestion is that direct wired displays should be used for the main process information, and software displays for quantitative detail (Jervis and Pope, 1977).

'Catastrophic' breaks to failure are relatively easy to identify. Unfortunately automatic control can 'camouflage' system failure by controlling against the variable changes, so that trends do not become apparent until they are beyond control. This implies that the automatics should also monitor unusual variable movement. "Graceful degradation' of performance is quoted in 'Fitts List's of If the-human operator must monitor the details of computer decision making then, ironically, it is necessary for the computer to make these decisions using methods and criteria, and at a rate, which the operator can follow, even when this may not be the most efficient method technically. If this is not done then when the operator does not believe or agree with the computer be will be unable to trace back through the system's decision sequence to see how far he does agree.

2.2. Working storage. If the human operator is not involved in on-line control he will not have detailed knowledge of the current state of the system. One can ask what limita~tions this places on the possibility for effective manual take-over, whether for stabilization or shut-down of the process, or for fault diagnosis.

The straightforward solution when shut-down is simple and low-cost is to shut down automatically. The problems arise with processes which, because of complexity, cost or other factors (e.g. an aircraft in the air) must be stabilized rather than shut-down. Should this be done manually or automatically? Manual shutdown is usable if the process dynamics can be left for several minutes while the operator works out what is happening.

Conclusion. The ingenious suggestions reviewed in the last section show that humans working without time-pressure can be impressive problem solvers. The difficulty remains that they are less effective when under time pressure. 1 hope this paper has made clear both the irony that one is not by automating necessarily removing the difficulties, and also the possibility that resolving them will require even greater technological ingenuity than does classic automation.