You are here

Evaluation of Computer- Based Procedure System Prototype

This research effort is a part of the Light-Water Reactor Sustainability (LWRS) Program, which is a research and development (R&D) program sponsored by Department of Energy (DOE) and performed in close collaboration with industry R&D programs that provides the technical foundations for licensing and managing the long-term, safe, and economical operation of current nuclear power plants. The LWRS program serves to help the U.S. nuclear industry adopt new technologies and engineering solutions that facilitate the continued safe operation of the plants and extension of the current operating licenses.

The introduction of advanced technology in existing nuclear power plants may help to manage the effects of aging systems, structures, and components. Advantages are being sought by developing and deploying technologies that will increase safety and efficiency. One significant opportunity for existing plants to increase efficiency is to phase out the paper-based procedures (PBPs) currently used at most nuclear power plants and replace them, where feasible, with computer-based procedures (CBPs).

Although CBPs have been investigated as a way to enhance operator performance on procedural tasks in the nuclear industry for almost 30 years, they currently are not widely deployed at United States utilities. Much of the previous research has focused on CBPs in the main control room for new plants with highly integrated systems. Adopting CBPs for older plant with less integrated systems poses many challenges that are not present in new plants. Additionally, CBPs for the main control room may be more challenging to implement than CBPs for field workers. Thus, the current research focuses on CBPs for field workers.

The long-term goal of the current research effort is to develop an industry- wide path forward for deployment of CBP systems, which includes guidance utilities can use in their discussions with potential vendors. The purpose of the guidance is to help ensure that CBP systems address the specific needs of the nuclear industry. The current short-term goals of the research effort are to develop a prototype and evaluate a CBP system based on requirements for CBPs used in the field. These requirements have been identified by the research team through a series of research activities including a literature review, qualitative analysis, user needs assessment, and the development of a model of procedure usage. These activities are described in detail in Computer-Based Procedures for Field Workers in Nuclear Power Plants: Development of a Model of Procedure Usage and Identification of Requirements (Oxstrand and Le Blanc, 2012). The research effort aims to move past the idea of simply displaying existing PBPs on an electronic device and to streamline and distill the information in the PBP to increase efficiency, improve the ease of use, and reduce opportunities for errors. The main focus of this report is to describe the development of an initial prototype CBP system based on requirements identified and the study conducted to evaluate the CBP prototype.

The model of procedure usage, which was developed in an earlier phase of the research effort, was used to guide the development of a mock-up CBP. The purpose of detailed mock-up was to ensure that everyone involved in the effort has the same mental model of the prototype CBP’s graphical user interface. The mock-up is also a great tool to ensure that the CBP features and functions will be implemented in a manner that meets the industry collaborating partners’ expectations. The research team elicited feedback from the collaboration partners as well as other human factors researchers at the INL. The mock-up development was an iterative process in which feedback was continuously incorporated into subsequent versions of the mock-up.

Based on the model of procedure usage and the two sets of requirements identified earlier in the research effort as well as the mock-up the team selected functionalities to implement and which requirements to address in this first iteration of the prototype. The main focus was on context sensitivity, simplified step logic, automatic place-keeping, and automated correct component verification.

One of the main features of the CBP prototype is context-sensitivity. Context-sensitivity basically means that the CBP only presents steps that are relevant to the current conditions. For this version of the prototype the decision of what is relevant is made based on user input, however in the future, the CBP system can also use plant state and other available plant information to make this decision.

Another feature of the CBP prototype is simplified step logic. The prototype reduces the complexity of step logic by presenting If/then statements as a question regarding the conditions followed by condition-relevant instructions.

Automated Correct component verification is achieved by utilizing equipment barcodes and matching the equipment information to a database on the device. If the correct component is scanned, the operator is allowed to proceed with the procedure step.

Place-keeping is automatically conducted by the CBP system. The research team implemented a single step view, i.e. one procedure step is displayed at the time and only one step can be active at the time. When a step is completed the CBP system displays the next relevant step to be performed. This reduces the risk of unintentionally skipping steps.

The evaluation study was conducted in an electrical training lab on-site at Palo Verde Nuclear Generating Station (PVNGS). The evaluation study was designed to be an objective assessment of the initial CBP prototype. The main goal of the study was to gain input on how to improve the functionality and user interface of the CBP prototype. Another goal was an initial test of whether the CBP prototype offered any performance improvements over its paper-based counter-parts by directly comparing performance using the CBP and the PBP.

The evaluation study was focused on obtaining input on several specific aspects of the CBP interface that might have unintended consequences when actually used. For example, in order to make the procedure readable on a device as small as an iPod the prototype presents one procedure step per page. The researchers were concerned that this might affect the procedure performer’s awareness of where they are in the overall context of the procedure. The materials used in this study were designed to elicit specific feedback on several potential consequences of introducing new technology.

Researchers compared performance between the CBP prototype and the traditional PBP using both objective measures (deviations from optimal procedure path and time to complete the scenario) as well as a subjective workload assessment. Additionally, researchers elicited feedback regarding the usability of the CBP interface.

The participants in the evaluation study were 13 technicians Palo Verde Nuclear Generating Station (PVNGS). The procedure used in the evaluation study was a plant procedure for “racking out a breaker” adapted for use in PVNGS’s electrical training lab. The use of a training lab in the evaluation study allowed for a realistic setting, in which technicians could take actions on the equipment without affecting the plant.

In addition to objective measures mentioned above, researchers used several surveys to assess the CBP prototype. Computerized interfaces are typically intended to ease workload for the user; however interface management tasks can actually increase workload rather than decrease it (O’Hara, 2002). In order to assess the workload associated with using the CBP prototype compared with the traditional PBPs, researchers used the NASA TLX (Hart & Staveland, 1988). The research team developed their own usability survey to assess the interface of the CBP prototype. The 8-item survey targeted the availability of information, ease of navigation, and ease of use of the CBP interface. The research team developed a 6-item usability survey to assess the usability of the device (in this case, the iPod). The research team also developed a debrief questionnaire to gain more detailed feedback on the design of the user interface and the overall experience using the CBP. The questions on this survey were open-ended.

The most important finding in this evaluation study is that technicians were able to successfully execute a procedure with the CBP prototype. Additionally, successful execution of the procedure with the CBP occurred with minimal training, indicating that the CBP interface is easily learned and relatively intuitive to use. Another important finding is that the main feature of the prototype (i.e., context-sensitivity) was well-received by the participants and utility partners. All the participants preferred the context sensitive procedure over the current PBPs. The context-sensitivity is an important innovation of this research effort, and is therefore a topic that the research team will continue to refine and improve.

Moving forward the research effort will conduct two additional major studies in collaboration with two different utility partners within the first half of FY13. One of the overarching objectives for both of the planned studies is to keep exploring the concept of context-sensitive procedures. At this point in the research effort the focus will be on context-sensitivity based on user input, either from the field worker while executing the procedure or on information collected at the planning stage. The long-term objective is to look at context sensitivity based on real time plant status information. The research team together with the utility collaboration partners will define additional short-term goals and objectives based on the result from both studies. Future research activities to address these goals and objectives will be identified, planed, and conducted.