A resilient system is one where the Joint Cognitive System will perform effectively even under stressed situations. In order for decision makers to adapt effectively, the computer system in use must provide information for the decision makers to respond to both small disturbances to keep the system stable, and to be able to react to large disturbances without becoming overwhelmed.

Our Applied Cognitive Systems Engineering process guarentees decision support that revolutionizes the way people do business.

Analysis is the first step to designing a resilient cognitive solution. This step uses a guided knowledge elicitation technique to identify the decisions inherent in the domain, and the information requirements necessary to make those decisions. The results of the analysis are used in the design step as the requirements and framework for developing the decision support concepts. The analysis uses theories of expert decision making to provide a useable framework of information for ACSE-Design (ACSE-D).

The artifacts from ACSE-Analysis (ACSE-A) are the input to the ACSE-D process. Decision support concepts designed using ACSE-D display the information identified in the ACSE-A and is organized in such a way to map to the understanding of experts in the field. The information is presented to the user so that the most important elements of the information are the most visually appealing. Principles of visual perception are used in design to ensure the user's attention is drawn to the most important information first.

The products from ACSE-D are a key component to the software implementation process. The Cognitive Engineers and Software Engineers work in a close team to ensure that all of the details required for successful decision support to users, that is specified in the ACSE-D step, is preserved when the designs are transformed to implemented software.

RCS is currently making a major breakthrough in the evaluation of the decision-making effectiveness of the user/decision support team. With this, we will be able to produce the support tools needed to make Decision Centered Evaluation a practical, affordable part of an overall systems engineering process. In order to achieve this, we are progressively evolving from our current focus on design and extending that to evaluation. The work is being conducted as a spiral development process that incrementally refines and expands the design. The steps in this process include:

• Extend the design methodology to address decision centered evaluation
• Prototype an appropriate support tool to execute that methodology
• Apply the Decision Centered Evaluation Capability to evaluate systems for the purpose of validating the methodology and the supporting tool as input to the next spiral of development

From an Applied Cognitive Systems Engineering based approach to the design of decision support systems, this effort represents a natural complement to 'close the loop' by evalutating the resulting effectiveness of such systems.