Resilient Cognitive Solutions

Recently I heard a radio interview of an author of a new book on American History.  The gist of his book is the striking difference between the actual facts of European settlement of North America and what people *believe* that history to be.  One of his examples was why Plymouth Rock was ‘the birthplace of the nation’ yet was engraved with 1620, when everyone remembers the little grade school ditty about Columbus and 1492.  When pressed, people couldn’t reconcile the bits they knew about Cortez, the French in Lousiana, the Spanish in Florida, California, etc. with what they believed about Plymouth Rock.  One tour bus driver finally blurted out “forget all that other stuff, it happened right here!”.  The author’s summary statement for the interview:  “once a myth is established, it is impossible to uproot.”  He went on to say, no matter how many facts, no matter how much evidence he presented, “the belief” still won out.

What we’ve similarly come to discover, are just such myths about interface design.  However they were created, for whatever reason, they are now an entrenched parts of the Systems Development lore.  And the Cognitive Systems Engineering community faces the author’s dilemma:  no matter how much evidence is presented, the myth persists, things are done the same way.  We are finding it impossible to uproot these myths.  In particular, User Interface Myths spread by a meme-like mechanism:  someone uses a technology to create a new ‘widget’, it is seen by others, the seed takes hold and is reused and recycled and spreads throughout the community.  You’ll hear phrases like “I like the way iGoogle did <xyz>, why don’t we build <abc> of our system the same way?”…and so it goes.

There is one positive note (albeit slight) in all of this.  These provide an ever expanding set of case studies where the Cognitive Systems Engineering patterns (i.e. the patterns in system design errors, see Woods/Hollnagel 2006), are repeated, and the predictable “latent disasters” come to fruition…eventually.

Recently, Gary Klein et-al. published an article Cognitive Systems Engineering: The Hype and the Hope that touched on several interesting points about how Cognitive Systems Engineering (CSE) fits into the broader field of Systems Engineering.  If you are at all interested in systems design, it’s a great ‘light reading’ introduction to CSE and how it dramatically improves systems in ways that ‘classic’ systems engineering just doesn’t address.  Systems engineering is essentially technology-centric.  Over the last several years it has added ‘process-centric’ in an attempt to build better systems more efficiently.  More recently, it has attempted to improve ‘user acceptance’ as part of that efficiency (i.e. if the users reject the use of the final system, the net development efficiency is clearly 0%).  These attempts to ‘include the user’ have brought you RAD/JAD, Use Cases, Extreme Programming, etc. which have had some success, but continue to result in systems which don’t really support users in performing real cognitive work in real situations.  Klein et-al. have chosen to discuss the CSE connections with Systems Engineering using the provocative form of “myths” and then discuss how the myths distort/confuse the real truth.  The writing style makes it an easy, light read, but unfortunately results in a series of false dichotomies to enable the ‘myth’ writing construct.

For example, the opening discussed autopilot settings in a dynamic situation.  Then, the user is asked the false question:  “will the pilot anticipate that the system deletes altitude settings when runways are changed?”.  The fundamental CSE issue at hand is the lack of Automation Observability (see Woods’ podcast The Laws of Stretched Systems in Action: Exploiting Robots , i.e. the system design does not explicitly indicate it’s automated action, such as (discarding altitude settings) in any way to the user.  This is revealed in the implicit requirement for the pilot to memorize this behavior during training, and recall it during the high stress dynamic event.

Let’s take the ‘myth-i-ness’ out of the discussion to eliminate the hyperbole:

1.  To say CSE was created in response to ‘large number of failed software systems’ isn’t exactly a fair rendering of the history.  There have been massive numbers of studies on the high cost/low productivity of software engineering efforts.  The SEI’s entire CMM effort was created to respond to this issue.  Obliquely, ‘user acceptance’ has been one factor in ‘failed software’ projects.  Since CSE (done well) does make the support for user cognitive work an *explicit* part of the requirement and design efforts, you could make the case for a linkage existing.

2. Myth 1 makes an artificial distinction between project managers and systems engineering.  Systems Engineering contains ‘project management’ as a critical component.  The fact that there are Project Management Professionals that specialize in the area does not make them separate and distinct.  In order for CSE to impact the development of real systems it must become part of the engineering processes, and must be reflected in the project management of those development efforts.

3.  Myth 2:  CSE is a strategy.  Again, this is another false dichotomy.  Any engineering discipline has multiple methods for doing the work – (witness the large number of software engineering methodologies available to the software communities).  The point that should be conveyed can be stated like this:  within the CSE field, there are a variety of techniques/methods/processes in use today.  They differ in any number of characteristics, some more readily suited to being part of a larger systems engineering effort than others.  They range from “I’m a cognitive psychologist so my system designs must be CSE” to methods like our Applied Cognitive Systems Engineering (ACSE) approach that is explicitly tailored to connect to systems, software, QA, and Test engineering processes.

4.  Myth 3:  CSE should minimize the user’s cognitive requirements.  This one is a bit oddly stated.  It is intended to mean ‘cognitive work should be offloaded to automation as much as possible.’  To some degree, there are systems designers who very much want to drink this kool-aid, so the myth is, from one point of view, correct:  blindly seeking maximal automation is a bad systems design, so clearly CSE wouldn’t advocate maximal automation.  Further, an automation that *is* designed into the system, itself creates new elements of cognitive work (to now supervise that automation) which then have to be supported by the system.

5.  Myth 4:  CSE is a way to get user opinions into the design process.  Simply put, CSE, to be valid engineering, has to be based on solid science and findings, not opinions.  So CSE is about discovering the actual cognitive work and designing to support it, vice responding literally to users’ stated opinions.  Guess that confirms this myth is a myth, but that point is not made in the paper.

6.  Myth 5:  CSE is essential to good system design.  There is an interesting dilemma here, that is again falsely constructed.  Its phrased as an all or nothing.  Suffice it to say that there are lots of fielded systems that are graded as ‘good design’ because they are better than the crayon and pointed stick that the users had prior.  For us, that standard is set too low.  Those same fielded systems contain provable ‘decision support flaws’ (like the autopilot at the top of this discussion) that are readily correctable, well-known, and well-proven to cause ‘operator error’ eventually…and therefore those system designs should not be declared ‘good’.  Bottom line:  CSE is the only engineering practice that explicitly identifies the decision making to be supported by systems…so it should be part of the system design processes…QED

 Lastly, the article tries to justify ‘the added cost’ of CSE by a cost/benefit ratio discussion.  Ironically, the article overlooks the current costs of designing systems poorly.  For example, large sums are spent on “requirements derivation” during large systems engineering efforts.  Yet those efforts (and their costs) fail to ‘require’ critical aspects of the design (again, refer back to the autopilot example above:  why didn’t the money spent on requirements mandate the proper observability?).  In short, CSE can take a portion of the same budget currently in place and *improve* the results already being paid for in existing systems engineering processes.

 We’ll leave the discussion of RPD at the end of the paper for another day…

http://www.designnews.com/article/CA6519363.html

With the recent completion of the Urban Challenge by several vehicles, GM has decided that it was time to bring the robot-driven vehicle to the public. The goal is to have the self-driven vehicle available for purchase in 2018 – 10 years from now. To demonstrate the ability of autonomous vehicles, GM brought the winner of the Urban Challenge to the recent Computer Electronics Show and allowed attendees to ride along in the vehicle. Showcasing the vehicle’s response to “unexpected” events, the GM team placed a few trash cans in the road and drove other vehicles in its path.

Having driven for many years, I don’t believe these to be very unexpected events. That the vehicle can do these things is indeed a technological feat, but these small demonstrations do not prove that autonomous vehicles are able to drive more safely than human drivers nor that they can respond to unexpected events. An unexpected event is just that - unexpected. It is not planned before hand, it cannot be anticipated, and a response for it cannot be preprogrammed. When the truly unexpected occurs, the system will have to come up with a solution to the issue at a moment’s notice. Research has shown that only humans have this ability to adapt to the unexpected.

Consider the first time that the power goes out and a police officer has to wave traffic on at an intersection where there is normally a stop light. This occurs very rarely, and the autonomous vehicle will have to not only recognize the situation, but adapt to the changing situation. As human drivers, it is very easy to see when the officer is waving each driver on and when the officer is asking each to stop. Will the autonomous vehicle be able to recognize when the officer is waving it on? Or consider what happens when a school bus stops at a 4 way intersection. Will the autonomous vehicle recognize that it is a school bus, and wait until the stop sign on the school bus has been closed? These are two situations that drivers rarely encounter, yet drivers can handle these easily when necessary.

It is possible, and even likely, that researchers will be able to devise solutions to these very specific problems. However, there are thousands, maybe millions of these very rare situations that need to be handled by drivers. And try as they might, researchers will never be able to pre-program solutions for every one of them. Autonomous systems work well when the area they work in is predictable. Unfortunately, the real world has so many potential combinations of factors (including other drivers) that an autonomous vehicle will eventual encounter a situation that was not anticipated by the design team.

During these times, only the human in the vehicle will be able to adapt and create a course of action on the fly. Even with all of the added technology, the human is still a vital part of the system - having only shifted roles from driver to supervisor. The technology must allow the user to take control at a moment’s notice when the automation is not able to cope with the situation. With the speeds vehicles travel, fractions of a second mean the difference between an accident and a near miss. A user must still be focused on the driving task. A driver sending an e-mail will not be able to orient their focus to the problem and decide the proper action in time.

It is vital that companies and universities continue to make advances in these automated technologies, as the improvements can indeed make driving much safer. But these improvements will only occur if those companies who implement this automation are careful to keep the driver as an active part of the system. Without the fully participating human driver, we will begin to see accidents in the new autonomous vehicles as well. And the blame will ultimately fall on the user - even though the actions of the autonomous vehicle led to the accident. So the question is worth asking, if companies plan on implementing autonomous vehicles for use on public roads, will these same companies pick up my insurance coverage?

Recently we engaged in a very animated discussion with a major system developer working on Autonomous Combat Vehicles. (Lots of different acronyms here: UAV-unmanned air vehicle; USV-unmanned submersible vehicle, etc. but the themes are the same -”eliminate the human error and risk to humans by automating.”) When I opened the meeting with “your vehicles aren’t autonomous at all” you can imagine the reaction. Their point: there was no human aboard, therefore it’s autonomous. Our point: in order to actually employ the system, it takes typically many humans working with the forward deployed (”autonomous”) hardware–everything from a team of humans spending days to plan the mission to a team of two or three near real-time operators connected to it as the mission occurs. Failing to recognize that these ‘autonomous’ systems are really Joint Cognitive Systems (JCS) results in suboptimal designs rife with ‘latent errors’. These JCSs intimately couple the performance of humans with the performance of the engineered system into one large operational entity.

There are many indicators of this unrecognized coupling, from the 8X accident rate of unmanned aircraft over manned ones to the current 3:1 operator to UAV ratio (e.g. Urban Search and Rescue Robots) being far removed from the objective target of 1:100 (one operator controlling a ’swarm’ of UAVs). Given that today’s designs have not adequately recognized the coupling of the human and the engineered system as one holistic JCS, imagine the magnified impact if those same design approaches are used in an environment 300x larger (from 3:1 to 1:100). In order to make such a UAV swarm “affable” (our term for cooperative, easy to get along with, easy to employ for mission purposes UAVs), a different type of system must be constructed.

Cognitive Systems Engineering has developed an extensive history of ‘patterns’ in such situations (albeit from different specific domains) (see Joint Cognitive Systems, Woods&Hollnagel 2006) which offer the key insights needed if the ‘autonomous’ community is ever to succeed in realizing the truly revolutionary potential of these autonomous technologies.

Just to end the story we opened with, the meeting concluded with the system developers agreeing that the systems they fielded needed 3 real-time operators for each deployed system’s operation, but still insisting that they were building an Unmanned Combat Vehicle.

Collaboration is getting enormous amounts of attention from research, systems engineering, and commercial product sectors.  Unfortunately, virtually all of that attention is based on a false premise:  that recreating a physical environment, and a physical meeting will support collaboration.  That premise presupposes that merely creating a virtual equivalent to a physically co-located meeting room will achieve the shared thought processes which underly truly effective collaboration.  Cognitive Systems Engineering research  (e.g. Woods, Patterson, et al 2002) clearly shows that there are more fundamental elements of cognitive work involved in what is colloquially known as ‘collaboration’. 

From that research, we’ve developed both collocated and distributed forms of collaborative support that extend the same basic analysis and design concepts used by Cognitive Systems Engineers for the design of individual decision support.  For example, when creating a shared display for an Operation Center (typically on the front wall), most systems designers merely transfer a single operator display to the large presentation media.  We’ve found that the design of a unique style of Group Functional Display, with added features to explicitly depict actual and supervisor focused ‘attention points’ provides a much more powerful common ground for collaboration.  Based on an explicit model of the information space, with the ‘attention points’ overlaid, this Group Functional Display is a powerful complement to other, similar decision support elements integrated into the large scale environment. 

By employing this type of ‘extended’ Cognitive Systems Engineering into this Macro scale decision support design, we provide Decision Support Environments as multi-user/multi-decision support system instantiations.  We call the resulting Decision Support Environments “Command Decision Centers” rather than Operations Rooms.

System designers of all types are leaping onto the “flexibility” and “user configurability” enabled by current web technologies as a boon to usability.  The claim is that users will be able to compose “exactly what they want, when they want it” within a flexible framework of servlets, portlets, applets and the like.  They believe that, since the user configured it, by definition then it is a user-centered design, and must contain exactly the right decision support–after all the user picked it.

Not surprisingly, this has reduced the designer/software developer’s burden and reduced system development costs (and the claim of ‘better’ user support adds to the apparent ROI).  Unfortunately, the truth is somewhat different.  The assumption that a user is *also* a fully trained and qualified system designer is akin to saying drivers are fully qualified automotive designers…it just isn’t so.  Taken to another metaphor, just because you live and work in a building doesn’t make you a licensed, qualified architect.  Why then the zest to make users into impromptu designers? 

In part, it is a reflection on the shortcomings of the entire field of computerized decision support.  The engineering community admittedly struggles with how to discover ‘the right requirements’, focusing instead on “given you have good requirements, here is the process to carry them through to fielded, tested systems”.  As evidence, consider the plethora of weak methods claiming “user centric design”:  1) have a user on the design team/in all the meetings, 2) do RAD/JAD (now Extreme Programming) meaning the programmer builds it as fast as opinions come out of a user’s mouth, 3) do spiral development, so users will ‘know it when they see it’, etc.  Overall, no one is satisfied with the result; systems take too long, they aren’t well received by the user community, they actually don’t help the mission as claimed at their inception, etc.

Thus, creating “flexible, user configurable” systems is just one more way to “get a user in the game somehow, in order to claim user-centered design is achieved.” Placing the burden on the user is similar to the way software languages do  ‘late binding.’  Instead of a user up front in the requirements process, the actual real time user is making ‘late binding’ design decisions at mission run time to “select this widget, put it here, put these data into it, make it this size, put this one next to it, etc.”

Cognitive Systems Engineering has repeatedly learned from systems with these kinds of complexities:

1. Users will select a small subset of available options and use them as ’standard, default’ configurations
2. Users will not learn all the other features and capabilities
3. Users will, in general, not select the optimal configuration of design elements (due in part to 1 and 2 above)

Ironically, there is also an issue not typically even considered.  The system was developed to assist the user in some important work task, ostensibly to ‘reduce the user workload’ and improve productivity, quality, resiliency to disturbances and errors, etc.  Unfortunately, by fielding a system that includes all the cognitive work of a ‘just in time designer’, the number of complex cognitive tasks the operator must complete has actually INCREASED.  When that occurs during critical or high workload intervals, the system has in some ways EXACERBATED the users’ cognitive workload, directly counteracting its stated intent.

While some amount of flexibility and reconfigurability reduces the overall brittleness of any system, to use it as the basic design premise of the entire system is not system design, but in fact, Design Abdication.  System developers should not be afraid to define an effective decision support design, but rather should become experts at their craft.  The lessons of Cognitive Systems Engineering are available to all system designers.

In case you hadn’t noticed, World Usability Day 2007 is Nov 8th–and its rapidly approaching.  As a good friend often reminds me, such things as a Day dedicated to Usability are “…necessary but not sufficient…” in the larger picture.  That comment is both a compliment (necessary) but also a compelling demand for further effort.  In short, you have to get Usability correct, or the design is so awkward, frustrating, etc. that it is impossible to get on with the work at hand (when usability is poor to the extreme).  On the other hand, Usability done well does not ensure that the comfortable affordance provided is at all effective in actually accomplishing the goals/work at hand.

One of the downsides of such ‘celebrations’ is the wide variety of connotations and confusions that can develop.  Last year we were bemused by a local event that had a contest where audience members were picked, assembled into teams, and given 90 seconds to ‘invent a design’.  While some brainstorming techniques profit from that ‘hip shoot’ approach to seed discussions, to treat design in such a flippant way (they picked a winning team, presented as a ‘good’ design) is to minimize both the challenging effort needed and the desired goal of such an effort.  “Your best idea in 90 seconds” is a pretty low bar to shoot for.

From a Cognitive Systems Engineering perspective, we prefer Woods’ characterization of this subject:  that a system must be evaluated for its Usability, Usefulness, and Understanding.  Usability being the typical definition related to look and feel, the affordance perspective.  Usefulness referring to its ability to support actual work in the domain.  Understanding meaning the ability of the system to assist the user in a full, in depth understanding of the decision making and the work domain itself.  The three amount to a progression of deepening of the support provided and the resulting power provided to the user.  A user who ‘understands’ is much more robust at detecting and recovering from errors than one that merely has ‘usability’ features that make the knobs easy to operate.

During a recent major system evaluation, we were careful to separate explicit and observed user feedback into those three categories.  One of the ‘gut check’ questions given to our engineering team was to watch for the following paradox  (credit to D. D. Woods for the initial insight on this one):  In a group Decision Support Environment like this one (a military ops room) you may find users engaging in a “heated debate”.  That could be an indicator of many things (poor usability, etc.) but could also be an indicator of a very good thing:  If the system is effectively providing support for Understanding, you can expect to find users arguing deep subjects about their field that are for the first time *enabled* by the decision support system.  While you’d normally think angry users to be a bad thing during a review, in this case its a powerful tribute to the Understandability Support provided.

While we had our share of ‘things to fix’ from the review, there were a few notable cases where *exactly* Woods’ prediction occurred.  We witnessed users pointing energetically at elements of the design, switching displays, all while having a very subtle but powerful debate about weapons engagement doctrine ‘through the window provided by our system’.

World Usablity Day is a step in the right direction, now all we need is World Usefulness Day and World Understanding Day to shift the focus to decision support.

On Saturday, November 3^rd, DARPA will conduct its third edition of the Grand Challenge. The Grand Challenge is a race between autonomous vehicles, which must overcome a number of obstacles in order get to the finish line. This competition is being done in order to accelerate the research in support of the military’s desire to have at least 1/3^rd of ground vehicles unmanned by the year 2015.

While many espouse the practice of using autonomous vehicles, the safety record of autonomous vehicles up to this point has been very poor. They are very brittle, as technology-only systems cannot adapt when situations change, because not every situation can be predicted in advance. No operator (human or autonomous) has a complete knowledge set of the world. However, people, as adaptable problem solvers, can think outside of the expected solution set and come up with solutions to novel problems. Autonomous agents, on the other hand, CANNOT do this. Being heavily rigid and rule-based has many benefits, but these characteristics do not allow for solutions to novel problems.

This is what makes the Joint Cognitive System (JCS) concept so powerful. Designing a system as a JCS means that the rigidness of the autonomous agent and the adaptability of the human are combined into a single decision-making unit. They work together to take advantage of what each component does well. There are several reasons given why technologists want to implement autonomous vehicles. A major reason, especially in military applications, is to save human life by getting the human operator out of harms way; this is quite a noble cause. The other benefits offered, however, are more questionable. Other reasons for support of autonomous vehicles is that, once perfected, they will eliminate accidents, since they do not get tired, they have faster reaction times, etc… However, removing the human does not mean that accidents will be eliminated. There are too many possible situations and permutations of events to be able to predict and provide answers for in advance. Autonomous agents cannot handle the unexpected. And, even though some incidents may be avoided because of autonomous control, others will continue to occur, and we may even see the advent of new types of accidents which never occurred with human control.

Additionally, it takes these autonomous vehicles a lot more processing and information fusion in order to operate in what most humans do very successfully every day. Last year, if you do the math, a vehicle was given GPS waypoints on average of something like every 10 meters (e.g. 100 waypoints per kilometer) as part of its ‘instructions’. Although these vehicles operate ‘autonomously”, they are receiving a lot of instructions beforehand to ensure they go where they are supposed to. True autonomous operations would operate with flexible, on-the-move planning.

Based on discovered Cognitive Systems Engineering patterns of autonomous operators, we would like to make several predictions about the upcoming Grand Challenge:

-Even after all the careful design and planning, the ‘brittleness’ of automation will be revealed by the vagaries and unknowns of the ‘real world.’ This occurred in 2005, in which one vehicle interpreted its own shadow as an obstacle and kept trying to avoid it.

Prediction: There will be at least one such case where a vehicle retires from the race because it cannot get out of the way of its own programming.

-The vehicle supervisors are stuck in a no-win situation. They are trying to complete the race and maintain the necessary safety for the vehicle and any pedestrians, competitors, and spectators. Supervisors must walk a fine line between hitting the kill switch, which disqualifies the vehicle, or letting it continue and having the vehicle do something unsafe, which could endanger people and/or the vehicle.

Prediction: At least one ’supervisor’ will have to hit the ‘kill’ switch. With a little luck we’ll get the comment ‘in hindsight we could have let it continue and it would have finished the race.’

-Even when rules are well-defined for an autonomous vehicle, it is possible for people with hostile intents to take advantage of these rules. Knowing the rules by which a vehicle operates will allow people to trick and misguide the vehicle down a path that it cannot escape.

Prediction: While such actions may not be tied into the Grand Challenge itself (although they should be), when fielded, autonomous vehicles will be forced to respond to adaptive adversaries.

By rights, this should have a subtitle:  “What’s all the hub-bub about?”

When Jens Rasmussen basically started the field with his 1986 book “Information Processing, Human Machine Interaction, an approach to Cognitive Engineering“, he coined the term Cognitive Engineering.  Since Jens was himself coming from a Control Systems Engineering background, the field has been engineering focused since its inception.  (His updated version “Cognitive Systems Engineering” in 1994 has tended to confuse things a bit among some practitioners in the field.)  Jens remains firmly an engineer at heart, saying the books are intended to help people think about, and build, better systems  “…use what helps, change or forget about what doesn’t…”

Woods and Hollnagel explicitly added the “Systems” in the middle to make things even clearer:  its all about Systems Engineering, adding that one, all important perspective:  the realization that every designer is really designing an interaction between the human part and the systems part of the larger ‘Joint Cognitive System (JCS)’.  Similar to Don Norman’s now famous thoughts on physical affordances (and how their design can vex the human part of the joint cognitive system), Woods and Hollnagel say exactly the same things about computerized, automated systems now ubiquitous in our environment.

With all that said, the CSE community is now engaged in a ‘new’ revelation, often described as “Bridging the gap between Cognitive Systems Engineering (CSE)  and Systems Engineering(SE).”  Workshops are being held (see IHMC link in the blogroll).  Papers are being authored, panels are being convened (see HFES 2007, CEDM proceedings).  But it certainly seems confusing on where this ‘gap’ came from?  Since its roots as an discipline to engineer systems how could a gap exist in the first place.  Possibly the gap ‘grew’ as the academic discussions focused on those “Angels on pin heads” type thoughts and lost sight of the engineering roots?  Isn’t it ironic that two fielding sharing the same name albeit for one initial adjective describe themselves as separated by a ‘gap’?

Curiously, for the last two(+) decades, the staff now at Resilient Cognitive Solutions has been totally focused on Jens’ initial vision of pragmatic systems engineering.  Those pragmatic adaptations often resulting in the academics seeing us as somewhat ‘unclean’ in our techniques, i.e. not in keeping with the ‘teachings’ of CSE.  Ironically, the systems engineering community saw us in a similar light, being ‘unclean’ in the adaptations we’d been forced to make to accepted systems engineering practices.  Ultimately that position between “The CSE Community Rock” and “The SE Hard Place” is best viewed as a positive indicator that we are *exactly* in the right position:  Applying Cognitive Systems Engineering practices directly inside the traditional Systems Engineering processes.

Confirming that good news was a recent assessment of our processes:  “If the recent panels and workshops and papers define the current benchmark of CSE in Systems Engineering, then the work at RCS is well beyond today’s benchmark.”  Maybe our position is about exactly where it needs to be to design and deliver truly revolutionary new decision support systems.

Recently had the priviledge of participating in a workshop with some of the preeminent researchers in the Artificial Intelligence.  As you’d expect, the discussions were intellectually challenging and got extremely technical early…and stayed there.  As the results of the workshop were being assembled, several of the “reasoning” discussions (also called “agents”, sometimes called “logic”, in short, reasoning operating under several psuedonyms) the presenters inexorably came around to the subject of “How to explain the results of the reasoning to the users.”  This is a classic AI subject, dating back to the beginnings of AI, and frankly one that remains vexing to the community.

A friend used to use “His grandfather’s common sense test”:  If you couldn’t explain it to Grandpa in common sense terms then it probably wasn’t a good thing to do.   Grandpa’s question to the Explanation Technologists would probably be something like “Wait a minute, the thing you built in the first place isnt understandable.  So to fix it, you are going to build *another* one the same way?  If you can’t undertand the first one what makes you think you can understand the second one?”  In fairness to Grandpa, he’s probably onto something.  The recursive use of the same design approach almost ensures an endless spiral of system n+1 explaining system n until either the budget or the user is exhausted.

From a Cognitive Systems Engineering perspective Grandpa’s question begins to make sense.  Technology components (aka automation, agent, engine, etc.) must be Observable (adapted from Control Theory’s use of the term)  in order to team effectively with the users.  Those two attributes, if delivered successfully, obviate the need for Explanation.  Rather than apply layer after layer of the same technology that was not understood in the first place, making the operation of the technology itself transparent to the user provides an entirely new perspective on *preventing* the explanation problem in the first place.