Business Intelligence in general, and Executive Dashboards for Business Intelligence specifically, are current areas of extensive blogging, lots of development activity…and in fact executives are making alot of money selling these dashboards to other executives. Unfortunately, it has also resulted in lots of incorrect marketing claims, urban legends of how executives think and what they need, and bluntly lots of very poor decision support design masquerading as ‘dashboards’. These (incorrect) urban legends have become a sort of societal meme, becoming widely held beliefs, but nonetheless incorrect. (Where is Snopes.com when you *really* need it?) The Human Factors and Ergonomics Society is even accepting ‘research studies’ on whether adding a 3D drop shadow behind a Red-Green-Amber coded tachometer degrades an executive’s ability to judge ‘how far into the red are we?’ (See for example Chipley, Barrow 2007)
Some of these common misconceptions of executives and their dashboards include:
- Executives are very busy and therefore can’t be bothered with ‘the data’
- Executives like/prefer/demand Red-Green-Amber stoplight charts and histograms and pie charts, and…
- Executives can’t understand anything more complex than <that pitiful list above>
The current ’state of the market’ in executive dashboard design overlooks even the most foundational research into what constitutes expert decision making in complex work domains. (An excellent starting point is still H. Simon’s “Models of Thought“, where Simon characterizes chess experts as thinking in abstractions of chess while novices focus on exhaustive memorization of the physical pieces.) An effective Decision Support System should support–and even promote–such expert levels of decision making performance. Yet, imagine an ‘Executive Dashboard for Chess’, designed using today’s state of the practice R-G-A, histograms, and trend lines. It’s ridiculous to envision good chess play from a set of even many bars and needles indicating “chess metric 1-green” and “chess metric 2-yellow”, etc. (Remember you cant have TOO many metrics, executives will get overwhelmed, and it must fit on an 800×600 display, or better yet fit on the BlackBerry Bold’s 480×320). In short, even for an “enterprise” as constrained as a game of chess, such a ’state of the market’ Executive Dashboard offers no support for any abstract conceptualization of chess.
Fortunately, some of the ‘executives’ have learned exactly this lesson (the hard way) and have begun seeking the kind of powerful decision support they deserve. These executives are supervising enterprises with more moving parts than the Space Shuttle Program, enough employees/subcontractors/consultants to swamp several football stadiums, and often more riding in the balance than a balance sheet. Dashboards are beginning to be seen for what they really are: Directed Attention Longshots (see Woods, Hollnagel 2006) that must provide full-breadth, informed support to the executives’ initial focusing and refocusing for decision making. In many recent system design reviews, executives have begun to ask the right ‘gut check’ questions of their system developers, e.g. “But for an enterprise of my size, something is *always* broken/out of alignment/out of tolerance–this display will *always* indicate “in the yellow” which isnt helpful to me at all!”
Executives, as customers of their own decision support environments, have properly begun to demand Decision Centered Design (provided by Cognitive Systems Engineering methodologies) to deliver more powerful and effective systems and not just ‘dumbed down dashboards.”
We just received our author copies of CRC Press’s new book: Applications of Cognitive Work Analysis. Chapter 10 of that handbook contains an end to end description of the Applied Cognitive Systems Engineering (ACSE) methodology, our Fourth Generation of ‘how to do’ Cognitive Systems Engineering in the design and development of real systems. By ‘real system’ we mean one that provides powerful, effective Decision Support for large scale, difficult, mission- and business- critical applications.
Putting a chapter on Cognitive Systems Engineering in a book entitled Cognitive Work Analysis reflects the naming oxymoron that prompted us to evolve from ACWA (Applied Cognitive Work Analysis) to ACSE (Applied Cognitive Systems Engineering). A Cognitive Systems Engineer must do far more than a Cognitive Work Analysis in order to provide adequate input and specification for the ‘construction’ of the software and surrounding system. Notably, the design decisions layered on top of that analysis, the underlying transformational layers, the representational design for the User Interface itself, and the interactions between all the systems components that define “The User Experience” (really “The Decision Support”) all require a Cognitive Systems Engineer to be on their game. This chapter attempts to describe not only how to do that, but also how it relates back to the underlying sciences of human cognitive and perception. We were excited to see the key Mapping Principle figure used in the cover art for the entire book!
The first generation of this engineering methodology began alongside some of the seminal researchers in the Cognitive Systems Engineering field: Woods, Lind, Rasmussen, Hollnagel as well as: Roth, Bennett, etc. Since those initial attempts at applying Cognitive Systems Engineering to real world systems, we’ve evolved through three subsequent incarnations of tools and techniques, to its current form, ACSE.
Enjoy the chapter! Hopefully it will give you a sense of how to analyse and design the computer system portion of a Joint Cognitive System.
Recently the Pittsburgh Technology Council and local news channels gave significant airtime to robotics pundits from Carnegie Mellon University and their General Motors sponsors claiming essentially (again) that technology will “Take human error out of the car“. Unfortunately, that has been the consistent claim of every technology maker, but the results do not support the claim.
During the interview, the technology advocates point to AntiLock Brakes as one such technology that is “better than humans”…interestingly, the actual experience after hundreds of thousands of driven miles is at best mixed. The National Highway Traffic Safety Administration’s data reports: “no benefit” in terms of reduced collisions, injury, etc. Studies commissioned by the vendors indicate at best “mixed results”. See a summary of the results. The industry has continued to ‘tune’ ABS in an attempt to improve the results in the field, but overall remains confused about why the raw benefits of the technology don’t deliver the claimed ‘massive improvements over human performance’.
To a Cognitive Systems Engineer, this result is not only understandable, it’s predictable. When any technology is introduced into a complex environment (read “the real world”) it must be introduced in a way that effectively *teams* with the user in accomplishing real world goals. So while “pulsating the brakes at up to 100 pulses/minute” is a great technology claim, how to get drivers to trust the surprising, rarely experienced sensory feedback, how to tune the technology to engage only at the appropriate times (e.g. not in deep snow or mud, not below X mph, etc.), remain real world, Cognitive Systems Engineering issues that technology purists continue to overlook.
Even the technologies widely proclaimed as successes in DARPA’s Urban Challenge were facing adjustments during testing like “lets reduce our clearance margin from 11 seconds to 8 seconds because this one particular intersection’s field of view only gives us a 10 second window”. Such adjustments are brittle corrections that do not work ‘in the wild’ where the network of system designers and tuners is unable to post-hoc analyze a single situation and specifically tune the system for that environment.
Ironically (and circularly) technologists respond with “we can wrap machine learning around the basic technology” so now it can learn such settings over time…the patch to the initial technology is to layer additional technolog all the while not fully considering the complex Joint Cognitive System(s) that are implicitly being created. By ’shaping’ the technology to team effectively with the adaptable, resilient, qualitative reasoning skills of the human the best overall system design is achieved. D. D. Woods’ discussions of “Making Automation a Team Player” in the previous link represent a totally new direction in employing the power of advanced technologies most effectively.
Recently, lots of buzz has erupted around “Visual Analytics”. It has resulting in the ordaining of PNNL as the National Visual Analytics Center (NVAC) with satellite Regional Centers (RVACs) around the world.
In a recent article, Visual Analytics has been described by Thomas as:
“Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces.” It is traditional scientific visualization, taken to the next level. In “Challenges in Visual Data Analysis” (2006), Keim and his co-authors note that visual analytics combines visualization with human factors, information, geospatial, and scientific analytics. As pointed out in that paper, “Especially human factors (e.g., interaction, cognition, perception, collaboration, presentation, and dissemination) play a key role in the communication between human and computer, as well as in the decision making process.”
Ironically, that description is a superficial description of one facet of Cognitive Systems Engineering, which has been ‘new’ since 1982. So again, it seems some good PR has overtaken that actual technical issues. Even more so, this ‘new’ field is centered on the relatively brittle end of the discipline: the graphical form. Cognitive Systems Engineering considers not only the ‘Representational Effect’ of the visual form, but spends considerable effort in explicitly discovering the nature of the decision making process, and how to effectively support it as an explicit part of an overall design process.
The Visual Analytics community focuses more on the visual form and asks “does it do something useful”, while the Cognitive Systems Engineering field asks “what work is being done, and how do we effectively support it with visualizations and other technologies as part of a Decision Support System design?” It considers the user together with the technology as a Joint Cognitive System rather than just a visualization. In short, why is the ‘new’ field of Visual Analytics not exploiting the 25 years of efforts by the Cognitive Systems Engineering community. Or maybe a better question: While it has a new name, is Visual Analytics really a “new science”?
A new marketing trend has emerged: Cognitive Radios, Cognitive Safety Systems, Cognitive Binoculars, etc. The problem is that the underlying ’system’ isn’t really any different, just the marketing hype has changed. TRW recently announced its new “Advanced Automatic Emergency Braking System” as “…another example of a Cognitive Safety System …” in a recent press release.
As with most things, this is a ‘good news/bad news’ story. With the increase in the awareness of ‘things cognitive’ has come a marketing pressure to jump on that wave and get some of that market. The good news is that there is an increased market awareness of designing systems based on insights into how humans make decisions and the support that should be provided by systems. The bad news is that old school system designs are just being ‘re-branded’ as Cognitive Systems. Claiming, as the above press release does, that “sensing the environment and reacting appropriately” makes it a ‘cognitive system’ essentially means that every control system ever built is such a ‘cognitive system’. They all have sensors in the world, they all have implicit (software) models of how to respond, and they all trigger actuators to have an impact on the controlled system.
Unfortunately, confusing yet more automation, with a (joint) cognitive system (see Hollnagel and Woods, 2005) confuses the marketplace, risks disillusioning an already stressed user community, and, overall, sets back true Cognitive Systems Engineering. While systems engineers should be praised for considering how humans think, their attempts at automated system mimicry do not constitute effective Decision Support System designs.
Recently I heard a radio interview of an author of a new book on American History. The gist of his book is the striking difference between the actual facts of European settlement of North America and what people *believe* that history to be. One of his examples was why Plymouth Rock is widely held to be ‘the birthplace of the nation in 1620′ when this plainly conflicts with the also widely held belief captured in the little grade school ditty about Columbus and 1492. Obviously, both cannot be ‘the first discovery’, yet people are not troubled by their internal knowledge incongruity. (In fact there are many other conflicting facts about the Vikings, Spanish, French, etc.) When pressed, people couldn’t reconcile the bits they knew about Columbus, Cortez, the French in Lousiana, the Spanish in Florida, California, etc. with what they believed about Plymouth Rock. As one example, a Boston history tour bus driver was feeling particularly harassed by the questioning finally blurted out “forget all that other stuff, it happened right here!”. (just incidentally, that is one typical human problem solving approach to conflicting data…not necessarily a good one, but one that does resolve the deadlock) The author’s summary statement for all this was striking: “once a myth is established, it is impossible to uproot.” He went on to say, no matter how many facts he offered to the contrary, no matter how much evidence he presented, “the myth” still won out.
What we’ve similarly come to discover are just such myths about interface design. However they were created, for whatever reason, they are now an entrenched parts of the Systems Development lore. And the Cognitive Systems Engineering community faces the author’s dilemma: no matter how much evidence is presented, the myth persists, system engineers continue to operate the same way. We are finding it impossible to uproot these system development myths. In particular, User Interface Myths spread by a meme-like mechanism: someone uses a technology to create a new ‘widget’, it is seen by others, the seed takes hold and is reused and recycled and spreads throughout the community. You’ll hear phrases like “I like the way iGoogle did <xyz>, why don’t we build <abc> of our system the same way?”…and so it goes, regardless of whether the work to be supported is the same as iGoogle or not (what work does iGoogle support anyway??).
There is one slight benefit in all of this: The continued use of “the myths” in systems under development provides an ever expanding set of case studies where the Cognitive Systems Engineering patterns (i.e. the patterns in system design errors, see Woods/Hollnagel 2006), are repeated, and the predictable “latent disasters” come to fruition…eventually. Maybe by building an ever higher mountain of evidence, the Cognitive Systems Engineers will be able to prove the author wrong and eventually defeat the myths.
Recently, Gary Klein et-al. published an article Cognitive Systems Engineering: The Hype and the Hope that touched on several interesting points about how Cognitive Systems Engineering (CSE) fits into the broader field of Systems Engineering. If you are at all interested in systems design, it’s a great ‘light reading’ introduction to CSE and how it dramatically improves systems in ways that ‘classic’ systems engineering just doesn’t address. Systems engineering is essentially technology-centric. Over the last several years it has added ‘process-centric’ in an attempt to build better systems more efficiently. More recently, it has attempted to improve ‘user acceptance’ as part of that efficiency (i.e. if the users reject the use of the final system, the net development efficiency is clearly 0%). These attempts to ‘include the user’ have brought you RAD/JAD, Use Cases, Extreme Programming, etc. which have had some success, but continue to result in systems which don’t really support users in performing real cognitive work in real situations. Klein et-al. have chosen to discuss the CSE connections with Systems Engineering using the provocative form of “myths” and then discuss how the myths distort/confuse the real truth. The writing style makes it an easy, light read, but unfortunately results in a series of false dichotomies to enable the ‘myth’ writing construct.
For example, the opening discussed autopilot settings in a dynamic situation. Then, the user is asked the false question: “will the pilot anticipate that the system deletes altitude settings when runways are changed?”. The fundamental CSE issue at hand is the lack of Automation Observability (see Woods’ podcast The Laws of Stretched Systems in Action: Exploiting Robots , i.e. the system design does not explicitly indicate it’s automated action, such as (discarding altitude settings) in any way to the user. This is revealed in the implicit requirement for the pilot to memorize this behavior during training, and recall it during the high stress dynamic event.
Let’s take the ‘myth-i-ness’ out of the discussion to eliminate the hyperbole:
1. To say CSE was created in response to ‘large number of failed software systems’ isn’t exactly a fair rendering of the history. There have been massive numbers of studies on the high cost/low productivity of software engineering efforts. The SEI’s entire CMM effort was created to respond to this issue. Obliquely, ‘user acceptance’ has been one factor in ‘failed software’ projects. Since CSE (done well) does make the support for user cognitive work an *explicit* part of the requirement and design efforts, you could make the case for a linkage existing.
2. Myth 1 makes an artificial distinction between project managers and systems engineering. Systems Engineering contains ‘project management’ as a critical component. The fact that there are Project Management Professionals that specialize in the area does not make them separate and distinct. In order for CSE to impact the development of real systems it must become part of the engineering processes, and must be reflected in the project management of those development efforts.
3. Myth 2: CSE is a strategy. Again, this is another false dichotomy. Any engineering discipline has multiple methods for doing the work – (witness the large number of software engineering methodologies available to the software communities). The point that should be conveyed can be stated like this: within the CSE field, there are a variety of techniques/methods/processes in use today. They differ in any number of characteristics, some more readily suited to being part of a larger systems engineering effort than others. They range from “I’m a cognitive psychologist so my system designs must be CSE” to methods like our Applied Cognitive Systems Engineering (ACSE) approach that is explicitly tailored to connect to systems, software, QA, and Test engineering processes.
4. Myth 3: CSE should minimize the user’s cognitive requirements. This one is a bit oddly stated. It is intended to mean ‘cognitive work should be offloaded to automation as much as possible.’ To some degree, there are systems designers who very much want to drink this kool-aid, so the myth is, from one point of view, correct: blindly seeking maximal automation is a bad systems design, so clearly CSE wouldn’t advocate maximal automation. Further, an automation that *is* designed into the system, itself creates new elements of cognitive work (to now supervise that automation) which then have to be supported by the system.
5. Myth 4: CSE is a way to get user opinions into the design process. Simply put, CSE, to be valid engineering, has to be based on solid science and findings, not opinions. So CSE is about discovering the actual cognitive work and designing to support it, vice responding literally to users’ stated opinions. Guess that confirms this myth is a myth, but that point is not made in the paper.
6. Myth 5: CSE is essential to good system design. There is an interesting dilemma here, that is again falsely constructed. Its phrased as an all or nothing. Suffice it to say that there are lots of fielded systems that are graded as ‘good design’ because they are better than the crayon and pointed stick that the users had prior. For us, that standard is set too low. Those same fielded systems contain provable ‘decision support flaws’ (like the autopilot at the top of this discussion) that are readily correctable, well-known, and well-proven to cause ‘operator error’ eventually…and therefore those system designs should not be declared ‘good’. Bottom line: CSE is the only engineering practice that explicitly identifies the decision making to be supported by systems…so it should be part of the system design processes…QED
Lastly, the article tries to justify ‘the added cost’ of CSE by a cost/benefit ratio discussion. Ironically, the article overlooks the current costs of designing systems poorly. For example, large sums are spent on “requirements derivation” during large systems engineering efforts. Yet those efforts (and their costs) fail to ‘require’ critical aspects of the design (again, refer back to the autopilot example above: why didn’t the money spent on requirements mandate the proper observability?). In short, CSE can take a portion of the same budget currently in place and *improve* the results already being paid for in existing systems engineering processes.
We’ll leave the discussion of RPD at the end of the paper for another day…
http://www.designnews.com/article/CA6519363.html
With the recent completion of the Urban Challenge by several vehicles, GM has decided that it was time to bring the robot-driven vehicle to the public. The goal is to have the self-driven vehicle available for purchase in 2018 – 10 years from now. To demonstrate the ability of autonomous vehicles, GM brought the winner of the Urban Challenge to the recent Computer Electronics Show and allowed attendees to ride along in the vehicle. Showcasing the vehicle’s response to “unexpected” events, the GM team placed a few trash cans in the road and drove other vehicles in its path.
Having driven for many years, I don’t believe these to be very unexpected events. That the vehicle can do these things is indeed a technological feat, but these small demonstrations do not prove that autonomous vehicles are able to drive more safely than human drivers nor that they can respond to unexpected events. An unexpected event is just that - unexpected. It is not planned before hand, it cannot be anticipated, and a response for it cannot be preprogrammed. When the truly unexpected occurs, the system will have to come up with a solution to the issue at a moment’s notice. Research has shown that only humans have this ability to adapt to the unexpected.
Consider the first time that the power goes out and a police officer has to wave traffic on at an intersection where there is normally a stop light. This occurs very rarely, and the autonomous vehicle will have to not only recognize the situation, but adapt to the changing situation. As human drivers, it is very easy to see when the officer is waving each driver on and when the officer is asking each to stop. Will the autonomous vehicle be able to recognize when the officer is waving it on? Or consider what happens when a school bus stops at a 4 way intersection. Will the autonomous vehicle recognize that it is a school bus, and wait until the stop sign on the school bus has been closed? These are two situations that drivers rarely encounter, yet drivers can handle these easily when necessary.
It is possible, and even likely, that researchers will be able to devise solutions to these very specific problems. However, there are thousands, maybe millions of these very rare situations that need to be handled by drivers. And try as they might, researchers will never be able to pre-program solutions for every one of them. Autonomous systems work well when the area they work in is predictable. Unfortunately, the real world has so many potential combinations of factors (including other drivers) that an autonomous vehicle will eventual encounter a situation that was not anticipated by the design team.
During these times, only the human in the vehicle will be able to adapt and create a course of action on the fly. Even with all of the added technology, the human is still a vital part of the system - having only shifted roles from driver to supervisor. The technology must allow the user to take control at a moment’s notice when the automation is not able to cope with the situation. With the speeds vehicles travel, fractions of a second mean the difference between an accident and a near miss. A user must still be focused on the driving task. A driver sending an e-mail will not be able to orient their focus to the problem and decide the proper action in time.
It is vital that companies and universities continue to make advances in these automated technologies, as the improvements can indeed make driving much safer. But these improvements will only occur if those companies who implement this automation are careful to keep the driver as an active part of the system. Without the fully participating human driver, we will begin to see accidents in the new autonomous vehicles as well. And the blame will ultimately fall on the user - even though the actions of the autonomous vehicle led to the accident. So the question is worth asking, if companies plan on implementing autonomous vehicles for use on public roads, will these same companies pick up my insurance coverage?
Recently we engaged in a very animated discussion with a major system developer working on Autonomous Combat Vehicles. (Lots of different acronyms here: UAV-unmanned air vehicle; USV-unmanned submersible vehicle, etc. but the themes are the same -”eliminate the human error and risk to humans by automating.”) When I opened the meeting with “your vehicles aren’t autonomous at all” you can imagine the reaction. Their point: there was no human aboard, therefore it’s autonomous. Our point: in order to actually employ the system, it takes typically many humans working with the forward deployed (”autonomous”) hardware–everything from a team of humans spending days to plan the mission to a team of two or three near real-time operators connected to it as the mission occurs. Failing to recognize that these ‘autonomous’ systems are really Joint Cognitive Systems (JCS) results in suboptimal designs rife with ‘latent errors’. These JCSs intimately couple the performance of humans with the performance of the engineered system into one large operational entity.
There are many indicators of this unrecognized coupling, from the 8X accident rate of unmanned aircraft over manned ones to the current 3:1 operator to UAV ratio (e.g. Urban Search and Rescue Robots) being far removed from the objective target of 1:100 (one operator controlling a ’swarm’ of UAVs). Given that today’s designs have not adequately recognized the coupling of the human and the engineered system as one holistic JCS, imagine the magnified impact if those same design approaches are used in an environment 300x larger (from 3:1 to 1:100). In order to make such a UAV swarm “affable” (our term for cooperative, easy to get along with, easy to employ for mission purposes UAVs), a different type of system must be constructed.
Cognitive Systems Engineering has developed an extensive history of ‘patterns’ in such situations (albeit from different specific domains) (see Joint Cognitive Systems, Woods&Hollnagel 2006) which offer the key insights needed if the ‘autonomous’ community is ever to succeed in realizing the truly revolutionary potential of these autonomous technologies.
Just to end the story we opened with, the meeting concluded with the system developers agreeing that the systems they fielded needed 3 real-time operators for each deployed system’s operation, but still insisting that they were building an Unmanned Combat Vehicle.
Collaboration is getting enormous amounts of attention from research, systems engineering, and commercial product sectors. Unfortunately, virtually all of that attention is based on a false premise: that recreating a physical environment, and a physical meeting will support collaboration. That premise presupposes that merely creating a virtual equivalent to a physically co-located meeting room will achieve the shared thought processes which underly truly effective collaboration. Cognitive Systems Engineering research (e.g. Woods, Patterson, et al 2002) clearly shows that there are more fundamental elements of cognitive work involved in what is colloquially known as ‘collaboration’.
From that research, we’ve developed both collocated and distributed forms of collaborative support that extend the same basic analysis and design concepts used by Cognitive Systems Engineers for the design of individual decision support. For example, when creating a shared display for an Operation Center (typically on the front wall), most systems designers merely transfer a single operator display to the large presentation media. We’ve found that the design of a unique style of Group Functional Display, with added features to explicitly depict actual and supervisor focused ‘attention points’ provides a much more powerful common ground for collaboration. Based on an explicit model of the information space, with the ‘attention points’ overlaid, this Group Functional Display is a powerful complement to other, similar decision support elements integrated into the large scale environment.
By employing this type of ‘extended’ Cognitive Systems Engineering into this Macro scale decision support design, we provide Decision Support Environments as multi-user/multi-decision support system instantiations. We call the resulting Decision Support Environments “Command Decision Centers” rather than Operations Rooms.
Stumble it!