We’ve learned (the hard way) that many of the systems we’re asked to redesign, evaluate, or replace have (in our atypical Cognitive Systems Engineering point of view) such significant, fundamental issues that we now start with what we call ‘Analysis by Inspection.’ Essentially, there is no reason to conduct expensive and intrusive observation opportunities (such as simulator tests, etc.) until the larger ‘rough edges’ can be knocked off. This is a bit unorthodox for many customers who expect a hoard of clip board carrying evaluators instantly descending on their users. By ‘Analysis by Inspection’ we mean taking a step back from the system and ‘looking at it’ from a Cognitive Systems Engineering perspective. It’s an extremely productive way to discover the ‘low hanging fruit,’ but it also allows an initial assessment of whether the system is ready for more in-depth, but much more expensive. evaluation approaches.
For this particular evaluation our initial efforts are going well. The startup activities included (for example) a full day train up session (described in Part 1) and reading the user manual. Our Analysis by Inspection compiled some of the typical information:
- This system is composed of over 100 individual windows, popups, dialog boxes, etc. (The developers were surprised by this number as their initial estimates were much smaller. The system uses a mix of buttons, tabs, menu bars, pull down menus, etc. (not unique or surprising), but we immediately noticed some ‘clustering’ of the 100 displays and the usage of various controls: some favored tabs, some favored buttons, others menus.
- Between 100 and 200 controls, a combination of buttons, icons, mouse actions, moused menus, menu bars.
- The ‘older’ parts of the system showed evidence of incremental, agile development, sometimes having four different ways to navigate to the same state (a classic sign of a ‘literal’ style of User Centered Design, meaning at some point in time, a user said ‘I want to be able to get from here to here’ and the developer did exactly that, rather than consider a larger context of the design of the overall system.)
Much of the work of this ‘Inspection’ was captured as sketches and notes on the Project Wall for this effort. The system was ‘disassembled’ (picture a computer system version of TLC’s Overhaulin‘) where much of the statistics above began to jump out, and some interesting evaluation insights occurred:
- The ‘disassembled’ system components were linked back together into sort of a navigational State Transition Diagram. User actions on controls changed the ‘display state’ of the system. As soon as you step back from that diagram, complex clusters of paths become apparent (like a complex part of a city street map). Even more interesting are where the developers responded to user demands by installing menus of top level ’shortcuts’ (our explanation, yet to be confirmed). These shortcuts jump deep into the diagram, at the cost of damaging any Mental Model of the System being formed by the user. From a Cognitive Systems Engineering standpoint, we normally try to design the system to overlay on the Mental Model of the work domain itself, so the user doesn’t have to understand the work and extend that mental model with a second, unique one of the system (not even factoring in the complexity of the interrelationships needed to use the system to accomplish work in the world.)
- The system formed itself into clusters from several dimensions: a) density of ‘action paths’ linking the displays within the cluster (and a corresponding paucity of links between these clusters). Our unconfirmed assessment of this indicates separate development ’scrums’ over the life of the program–we’ll be confirming this with the development team. It would be interesting if we were able to be ’system archaeologists’ and rediscover its genesis by the fossilized evidence within the system. b) A difference of the interaction choices across these clusters (preference for tabs vs. menus vs. buttons.
- Several areas where ’system objects’ paralleled ‘world objects,’ meaning sometimes the user interacted with the basic object itself, sometimes with a system generated surrogate (typically linked by a naming text string). This required the user to remember which type of actions occurred on which side of this ‘divide,’ along with the differing command styles to operate on these two different classes of object.
The productivity of this quick “Analysis by Inspection” is evident on the project wall: the team used bright orange post-it notes to flag ‘hot spots’ in the system for further inspection. In a period of two days, over 50 of these orange notes and the associated ’so what’ descriptions of each were generated. We’ll be summarizing this initial assessment with the customer soon, and together making some decisions about whether some development should be done ‘to knock some corners off’ before we get into the more elaborate, expensive types of observations/evaluations.
Coming soon: Part 3: Delivering Initial Findings
Stumble it!