Jefferies et al, an evaluator from the University of New Orleans carried out a study that evaluated four different techniques to be used in conjunction with a user interface created for a software product prior to its release by four groups. The four methods used by Robin Jefferies were heuristic evaluation, software guidelines, cognitive walkthroughs, and usability testing. At the end of such testing, Jefferies found that heuristic evaluations produced the best results, as this technique acknowledged the most usability problems, reporting one-third of the most severe problems and two-thirds of the least severe. Furthermore in this study, Robin stated that the serious problems discovered by several user interface specialists required the least amount of effort; hence heuristic evaluations had a distinct cost/benefit advantage.
Karat et al also carried out a comparable study into the empirical testing with individual and team walkthroughs using heuristic evaluations. The ‘walkthroughs’ in her study were fundamentally heuristic evaluations since each evaluator makes use of 12 usability guidelines, 8 of which come from Jakob Nielsen's original set of heuristics, to find usability problems. Opposing to this ‘aforesaid’ study, the experiential testing condition identified the largest number of problems and also a important number of moderate severe problems that were missed by the so called ‘walkthrough’ conditions. In addition to empirical testing, the heuristic evaluation was considered more cost-effective since it required the same or less time to recognise each problem when compared to walkthroughs. About a third of the significant usability problems identified were common across all three of the methods tested.
The distinctions that have occurred within the last two studies can be speculated to a certain extent. The fact that user testing was more cost-effective in one study and the opposite in the other case may be due to the differences in walkthrough procedures and the type of data analysis. The differences in evaluator expertise or the length of time to perform the heuristic evaluations could have caused the variation in which method found the most usability problems in the user interface. The higher degree of overlap in the experiment conducted by Karat may be due to the essential differences in the two scenarios used in the two studies.
As can be seen, many factors can affect the comparative achievement of any individual method or technique. Therefore, this makes it very difficult to compare different evaluation methods; hence I think the best approach is to remember that each method has its strengths and weaknesses depending on its application.
Heuristic Evaluation; as a discount usability engineering method, I do not think that heuristic evaluations guarantee to find every last usability problem in a design of a interface. For example, if there’s a system that is highly domain-dependent and the evaluators has little domain expertise, usability problems are likely to be overlooked. For reasons similar to this, I think various usability methods should be combined for each project, however this may vary slightly depending on the exact characteristics of the system in question. Therefore, I would agree with Janice Bradford, an evaluation expert from San Jose College, by saying I would recommend using the heuristic evaluation method with a form of user testing. Characteristically, this would work by one individual performing a heuristic evaluation to clean up the interface and remove as many usability problems as possible. Hence; after a redesign, the interface would be subjected to user testing both to check the outcome of the iterative design step and to find any remaining usability problems that were not picked up by the heuristic evaluation. I think altering between heuristic evaluation and user testing would be advantageous to the interface, firstly as ‘a heuristic evaluation’ can eliminate a number of usability problems without the need to "waste users", who can be difficult to find and schedule in large numbers, and secondly, these two categories of usability evaluation methods have shown to find fairly distinct sets of usability problems.
However, if using this heuristic and user testing approach, one must not forget that Heuristic evaluations should be used with empirical user testing and not as a replacement for user testing.
I believe that heuristic evaluations can bear advantages and disadvantages when performing evaluating method. I think advantages of Heuristic evaluations are that the method is easy to use, fast and as cheap as you want it. In addition to this, since each observed usability problem is explained with reference to an established usability principle, it is fairly easy to generate fixes. I think that this is a good method for finding both major and minor problems within a user interface. As Heuristic evaluations can be employed early in the design life cycle to find usability problems, I feel this can make them considerably easier and cheaper to fix, than if the problems are discovered in later phases of the design or not at all; which make a heuristic evaluation stand a better chance of actually being used in practical design situations.
When applying heuristics correctly, there is a need for a certain degree of knowledge and experience. These usability experts are sometimes hard and expensive to come by, especially if they need to have domain expertise. If the evaluators are not part of the development team, they may not be aware of technical limitations on the design or why certain design decisions were made. Differences between development team and evaluators may occur, which can obstruct the communication and correction of the problems identified during the evaluation process. Heuristic evaluations are loosely structured and therefore run the risk of finding one-time, low-priority problems, however these problems may not be important to correct. Finally, I think the real disadvantage to the heuristic method is that is does not allow a way to assess the quality of redesigns. Overall, I feel heuristic evaluations still play an important role in the design life cycle of a user interface. I further believe if this method is implemented correctly, it can provide powerful results. I would like to finish by saying that even if the impact of a heuristic evaluation on an interface is minimal due to organisational constraints, the technique is quick, cheap and a easy evaluation which is always better than nothing.
References and Links
[1] Desurvive, H. Chapter 7: Faster; Cheaper!! Are Usability Inspection Methods as Effective as Empirical Testing? In , Nielsen, J. and Mack, R.L., Eds. John Wiley and Sons, New York, 1994, pp. 173-202.
[2] Dix, A., Finlay, J., Abowd, G., and Beale, R. Chapter 11: Evaluation Techniques. Human Computer Interaction. Prentice Hall, 1993. pp.363-400.
[3] Jefferies, R., Miller, J., Wharton, C., and Uyeda, K. User interface evaluation in the real world: A comparison of four techniques. In Proceedings of CHI '91, (New Orleans, LA, April 28 – May 3, 1991), ACM, New York, pp. 119-124.
[4] Karat, C., Campbell, R., and Fiegel, T. Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proceedings of CHI '92, (Monterey, CA, May 3-7), ACM, New York.
[5] Karat, H. Chapter 8: A Comparison of User Interface Evaluation Methods. In , Nielsen, J. and Mack, R.L., Eds. John Wiley and Sons, New York, 1994, pp. 203-233.
[6] Nielsen, J., and Molich, R. Heuristic evaluation of user interfaces. In Proceedings of CHI '90, (Seattle, WA, April 1-5), ACM, New York, pp. 249-256.
[7] Nielsen, J. Finding usability problems through heuristic evaluation. In Proceedings of CHI '92, (Monterey, CA, May 3-7), ACM, New York, pp. 372-380.
[8] Nielsen, J. Chapter 2: Heuristic Evaluation. In , Nielsen, J. and Mack, R.L., Eds. John Wiley and Sons, New York, 1994, pp. 25-62.
I extracted quotes and text from the and the used material provided by my tutor.
![Heuristic evaluation was used by Jakob Nielsen in 1990 to describe a type of inspection method [5], for identifying usability problems of computer software](https://mbt-essays-prod-public.s3.eu-west-1.amazonaws.com/97937/listing/97937_1.jpg)
