Dissenting Opinion
In this blog entry, I’d like to present my ideas on what a usability study is for, improve upon the development process presented by Derrick Parkhurst, and argue against some specific points (PowerPoint slides) he made.
The Purpose of Usability Studies
There are a number of reasons why you might want to run usability tests. some people (QuickTime video) run usability tests to certify that a product is (or in this case, isn’t) usable by a test audience. Others may run usability tests to learn when their product is “good enough” to release. But I think the best reason for running usability tests is to learn how to improve your product.
In the first two cases, you will probably want to obtain more quantitative than qualitative results. Would you consider your product usable if more than 50% of the users can perform some task within a certain period of time? What metrics do you choose to determine when to stop “fixing” your product and release it?
But in the third case, the qualitative results will be more abundant than the quantitative results. How can you make the user’s experience more enjoyable? How can you help them feel more productive? While there are quantitative underpinnings in the methods and techniques you can employ to improve the user’s experience, much of the measurement of the experience will be qualitative. This is not a Bad Thing.
Software Development Process
Parkhurst recently presented a slide showing his idea of the development process of an e-commerce website (see below).
I think there are some problems with this process. I propose that there should be Designers between Clients and Developers.
The designers would work with the clients to determine their needs and the needs of the end-users of the product. The designers would also be responsible for determine who the target audience is and would design the product features for that audience.
Specific Counter-arguments
Step 1: Who are your users?
Parkhurst argues that it’s difficult to determine who the users of a product are and that testing the wrong users would prevent you from generalizing the test results to the general audience of the product. I agree with this so far. I disagree, however, in his thinking that this should slow down or prevent usability testing. As I outlined in the software development process above, the audience should have been prior to the developers writing the software—during the design process. By the time you have software written for users to test, you should already know who the target audience is.
Step 2: What if your product functions don’t map onto user needs?
If you’ve got a good handle on your target audience and you’ve designed with them in mind, then this shouldn’t be too big of an issue. Of course, there will be some problems—after all, that’s why we’re doing the testing!
If the product functions don’t map onto the user needs, then we’ve either designed the software incorrectly or programmed it contrary to design specs.
Step 3: There is an inherent conflict between “friends and family” and reality.
This is sometimes (but not always) true. If your friends and family are within your target audience, then they are viable test participants. They may not verbally criticize your software as much as disinterested parties, but you shouldn’t be relying solely (or even primarily) on verbalization. Most users aren’t accustomed to thinking out loud as they use software, so any information you gain from this will be qualitative anyway. The only quantitative information you’ll obtain from the test will be based on the user’s actions.
Step 4: The laboratory setting threatens ecological validity.
True enough. But ecological validity isn’t necessary to the overall validity of the experiment. If it were, then we’d have to throw out most of the quantitative research that we’re basing our design decisions on.
An example: Parkhurst does a lot of work with gaze tracking and gaze detection. Most people don’t sit at home with a gaze tracking strapped to their head. Are the results that Parkhurst obtains from these experiments invalid?
Conclusion
Many of the problems mentioned above should be addressed in the design phase of the project and the solutions should be established well before the user testing phase begins.