A comparison of current approaches to usability measurement
Dr J Kirakowski & Dr R Murphy 2009
link
This paper looks at how 15 different organisations evaluate percieved usability. The results are centered around presentations made each other at the Usability Professionals' Association (UPA) workshop in 2009. This a preliminary paper with a follow on planned.
The Comparative Usability Evaluation (CUE) seminars are a series of occasional seminars organised by Prof. Rolf Molich and colleagues at the UPA conferences since 1998. The first CUE result suggested that some usability experts found the use of statistical data difficult. The latest CUE seminar(the 8th)is more concerned with measurement methods of usability, where when a website is redone it can compared to the measurement of its previous incarnation.
The requirements were fairly strict in that were five tasks to be carried out by end users on the web site. For each task, the following measures had to be taken:
1. Time on task (efficiency)
2. Success (effectiveness)
The results were divided into different sections:
1. Context of Use
The ISO 9241 definition of usability states that usability may only be meaningfully measured in context. For most the users demographic data was not gathered, and for some no knowledge of participants were known. Six groups used a remote unattended methodology by which it is meant that users interacted with the web site at home on their own computers, onto which a test suite had been downloaded. Some used naturalistic setting and some a laboratory setting
2. Sample Size
the largest was 314. Smallest was 9. Average was 20.
3. Effectiveness
Some measured effectiveness in boolean style while others. Measured success as a percentage.
4. Efficiency
Most common measurement of efficiancy was a time task. A ping methodology was also used.
5. Satisfaction
Questionaires (SUS) were used by many to evaluate. However this data shows that SUS is strongly skewed towards the high numbers when applied to web sites and although its range is transformed to the interval {0:100} only 40 discriminations are possible.A false sense of optimism can be presented from the WAMMI questionnaire. Usability score. This corresponds well with user comments as reported by all the groups.

All the mainstream measures reviewed in this paper have exhibited major conceptual flaws in terms of measurement. We also recommend that either more work be done to create a well-validated questionnaire for user satisfaction with web sites that is public domain. The way data is gathered by remote unattended testing methodologies at present is not controlled in terms of precision and we recommend that a greater emphasis be placed in remote unattended testing on data verification.