Personnellement, j’ai une formation universitaire en psychologie où j’ai appris l’importance de la validité interne et externe des données recueillies notamment lors des tests. J’ai fait quelques bonds et poussé quelques profs de marketing dans leurs retranchements quand ils m’expliquaient « qu’on valide le quanti avec le quali » ou l’inverse (ça changeait suivant les profs). Tout cela pour vous dire que Jeff Sauro vient de publier un livre disponible en France : Quantifying the User Experience: Practical Statistics for User Research. Bien sur là, ça ne laisse plus la place à l’improvisation et pif au mètre !

Pour vous faire une idée du travail de Jeff Sauro, vous pouvez lire son blog et retrouver, sur son site, des outils statistiques bien pratiques.

Le sommaire du livre :

1. Introduction & How to Use this Book
1. Visual Guide to What Test
2. Skipping the formulas
2. Quantifying User Research
1. What is User Research?
2. Usability Tests (lab and remote)
1. Benchmarking
2. Comparative Testing
3. Qualitative Studies
3. Surveys
4. Requirements Gathering
5. A/B Testing
6. Questionnaires
7. Using Inferential Statistics with usability Data
8. Samples Size, Normality and other statistical concerns
9. Measuring Usability: Quantifiable Aspects of Usability
1. Introduction: Metrics as independent to formative and summative tests
1. Completion
2. Time
3. Satisfaction
4. Errors
5. Clicks / Page Views
6. Combined Scores
7. Problems Discovered
3. How precise are our estimates: Confidence Intervals
1. Confidence Interval = Twice the Margin of Error
2. Confidence Intervals Provide Precision & Location
3. Three Components of a Confidence Interval
1. Confidence Level
2. Variability
3. Sample Size
4. Confidence Interval for a Completion Rate
1. Confidence Interval History
2. Wald Interval: terribly inaccurate for small samples
3. Exact Confidence Interval
4. Adjusted-Wald: Add Two Successes & Two Failures
1. Best Point Estimates for a Completion Rate
1. How accurate are point estimates from small samples?
5. Confidence Interval for a Problem Occurrence
6. Confidence Interval for Rating Scales and other Continuous Data
7. Confidence Interval for Task Time Data
1. Mean or Median Task Time?
2. The Geometric Mean
8. Log Transforming Confidence Intervals for Task Time Data
9. Confidence Interval for a Median
4. Did we meet or exceed our goal?
1. Introduction
1. One-Tailed and Two-Tailed Tests
2. Comparing a Completion Rate to a Benchmark
1. Small Sample Test
1. Mid-Probability
2. Large Sample Test
3. Comparing a Satisfaction Score to a Benchmark
1. Do at Least 75% Agree? Converting Continuous Ratings to Discrete
1. Disadvantages to Converting Continuous Ratings to Discrete
2. Net Promoter Score
4. Comparing a Task Time to a Benchmark
5. Is there a statistical difference between products?
1. Comparing two Means (Rating Scales & Task Times)
1. 2-sample t-test (between subjects)
1. Confidence Interval around the Difference
2. Paired t-test (within subjects)
1. Confidence Interval around the Difference
2. Comparing Completion Rates
1. Small Samples : Fisher Exact Test
2. Large-Samples : The N-1 2-proportion test
1. Confidence Interval around the Difference
2. Relationship between Chi-Square Tests and 2-proportion tests
3. A/B Testing & Conversion Rates
6. What Sample Sizes Do We Need?  Part 1: Summative Usability Studies
1. Introduction
1. Why Do We Care?
2. The Type of Usability Study Matters
1. Basic Principles of Summative Sample Size Estimation
2. Estimating Values
1. Example 1: A Realistic Usability Testing Example Given Estimate of Variability
2. Example 2: An Unrealistic Usability Testing Example
3. Example 3: No Estimate of Variability
3. Comparing Values
1. Example 4: Comparison with a Benchmark
2. Example 5: Within-Subjects Comparison of an Alternative
3. Example 6: Between-Subjects Comparison of an Alternative
4. Example 7: Where’s the Power?
4. What Can I Do to Control Variability
1. Sample Size Estimation for Binomial Confidence Intervals
2. Binomial Sample Size Estimation for Small Samples
3. Sample Size for Comparison with a Benchmark Proportion
4. Sample Size Estimation for Proportions & Chi-Squared Tests
7. What Sample Sizes Do We Need?  Part 2 :  Problem Discovery
1. Using a Probabilistic Model of Problem Discovery to Estimate Sample Sizes for Formative User Research
2. The famous equation (P(x ≥ 1) = 1 – (1 – p)n
1. Deriving a sample size estimation equation from 1 – (1 – p)n
2. Using the tables to plan sample sizes for formative user research
3. Assumptions of the Binomial Probability Model
4. Additional Applications of the Model
1. Estimating the composite value of p for multiple problems or other events
2. Adjusting small-sample composite estimates of p
5. Estimating p
1. Adjusting the Initial Estimate of p
2. Using the Adjusted Estimate of p
3. Investigating Sample Size Effectiveness
4. Estimating the Number of Problems Available for Discovery
5. What Affects the Value of p?
8. Attitudinal Measurement with Questionnaires
1. Scales, Labels and Points
2. Post-Task Questionnaires
3. ASQ, SMEQ, 1-question Likert
4. Post-Test
5. SUS, SUMI, PSSUQ, Homegrown scales
6. Usability and Loyalty
7. Net Promoter Scores and SUS
9. Controversies in Measurement & Statistics
1. Industrial versus Scientific: Purpose of statistics is to help in better decision making over the long run
2. Multi-Point Scales
3. p-values and NHST
4. Parametric versus Non-Parametric Statistics
5. Which confidence level
6. When x=n or x=0 what confidence level do you use?
7. Multiple testing versus omnibus testing
8. 2 x 2 tables
10. Final Thoughts on Statistics for User Research
11. Appendix A:  A Crash Course in Fundamental Statistical Concepts
1. Central Tendency: Mean & Median
2. Standard Deviation & Variance
3. Population Parameters and Sample Statistics
4. Standard Deviation
5. Margin of Error
6. Alpha
7. Standard Error of the Mean
8. Central Limit Theorem
9. The normal distribution
10. The Binomial Distribution
11. Normal Approximation to the Binomial
12. Introduction to Hypothesis Testing
1. The Null and Alternative Hypothesis (Ho and Ha)
2. Type I and Type II Errors
3. Confidence and Power
4. Making decisions from p-values
1. If p is low reject the Ho
5. One and Two Tailed Tests
13. Mechanics of Test Statistics
1. z statistics
2. t-statistics