In randomized crossover designs, intraclass correlation coefficients (ICCs) are often used to assess the concordance between scores on different administration versions of patient-reported outcome (PRO) measures. An ICC and its associated criterion for “adequate” concordance enable analysts to simplify information and provide researchers with a quick and easy way to interpret analysis output. This strength of the ICC—its simplicity—may also be a weakness. Analysts may overlook important information (e.g., biases, outliers) when ICCs are used as the primary method for assessing concordance. One way to avoid overlooking important information is to include the evaluation of Bland-Altman plots when assessing concordance. Bland-Altman plots allow one to visually determine whether two measures produce similar scores, therefore, supplementing the concordance information gained from the ICC evaluation. ICCs and Bland-Altman plots complement each other’s strengths. ICCs provide an efficient and concise estimate to determine the comparability of versions, while Bland-Altman plots provide a greater level of detail that incorporates a broader view of the analyzed distributions. The use of the two methods together provides a more holistic view of concordance. We present Bland-Altman plots and corresponding ICCs under a randomized crossover-design, using the Lung Function Questionnaire, a PRO instrument originally designed to be administered via paper, and later via three alternate administration versions (Web, interactive voice response system, and interview). We provide examples to illustrate instances in which ICCs and Bland- Altman plots agree and disagree. GSK study number: ADC001HO.
