The quality assessment of observational studies included in systematic reviews is key to the interpretation of those reviews. The large number of available tools and the lack of agreement on which domains to cover reflect the challenges involved in quality assessment. While no tool has been specifically designed for pharmacoepidemiologic studies, the Newcastle-Ottawa Scale (NOS) is most commonly used.
To compare the NOS with the recently proposed RTI item bank (RTI IB) and to validate the RTI IB.
This quality assessment was performed in the context of a systematic review of the cardiovascular safety of glucose-lowering drugs in patients with type 2 diabetes mellitus, within the European Commission-funded Safety Evaluation of Adverse Reactions in Diabetes (SAFEGUARD) project. We tailored both tools and added 4 questions to the RTI IB to assess aspects of pharmacoepidemiologic studies overlooked by both tools. Two reviewers assessed the quality of the 44 included studies along with the data extraction. The RTI IB was applied independently by the 2 reviewers, who agreed on which responses conveyed low, unclear or high risk of bias. For the 31 questions in the RTI IB, the observed agreement was calculated as the percentage of studies on which both reviewers gave the same bias assessment. Chance-adjusted inter-rater agreement was estimated with the first-order agreement coefficient (AC1) statistic.
While the NOS required less tailoring and was easier and faster to use than the RTI IB, the RTI IB resulted in a more thorough assessment. The RTI IB includes most of the questions in the NOS. The median observed agreement with the RTI IB was 77% (25th percentile-75th percentile: 61%-89%); the median AC1 statistic was 0.66 (0.51-0.88).
The RTI IB facilitates a more complete quality assessment than the NOS but is more burdensome. The observed agreement and AC1 statistic in our application of the RTI IB were higher than reported by the tool’s developers.
