Skip to main content

Systematic Reviews: Appraising Studies

High-quality systematic reviews are designed to identify all relevant published and unpublished evidence and provide an unbiased synthesis of the findings.

Appraising Studies

The results of a study are not worth very much if the study was not conducted very well.  You need to evaluate the quality of each study you include in your systematic review. This critical appraisal is often called "quality assessment" or "risk of bias assessment."  Your evaluations are shown in a table format in your article. There are several tools to help you evaluate various types of studies and create a table.  

Evaluation Tools for Controlled Trials

Cochrane's ROB 2 Risk of Bias Tool evaluates randomized controlled trials on seven items: 

  • Random sequence generation
  • Allocation concealment
  • Blinding of participants and personnel
  • Blinding of outcome assessment
  • Incomplete outcome data
  • Selective reporting
  • Other possible sources of bias

For more information on risk of bias, see the Cochrane Handbook, Chapter 8: Assessing risk of bias in included studies. 

Jadad Scaleassigns trials a score of 0-5 based on the quality of randomisation, blinding, and withdrawls

Robvis: a free online tool to create high-quality tables that summarize the risk of bias across studies with red, green, and yellow circles. 

Evaluation Tools for Non-Controlled Trials

CASP appraises systematic reviews, randomized controlled trials, cohort, case control, diagnostic,  economic evaluations, diagnostic studies, qualitative studies, and clinical prediction rules.

Cochrane's ROBINS-I tool evaluates cohort studies, quasi-randomized trials, case-control studies, cross-sectional studies, interrupted time series, and controlled before-after studies. 

Newcastle-Ottawa Scale:   appraises studies on 3 factors: the selection of the study groups; the comparability of the groups; and the ascertainment of either the exposure or outcome of interest for case-control or cohort studies respectively. 

NIH Study Quality Assessment Tools evaluate controlled intervention studies, systematic reviews, meta-analyses, observational cohort and cross-sectional studies, case-control studies, and before-after (pre-post) studies with no control groups.

JBI Critical Appraisal Tools: evaluate cross-sectional, case control, case reports, case series, cohort, diagnostic test accuracy, economic evaluations, economic, prevalence, qualitative, quasi-experimental, and systematic reviews. 

Guidelines for Assessing Studies

Assessing the Risk of Bias in Systematic Reviews of Health Care Interventions (from AHRQ Methods Guide for Comparative Effectiveness Reviews)