Once your team has agreed up on the studies to be included, the team will need to critically appraise each article and assess for risk of bias.
Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.
Critical appraisal methods address both the credibility and rate the confidence in the quality of the summarized evidence. The confidence level can be determined by assessing the risk of bias, inconsistency of results, imprecision, indirectness of evidence, publication bias, and so on.
For the purposes of a systematic review, critically appraising the literature is not meant to ultimately make your decision of inclusion, but moreso to provide guidance for the team in deciding whether the available evidence is credible and how confident one is with the results obtained.
Below you will find some tools and resources to help in your critical appraisal step.
The checklists available for critical appraisals are created based on study type. As such, it would be in your best interest to review different study design types.
This table is a chart of clinical questions as it relates to study designs. It does not cover all study designs, but it is still helpful for determining them for a large amount of studies.
| Clinical Relevance | Clinical Questions | Suggested Best Method of Investigation |
|---|---|---|
| Etiology or Causation | What caused the disorder and how is this related to the development of illness? |
Randomized controlled trial (RCT) Case-control study Cohort study |
| Therapy | Which treatments do more good than harm compared with an alternative treatment? |
RCT Systematic review Meta-analysis |
| Prognosis | What is the likley course of a patient's illness? What is the balance of the risks and benefits of a treatment? |
Cohort study Longitudinal study |
| Diagnosis | How valid and reliable is a diagnostic test? What does the test tell the doctor? |
Cohort study Case control study |
| Cost-effectiveness | Which intervention is worth prescribing? Is a newer treatment worth prescribing compared with the older treatment? | Economic analysis |
Source: Al-Jundi,A.Sakka,S.(2017).Critical Appraisal of Clinical Research,J Clin of Diagn Res. 11(5), JE01-JE05. https://www.doi.org/10.7860/JCDR/2017/26047/9942
Conducting a risk of bias assessment to identify design and other flaws and limitations in studies is an important step in the review process. Bias is unavoidable and it can occur in the design and methodology of a study which is why it is necessary to systematically check along with your team member to minimize bias.
A study sufficiently free from bias has what's known as internal validity and if the study can be generalized to the clinical or a wider population, than the study will be said to have external validity.
| Type of Reporting Bias | Definition |
|---|---|
| Publication bias | The publication or non-publication of research findings, depending on the nature and direction of results. |
| Time-lag bias | The rapid or delayed publication of research findings, depending on the nature and direction of the results. |
| Language bias | The publication of research findings in a particular language, depending on the nature and direction of the results. |
| Citation bias | The citation or non-citation of research findings, depending o the nature and direction of the results. |
| Multiple (duplicate) publication bias | The multiple or singular publication of research findings, depending on the nature and direction of the results. |
| Location bias | The publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results. |
| Selective (non-) reporting bias | The selective reporting of some outcomes or analyses, but not others, depending on the nature and direction of the results. |
| Funding bias | Funders may have a vested interest in demonstrating positive outcomes for one group/intervention. |
| Confounders | Were the participant characteristics, such as age, sex, or health status, similar across all treatments? Participants should be equally balanced in terms of variables considered important to study outcomes (e.g. sex, age, health status) otherwise there is a risk that results will be biased in favor of one group or intervention. |
| Analysis | Where all the data for all participants included in the final analysis (even those participants who withdrew?) If there are data missing for a number of participants and these are not accounted for, published results will not properly reflect the results of the study. |
Source: Boland, A., Cherry, M. G., & Dickson, R. (2017). Quality Assessment: Where Do I Begin? In Doing a systematic review: A student's guide. essay, Sage.
Wiley-Blackwell. (2019). Cochrane Handbook for Systematic Reviews of Interventions.
NOTE: Scoping reviews don't typically include a risk of bias assessment.
“A key difference between scoping reviews and systematic reviews is that the former are generally conducted to provide an overview of the existing evidence regardless of methodological quality or risk of bias (4, 5). Therefore, the included sources of evidence are typically not critically appraised for scoping reviews.”
Source: From Tricco, A., Lillie, E., Zarin, W., O'Brien, K., Colquhoun, H., Levac, D.,...Straus, S. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine, 169(7), 467-473. https://doi.org/10.7326/M18-0850.
| Factors Decreasing Quality Level | Factors Increasing Quality Level |
|---|---|
| Limitations in study design and implementation suggesting high likelihood of bias. | Large magnitude of effect (e.g. relative risk > 2 or < 0.5) in the absence of plausible confounders in observational studies. |
| Indirectness of evidence (indirect population, intervention, control, outcomes). | All plausible biases would reduce a demonstrated effect or suggest a spurious effect when results show no effect in observational or randomized studies. |
| Unexplained heterogeneity or inconsistency of results (including problems with subgroup analyses). | Dose-response gradient in observational studies. |
| Imprecision of results. | |
| High probablity of publication bias. |
Source: Patel JJ, Hill A, Lee ZY, Heyland DK, Stoppe C. Critical Appraisal of a Systematic Review: A Concise Review. Crit Care Med. 2022 Sep 1;50(9):1371-1379. doi: 10.1097/CCM.0000000000005602. Epub 2022 Jul 18. PMID: 35853198.
In addition to credit given for various images, parts of this guide were adapted from work/guides by:
New York University, University of Minnesota, Macquarie University
Used with permission or in accordance with Creative Commons Licensing.