Systematic reviews

We recommend reading the following information about this particular type of research design before contacting us.

Staff, researchers and postgraduate research students who are conducting a systematic review can contact us at [email protected] to consult with a member of the team. We will help you understand the process, provide insight into developing a search strategy and also advise on selecting bibliographic databases and other resources for a comprehensive literature search.

“A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making.” Source: Cochrane Library (2019). About Cochrane Reviews | Cochrane Library. [online] Cochranelibrary.com. Available at: https://www.cochranelibrary.com/about/about-cochrane-reviews.

Further reading:

Doing a systematic review : a student’s guide | Edge Hill University (worldcat.org)

An introduction to systematic reviews

Systematic reviews primarily aim to facilitate well-informed decision-making with minimal bias by enhancing the quality of evidence. The need for systematic reviews has emerged because for most empirical questions the relevant literature is extensive, and clinicians and policymakers cannot spare the time nor have the skills to retrieve all the pertinent literature. They also cannot afford to appraise and synthesise every piece of evidence to identify the most suitable literature which will be “the all-things-considered” answer to a research question.

Without systematic reviews, decisions are likely to be based on a subset of publications which may not be representative of the whole literature, and there will be a risk that the reviewers will consciously or unconsciously select those publications which best support their views. Systematic reviews minimise bias by using techniques and presenting the literature synthesis in a format accessible to decision-makers. Unsurprisingly, systematic reviews are used increasingly to make clinical decisions, write clinical guidelines, and set research agendas.

Source: SOFAER, N. and STRECH, D. (2011). THE NEED FOR SYSTEMATIC REVIEWS OF REASONS. Bioethics, 26(6), pp.315–328. doi:10.1111/j.1467-8519.2011.01858.x. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3458717/#:~:text=The%20primary%20purpose%20of%20the,the%20relevant%20literature%20is%20extensive

Scoping review

This type of review delivers an initial assessment of the potential size and scope of available research literature. It focuses on identifying the nature and extent of research evidence (usually including ongoing research).

Rapid review or Rapid evidence assessments

An assessment of what is already known about a policy or practice by using systematic review methods to search, locate and critically appraise existing research.

Meta-analysis

A special technique that statistically combines the results of quantitative studies to provide a more precise effect on the results.

Mixed methods/mixed studies

Refers to any combination of methods where one significant component is a literature review (usually systematic). Within a review context, it refers to a combination of review approaches. For example, combining quantitative with qualitative research or outcome with process studies.

Source: Grant, M.J. and Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2), pp.91–108.

Not sure if you should conduct a systematic review or a scoping review? This article could help your decision-making. 

Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach.

1. Formulate Question
2. Write and Register Protocol (note to students: please check with your supervisor for registration)
3. Run Search
4. Select Eligible Studies
5. Assess Quality of Eligible Studies
6. Extract Data
7. Synthesise Results
8. Interpret and Report Findings

To see if a review on your topic is currently in progress or has already been published by someone else, use the resources listed below. If a review already exists, check its currency and quality. Does it need updating or is it of poor quality? If so, consider whether a new systematic review should be performed.

Cochrane Library

MEDLINE

CINAHL COMPLETE

PsychINFO

PROSPERO

TRIP

A systematic review protocol describes the review’s rationale, hypothesis, and planned methods. It should be prepared before a review is started and used as a guide to carry out the review. The protocol should be registered and publicly available (note to students: please check with your supervisor for registration).

Refer to PRISMA statement for more information:

Detailed protocols should be developed theoretically, made available in the public domain, and registered in a registry such as PROSPERO.

Registration provides transparency in the review process. It helps counter publication bias by providing a permanent record of prospectively registered reviews, irrespective of whether they are eventually published or not. It helps safeguard against reporting biases by revealing any differences between the methods or outcomes reported in the published review and those planned in the registered protocol. This should improve quality and increase confidence that policy or practice informed by the findings of a systematic review is drawing on best-quality evidence.

Registration allows those commissioning or planning reviews to identify whether there are any reviews already underway that address their topic of interest. This helps avoid unintended and economically wasteful duplication of effort.

PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health-related outcome. Key features from the review protocol are recorded and maintained as a permanent record. PROSPERO aims to provide a comprehensive listing of systematic reviews registered at inception to help avoid duplication and reduce opportunities for reporting bias by enabling comparison of the completed review with what was planned in the protocol.

PROSPERO is produced by CRD and funded by the National Institute for Health Research (NIHR).

Allocate a few hours for the registration process. Registering your protocol is free of charge, editing it is allowed, however PROSPERO system keeps a record of all those changes.

Your protocol is a proposal of the review you are planning to perform and describes how you are going to go do it.

Your protocol needs to include the following sections:
(source: https://training.cochrane.org/handbook/current/chapter-ii)

Title

Protocol

• Background
• Objectives
• Methods
• Criteria for selecting studies for this review
• Search methods for identification of studies
• Data collection and analysis

Appendices

Information

• Authors
• Contributions of authors
• Sources of support
• Declarations of interest
• Acknowledgements

References

• Additional references

Figures and Tables

Full details of what should be included in your protocol can be found in this PRISMA checklists:

• PRISMA – 2020 checklist: recommended items to address in a systematic review protocol (PDF, 151 KB)
• PRISMA for Scoping Reviews

Some examples of previously published protocols:

• The impact of wait time on patient outcomes in knee and hip replacement surgery: a scoping review protocol
• Nutritional vitamin D supplementation and health-related outcomes in hemodialysis patients: a protocol for a systematic review and meta-analysis

Note to students: for exact protocol requirements please check with your supervisor.

our literature search will need to be more in-depth than prior literature searches you may have done to meet the requirements for systematic reviews. You will need to think of several synonyms for your search phrases and double-check that you are searching in each database’s title, abstract, and MeSH term fields. To ensure your search is as thorough and effective as possible, make sure you are correctly using the Boolean operators AND, OR, and NOT as well as another search syntax such as truncation and wildcards.

Formulating your search question using PICO:

Most systematic review search strategies use PICO to frame their search.

First, you need to formulate a clear question. This should generate search terms. There are models which may help you do this such as PICO:

P = POPULATION / PROBLEM (e.g., patient group, problem, condition, disease, gender, age etc.). – What’s the problem? Who’s affected?

I = INTERVENTION/EXPOSURE (e.g., treatment, exposure, diagnostic test, drug, procedure) – what is being done?

C = COMPARISON (not always applicable) – what’s the alternative, comparing your intervention with another treatment or test?

O = OUTCOME (e.g., reduced mortality, improved quality of life, length of stay, cost-effectiveness, complications). – What effect will it have? You may want to include a separate search for a setting if this is relevant, e.g., critical care, hospital, primary care, etc. If you want a particular study type. e.g., RCT, this can be added as a limit at the end of your search.

A question such as ‘Does the flu vaccine help reduce absenteeism in healthcare workers? could be broken down as follows:

It is not necessary to fill each column for the PICO model. If you have search terms in the first two columns, you can run a search.

The main thing to remember is you MUST break your question down into different concepts.

While PICO (Population, Intervention, Comparison, Outcome) is a commonly used framework for formulating research questions in systematic reviews, there are several other frameworks that can be used as alternatives or supplements. Here are a few additional question frameworks:

SPICE: This framework is commonly used in qualitative systematic reviews and stands for Setting, Perspective, Intervention/Phenomenon of Interest, Comparison, and Evaluation. It helps researchers focus on the contextual factors, perspectives, and experiences related to the phenomenon under investigation.

ECLIPSE: This framework is specifically designed for qualitative evidence synthesis and stands for Experts, Context, Location, Impact, Phenomenon of Interest, Setting, and Experience. It emphasizes capturing diverse perspectives and experiences related to the research question.

PICOT: This framework is similar to PICO but includes an additional element, Time. It stands for Population, Intervention, Comparison, Outcome, and Time. It is commonly used in healthcare research to account for the temporal dimension of the research question.

SPIDER: This framework is widely used for formulating research questions in qualitative systematic reviews and stands for Sample, Phenomenon of Interest, Design, Evaluation, and Research type. It helps researchers articulate the key elements of the question while considering the qualitative research design.

FAME: This framework is used specifically for research questions in evidence-based medicine and stands for Format, Action, Comparison, and Outcome. It focuses on intervention-based questions in clinical practice.

FINER: This framework is commonly used for research questions related to feasibility and stands for Feasible, Interesting, Novel, Ethical, and Relevant. It helps researchers assess the practicality and relevance of a research question.

Remember that these frameworks serve as guides and can be adapted based on the specific requirements of your systematic review or research study. Choosing a framework that aligns with your research objectives and allows for a clear and structured formulation of your question is essential.

Booth A, Noyes J, Flemming K, Moore G, Tunçalp Ö, Shakibazadeh E. Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Glob Health. 2019 Jan 25;4(Suppl 1):e001107. doi: 10.1136/bmjgh-2018-001107. PMID: 30775019; PMCID: PMC6350737.

Systematic review search strategies are more complex than those you are perhaps used to.

They frequently include many lines of search terms, a mixture of MeSH and free text terms, translating search syntax from one database to another, and use of advanced search syntax such as truncation (using *), proximity and wildcards. Please look at this guide by the University of Exeter Library for a detailed explanation of how to ‘decode’ search strategies in systematic reviews.

Electronic databases (embed URL for each database)

• AMED (access via an NHS Open Athens account)
• British Nursing Index (BNI) (access via an NHS Open Athens account)
• CENTRAL
• CINAHL
• ClinicalTrials.gov
• Cochrane Library
• EMBASE (access via an NHS Open Athens account)
• EMCARE (access via an NHS Open Athens account)
• HMIC (access via an NHS Open Athens account)
• Medline
• PsycINFO
• Scopus
• WHO ICTRP

More sources: Consult your course subject resources area for more search platforms.

Although most of the usual medical and healthcare databases are appropriate for use in systematic reviews, PubMed should be avoided and here is why:

1. PubMed lacks proximity operators search functionality, making it difficult to search consistently across several databases.
2. The major problem is that PubMed searches are not reproducible and transparent because PubMed uses machine learning algorithms working behind the scenes which are invisible to the searcher making transparency and reproducibility impossible which are both of key importance in scientific reporting and experiments. Without these present in the search strategy, a systematic review falls at the first hurdle when being critically appraised.
3. PubMed issue with PRISMA-S checklist was launched recently, outlining all the reporting requirements for literature searching in systematic reviews. On the PRISMA checklist item, 8 requires you to “include the search strategies for each database and information source, copied and pasted exactly as run”. Note ‘exactly as run’. This is not possible in PubMed, therefore a Medline search via EBSCO or another platform is suggested.

(Source: https://clinicallibrarian.wordpress.com/2021/05/10/writing-a-systematic-review-dont-use-pubmed/)

Hand searching involves scanning the content of pre-selected peer-reviewed biomedical journals, conference proceedings and abstracts. It is an important way of identifying very recent publications that have not yet been indexed by electronic bibliographic databases. Hand searching can also ensure the inclusion of all types of evidence including letters or commentaries, which may not be indexed by databases and therefore not captured as literature search results and/or remedy poor or inaccurate database indexing that can result in even the most carefully constructed search strategy, failing to identify those relevant studies. Choosing which journals to hand search can be achieved by analysing the results of the database searches to identify the journals that contain the largest number of relevant studies.

The term grey literature refers to information produced outside of conventional scholarly publication and dissemination routes and is frequently underrepresented in bibliographic databases; these include PhD theses, conference proceedings, reports, unpublished trial data, and more.

You may risk bias to your results if you leave out grey literature.

Types of grey literature:  

Grey literature sources recommended by Public Health England: Source: https://ukhsalibrary.koha-ptfs.co.uk/greylit/

After screening and including the articles which meet your inclusion criteria, it may be appropriate to ‘snowball’ your search involving screening the articles that cite your included papers.

You can search for the titles of each included article in Scopus and any listed citing article that meets your inclusion criteria should also be included in the review.

Getting started with Scopus toolkit

Note: Included studies found this way should be detailed in PRISMA diagram

After you have reviewed all the papers found in databases, including grey literature and decided which ones will be included in the review you can look through their reference lists for any articles in their reference lists which meet your inclusion criteria, those need to be included in your systematic review also.

Further reading:

Systematic approaches to a successful literature review

Note: Included studies found this way should be detailed in PRISMA diagram

• It is good practice to keep a record of what your search history (what you searched), and when.
• This will save time if you have a similar enquiry or need to re-run the search in future.
• It also shows the steps you took to find your information (for transparency).
• Most platforms allow you to save your search history permanently.

To remove the duplicates from your results you will need to use a reference manager such as RefWorks or EndNote which will perform the deduplication for you automatically.

At EHU you have access to RefWorks and EndNote.

The library can provide you with training for both software as well as two alternatives – Mendeley and Zotero.

Documenting database searches in a systematic review is important for transparency, reproducibility, and minimising bias. It allows others to replicate the search process, verify the accuracy of results and reduces the risk of selective reporting. Thorough documentation enhances the reliability and integrity of the review and supports evidence-informed decision-making.

To document the databases searched in a systematic review:

• Keep a detailed record of all databases and other sources searched.
• List the names of the databases searched.
• Record the search strategies used for each database, including search terms, operators, and limits.
• Note the date of each search.
• Include the complete search strings if applicable.
• Document any additional sources searched, such as reference lists or experts.
• Update the search log as the review progresses.
• Archive the search records for future reference and transparency.

PRISMA (the Preferred Reporting Items for Systematic Reviews and Meta-analyses) is ‘an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.’ The PRISMA checklist is a useful guideline of content that should be reported and included in the final write-up of your systematic review and will help when in the planning stages as well.

PRISMA-S is the reporting guideline for systematic review search methods.

Although PRISMA is widely used as a reporting guideline for systematic reviews in the UK, Canada and other countries, some researchers may choose to use other reporting guidelines depending on their specific research question and field of study.

In addition to PRISMA, some other reporting guidelines that are commonly used for systematic reviews include:

Cochrane Handbook for Systematic Reviews of Interventions: This is a comprehensive resource for conducting systematic reviews and meta-analyses of healthcare interventions, developed by the Cochrane Collaboration. Cochrane Handbooks can be found here. Please choose the appropriate link for your review type.

JBI Manual for Evidence Synthesis: The Joanna Briggs Institute (JBI) provides a variety of resources and tools for conducting systematic reviews in healthcare and social sciences, including their Manual for Evidence Synthesis.

ENTREQ (Enhancing Transparency in Reporting the Synthesis of Qualitative Research) Guidelines from the EQUATOR Network: These guidelines were developed to improve the transparency and reporting of systematic reviews of qualitative research.

To locate the right guideline, search for your type of review

It’s worth noting that there are many other reporting guidelines available for systematic reviews, and the choice of which to use may depend on factors such as the research question, study design, and field of study.

During this step, the review team needs to perform the laborious but essential first step of synthesis of the existing literature during which you will look to see if any papers meet the criteria for inclusion outlined in your protocol. Access this journal article which provides a practical set of best practice guidelines to help you carry out this essential step:

Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses.

To obtain the articles you wish to include for full-text screening, if immediate access is not available to you via the EHU library subscriptions, you will need to ask us to provide you with a copy by using this link: You want it, we get it.

The full text screening stage is a critical step in conducting a systematic review which follows “Title and abstract screening” step. It involves evaluating the full-text articles or documents that have been identified as potentially relevant during the initial screening phase. The purpose of this stage is to determine whether each article meets the predetermined inclusion criteria for the review.

To help you with this step you may utilise systematic review screening software such as Rayyan to help you screen your systematic review.

Rayyan is a web-based platform designed to streamline the screening process by facilitating reviewer collaboration. It allows for easy uploading and sharing of studies, enabling multiple reviewers to independently assess and categorise them based on inclusion or exclusion criteria. Rayyan also offers features such as automated deduplication, real-time progress tracking, and customisable screening forms. By utilising Rayyan, students can enhance the efficiency and accuracy of their systematic review screening process, ultimately saving time and improving the quality of their research.

Follow this link for a great user guide provided by Towson University.

For this step, you will need to assess the quality of the included citations and determine whether they are relevant, reliable, valid, or applicable.

Please explore the following 8 bite-sized critical appraisal programme modules by the NHS Knowledge For Healthcare Academy, designed to support you with understanding the different methods and tools to carry out critical appraisal

1. Introduction to critical appraisal.
2. Introduction to equity considerations in critical appraisal.
3. Introduction to critical appraisal of randomised controlled trials.
4. Introduction to interpreting results for critical appraisal.
5. Introduction to critical appraisal of systematic reviews.
6. Introduction to critical appraisal of qualitative studies.
7. Introduction to critical appraisal of diagnostic studies.
8. Introduction to critical appraisal tools.

For assessing the quality of the papers that meet the inclusion criteria you can choose a critical appraisal tool that are essentially questionnaires created to assist you to perform this process.

• CEBM
• CASP
• RoB 2
• AACODS (for grey literature)

Further reading:

The pocket guide to critical appraisal

Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies.

Reviewing Research Evidence for Nursing Practice : Systematic Reviews

Excel sheets (for manual extraction)

RevMan

Systematic reviews toolbox: search for tools which provide support for a specific: http://systematicreviewtools.com/

SPSS – You can access the licence key here.

What are forest plots

Watch this video to help you understand Meta-analysis.

Further reading:

Medical statistics made easy 3

Medical statistics: a textbook for the health sciences

A narrative of studying, evaluating, and synthesising the body of information in your review is known as a qualitative synthesis. All systematic reviews must include this type of synthesis.

In this section:

• Give a broad overview of the characteristics and conclusions of the studies that you included.
• Analyse the connections between research, looking for trends and diversity.
• discuss how the evidence collected relates to the review’s central question.
• If a meta-analysis was performed, you will need to describe, interpret and evaluate how reliable the results were.
• Discuss the advantages and disadvantages of the collected evidence as a whole, taking into account an overall evaluation of the risk of bias among them.
• Comment about any evidence gaps, such as patient groups that haven’t been properly researched or whose outcomes may vary.
• When applicable, compare the review’s conclusions with the prevailing opinion regarding the review’s question.

You may also opt for thematic analysis. For more information please read this article: Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8(1), 45-54. https://doi.org/10.1186/1471-2288-8-45

A quantitative synthesis, also known as a meta-analysis, combines and examines the findings of several research studies using statistical methods. A systematic review that combines studies, such as Randomised Controlled Trials (RCTs), with other types of research design should include meta-analyses. A meta-analysis is not present in all systematic reviews. Please refer to the Cochrane handbook for more information:

Chapter 10: Analysing data and undertaking meta-analyses

More useful resources:

Ten simple rules for carrying out and writing meta-analyses

Forero, D.A., Lopez-Leon, S., González-Giraldo, Y., Bagos, P.G., 2019. Ten simple rules for carrying out and writing meta-analyses. PLoS Comput. Biol. 15, e1006922. https://doi.org/10.1371/journal.pcbi.1006922

Quantitative synthesis of comparative studies.

Note: it is recommended to consult a statistician at this stage.

It is highly recommended to contact our Head of Research at the outset of your project to acquire best information you need on getting published.

Head of Research will be able to support you with the publishing process plus the options for publishing your work via Open Access.

Journal/Author Name Estimator (JANE) : https://jane.biosemantics.org/

Measuring Scholarly Impact on Scopus: Metrics – Scopus LibGuide – LibGuides at Elsevier