Empowering the Future of Evidence-Based Healthcare: The Cochrane Early Career Professionals Network

BACKGROUND Historically, peer reviewing has focused on the importance of research questions/hypotheses, appropriateness of research methods, risk of bias, and quality of writing. Until recently, the issues related to trustworthiness - including but not limited to plagiarism and fraud - have been largely neglected because of lack of awareness and lack of adequate tools/training. We set out to identify all relevant papers that have tackled the issue of trustworthiness assessment to identify key domains that have been suggested as an integral part of any such assessment. METHODS We searched the literature for publications of tools, checklists or methods used or proposed for the assessment of trustworthiness of randomised trials. Data items (questions) were extracted from the included publications and transcribed on Excel including the domain of assessment as described in the original publication. Both authors then independently assessed each data item to see if the original domain(s) could be re-categorised in 5 domains (governance, plausibility, plagiarism, reporting, statistics). RESULTS From the 41 publications we extracted a total of 284 questions and framed 77 summary questions grouped in 5 domains: governance (13 questions); plausibility (16 questions); plagiarism (4 questions), reporting (28 questions and statistics (16 questions). CONCLUSION The proposed menu of domains and questions should encourage peer reviewers, editors, systematic reviewers and developers of guidelines to engage in a more formal trustworthiness assessment. Methodologists should aim to to identify the domains and questions that should be considered mandatory, those that are optional depending on the resources available, and those that could be discarded because of lack of discriminatory power.

The use of visual methods in qualitative evidence synthesis (QES) adds a valuable dimension to the synthesis process by enhancing understanding and knowledge generation. This paper provides an overview of common visual methods that support various stages of QES, with a worked example from a Cochrane QES using meta-ethnography. The paper also addresses the involvement of stakeholders, and considerations of equity, diversity, inclusion, and reflexivity in the selection and application of visual methods. The authors utilised a combination of remote and in-person visual methods to initiate and develop their synthesis, involving stakeholders throughout the process. Methods used by the review authors included paper labels, cartoons, infographics, virtual whiteboards, and diagrams. The use of visual methods in the QES facilitated data visualisation, remote analysis meetings, interpretation of extensive data, and meaningful patient and public involvement. QES authors are encouraged to consider the use of visual methods, particularly when involving the public in the synthesis process. Keywords: Qualitative Evidence Synthesis, QES, Meta-ethnography, visual methods

Introduction Cochrane Rehabilitation recently developed a new definition of rehabilitation for research purposes with 80 global stakeholders, aiming to support and improve the production and reporting of primary and evidence synthesis rehabilitation studies. Objective 1. To compare how Cochrane Systematic Review (CSR) authors describe rehabilitation interventions against criteria derived from the new rehabilitation definition, and 2. To assess limitations or gaps in the rehabilitation definition. Methods: We analysed a sample of 124 randomly-selected CSRs tagged in the Cochrane Rehabilitation database. We converted the Cochrane Rehabilitation definition for research purposes into a set of 13 criteria grouped in the four PICO elements and searched for the corresponding key elements in each CSR. We verified if and where in the review these elements were present. Two reviewers rated each CSR, resolving disagreements a third author when needed. We analysed the findings using descriptive statistics. Results Eight (6.5%) out of 124 CSRs met all rehabilitation definition criteria. These were CSRs that investigated the effects of complex rehabilitation intervention. Three (2.4%) CSRs did not meet any PICO elements. Overall, the ”Intervention-General” element and disability criterion had the highest prevalence of absent and unclear reporting, while the “Intervention-Specific” and “Outcome” elements were most frequently reported, albeit not in the “Description of the intervention” section of the review. Discussion This study showed that the key elements of the new rehabilitation definition are almost always reported in publications identified as rehabilitation review, but not always consistently or clearly. The disability criterion was frequently unreported given that the main aim of rehabilitation is reducing disability. Also, the main elements of rehabilitation were frequently not reported. We did not find important gaps in the new definition. All elements of the new definition should be considered when writing review protocols and designing strategies and tools on rehabilitation topics.

The 2022 Cochrane Lecture challenged Cochrane and its community to enhance their equity efforts and presented specific issues needing attention. We have created an actionable plan to address these challenges. These include: 1. Setting priorities for reviews based on health equity and global burden of disease 2. Advocate and support initiatives to promote equity, diversity, and inclusion in Cochrane 3. Actively seek global representation in the governance of the Campbell and Cochrane Health Equity Thematic Group We advocate for the inclusion of health equity considerations in all Cochrane systematic reviews. To achieve this, we need the cooperation of all those involved, such as journal editors, funders, and researchers. We invite the global community to work with us to address these equity gaps in research and practice.

Introduction Qualitative evidence is increasingly incorporated into decision-making processes. Assessing the methodological limitations of primary studies is critical to making an overall assessment of confidence in findings from qualitative evidence syntheses (QES) using GRADE-CERQual. Current critical appraisal tools were not developed specifically for use in Cochrane reviews or GRADE-CERQual, and few are evidence-based. The aim of CochrAne qualitative Methodological LimitatiOns Tool (CAMELOT) was to address this gap. Methods We undertook this project in four stages: (1) systematic literature search to identify existing tools, (2) identification of evidence to support inclusion of potential CAMELOT domains (3) consensus survey to agree on the inclusion and definition of CAMELOT domains, and (4) human-centred design approach to develop and refine CAMELOT by exploring user experience. Results CAMELOT is a new evidence-based tool for assessing the methodological strengths and limitations of primary qualitative research studies in a QES. CAMELOT is comprised of twelve domains: four Meta domains that encourage review authors to consider those characteristics of the primary study that are beyond how the study was carried out, but which inform the conduct and design of the study, and eight Method domains which encourage review authors to consider how the study was designed, planned and/or conducted, and how study conduct and design fits with the information provided in the four Meta domains. Review authors make an assessment by identifying any concerns regarding the methods used in the study and considering the appropriateness of fit between the Meta and Method domains. Conclusion CAMELOT provides review authors with a transparent and systematic method to assess methodological limitations of primary qualitative studies. CAMELOT incorporates qualitative principles and focuses on appropriateness of fit between Meta and Method domains. In line with iterative tool development approach, CAMELOT will continue to be revised over time following extensive user testing and piloting.

This tutorial focuses on standardised mean differences (SMD) as effect measures in meta-analyses. We will explain what they are, when they should be used, how to correctly compute and interpret them, and some of the most common error made within evidence synthesis

Australia’s national living guidelines for COVID-19 were launched in April 2020 and include over 200 recommendations to guide the clinical care of patients with COVID-19. Until the guidelines were retired in August 2023, new evidence was continually monitored through daily surveillance. This paper describes how evidence surveillance was established (at a time when efforts to collate information on the novel coronavirus were in their infancy and often duplicative) and how it evolved throughout the pandemic. Among the wide range of sources monitored, the collections of COVID-19 research from leading medical journals, medRxiv preprints and PubMed auto alerts proved the most enduring in identifying time-critical and impactful evidence. The paper also explores how evidence was tracked and surveillance integrated into the overall evidence workflow by using messaging apps and communication platforms. Several factors contributed to the feasibility of daily surveillance, including the clearly defined scope of the guidelines; focusing efforts on maximum impact; minimising duplication by partnering with others; setting up simple but effective processes for managing evidence; and the value of continuous involvement of personnel from the outset. The example of Australia’s living COVID-19 guidelines has demonstrated the feasibility and essential role of daily evidence surveillance in maintaining COVID-19 living guidelines for over 3 years.

This paper introduces version one of an assessment tool developed to address the challenges posed by the assessment of data thickness and richness in primary qualitative studies for Qualitative Evidence Syntheses (QES). The tool has been in development since 2014. Three pilot versions from three review teams have been used in six Cochrane reviews. Key members from the original three review teams came together to create a consensus-based definitive version 1 of the tool for publication. Four review authors piloted the version 1 tool. The definitive version 1 assessment tool consists of two components: assessing the thickness of contextual data and assessing the richness of conceptual data. A sliding scale with four points is used to rate these aspects, offering nuanced and qualitative judgments. The accompanying guidance emphasizes the importance of assessing data that addresses the review question. Paragraph locked by Heather Melanie R Ames The paper provides guidance on how to apply the tool, emphasizing the importance of reaching a consensus among review authors, and fostering a shared understanding of what constitutes rich and thick data in the context of the review. The potential challenges related to the time and resource constraints of this additional review process are acknowledged. Version 1 of the data thickness/richness assessment tool represents a significant development in QES methodology, filling a critical gap in tools for evaluating the richness of conceptual data and the level of contextual detail in primary qualitative studies. It enhances the transparency and rigor of the sampling process and offers valuable insights for assessing the thickness and richness of data in primary qualitative studies that addresses the review requestion, objectives and context as specified in the review protocol. The authors invite feedback from the research community to further test, refine and improve this tool based on wider user experiences.

BACKGROUND There is increasing concern that a significant proportion of randomised controlled trials (RCTs) included in Cochrane reviews may not be trustworthy. Applying a trustworthiness screening tool (TST) has already had a clinically important effect on several reviews published by the Cochrane Pregnancy and Childbirth Group. OBJECTIVES We wanted to assess the impact of removing untrustworthy RCTs from already- published Cochrane reviews on a defined clinical area (ante- and post-natal nutritional interventions). METHODS We applied the tool to 18 Cochrane reviews (375 RCTs). The tool had four domains: i) is the research governance trustworthy; ii) are the baseline characteristics trustworthy; iii) is the study feasible; iv) are the results plausible?). When additional information was needed, authors were contacted using a standard template. At least two attempts were made to contact the authors. At the end of the evaluation process each study was classified as: i) included (YES to all domains); ii) excluded (retracted study); or iii) awaiting classification (any NO to the TST questions). RESULTS 95/375 studies (25%) were removed, affecting 14/18 (78%) reviews. 13/18 reviews (72%) showed a difference in the Summary of Findings tables (direction and size of effects and/or GRADE ratings). 6/18 Cochrane reviews (33%) were judged to require updating because of important differences in either in their conclusions, implication for practice, and/or implication for research. CONCLUSIONS Formal assessment of trustworthiness and inclusion only of studies that satisfy prespecified criteria for trustworthiness affect conclusions in a relatively large number of Cochrane reviews, with potentially important clinical implications for practice and research. The lack of consensus regarding the best tool(s) for assessing trustworthiness cannot be an excuse for ignoring this issue in future Cochrane reviews.

Introduction Skin cancer affecting the nail unit is rare but is associated with morbidity, and melanoma has a high mortality rate. The principal treatment is surgical excision and methods can be classified into digit-sparing surgery or amputation. Digit-sparing surgery (wide excision or Mohs surgery) may be safe and effective for malignancies involving the nail unit in comparison to amputation if there is not bony invasion. The objective was to assess the efficacy and safety of different methods of surgical excision for skin cancer involving the nail unit. Methods Prospective comparative studies (randomised controlled, non-randomised controlled and prospective observational studies) of surgical excision for skin cancer of the nail unit in all participants were eligible for inclusion. We searched electronic databases, trials registers and conference abstracts. We checked the reference lists of included studies and related systematic reviews for further references to relevant studies, and we contacted experts to enquire if they were aware of any additional relevant trials. We used standard methodological procedures expected by Cochrane. The primary outcomes were overall survival, disease free survival and adverse events/outcomes at 30 days. The secondary outcomes were quality of life outcomes. We planned to use GRADE to assess the quality of the evidence for each outcome. Results We did not identify any studies that met the inclusion criteria for this review. We have been unable to assess our outcomes of overall survival, disease free survival, adverse events/effects and quality of life. Conclusions As we have not identified any studies for inclusion, we are unable to assess the efficacy and safety of different methods of surgical excision for skin cancer involving the nail unit. Prospective research, ideally in the form of a randomised trial, is required in this field. Registration Cochrane Database of Systematic Reviews 2021, Issue 5. Art. No.: CD014590.