Evaluating the Consistency and Quality of Search Strategies and Methodology in Cochrane Urology Group Systematic Reviews
Jennifer A. Lyon, Carrie Price, Jamie Saragossi, and Clara Y. Tran
The Cochrane Urology Working Group has undergone multiple administrative and personnel changes over its history. Presently, there are nearly a dozen Information Specialists volunteering from several institutions and two countries. To identify areas for quality and consistency improvement, we evaluated the search methodologies in 64 Urology Group Reviews/Protocols.
64 published Urology Group Cochrane Systematic Reviews (CSR) and Protocols (CSP) were downloaded from the Cochrane Library on September 15, 2016. A data form was created using Qualtrics consisting of several sections including expanded PRESS (Peer-Review of Electronic Search Strategies) elements, database selection, search methodology reporting, & consistency. Four librarians, including two Cochrane Urology Assistant Information Specialists, evaluated 20 CSR/Ps each, allowing overlap to verify data extraction reliability.
Librarian reviewers noted high variability in CSR/P search methods including selection of databases searched, full search strategies reporting, and use of publication filters, clinical trial registries, conference proceedings, and journal TOCs. Within search strategies, common errors included wrong line numbers, misspellings, inappropriate syntax, and duplicated lines.
Cochrane Reviews are frequently viewed as the "gold standard" of systematic reviews, therefore the CSR/Ps should have high quality search methodology. This retrospective analysis represents a historical overview of search method reporting in CSR/Ps from the Cochrane Urology Group. Our results demonstrate a need for improved standardization including complete documentation of all strategies, more consistent database selection, less variation in strategy construction, increased use of peer-review and inclusion of the PRISMA Flow Diagram, and tighter adherence to the Cochrane Handbook/MECIR Standards.