Reviewing the reviews: How strong is the evidence? How clear are the conclusions?

The objectives of this paper were: a) to determine what can be learned from conclusions of systematic reviews about the evidence base of medicine; and b) to determine whether two readers draw similar conclusions from the same review, and whether these match the authors' conclusions. Three metho...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International journal of technology assessment in health care 2001, Vol.17 (4), p.457-466
Hauptverfasser: EZZO, Jeannette, BAUSELL, Barker, MOERMAN, Daniel E, BERMAN, Brian, HADHAZY, Victoria
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 466
container_issue 4
container_start_page 457
container_title International journal of technology assessment in health care
container_volume 17
creator EZZO, Jeannette
BAUSELL, Barker
MOERMAN, Daniel E
BERMAN, Brian
HADHAZY, Victoria
description The objectives of this paper were: a) to determine what can be learned from conclusions of systematic reviews about the evidence base of medicine; and b) to determine whether two readers draw similar conclusions from the same review, and whether these match the authors' conclusions. Three methodologists (two per review) rated 160 Cochrane systematic reviews (issue 1, 1998) using pre-established conclusion categories. Disagreements were resolved by discussion to arrive at a consensual score for each review. Reviews' authors were asked to use the same categories to designate the intended conclusion. Interrater agreements were calculated. Interrater agreement between two readers was 0.68 and 0.72, and between readers and authors, 0.32. The largest categories assigned by methodologists were "positive effect" (22.5%), "insufficient evidence" (21.3%), and "evidence of no effect" (20.0%). The largest categories assigned by authors were "insufficient evidence" (32.4%), "possibly positive" (28.6%), and "positive effect" (26.7%). The number of reviews indicating that the modern biomedical interventions show either no effect or insufficient evidence is surprisingly high. Interrater disagreements suggest a surprising degree of subjective interpretation involved in systematic reviews. Where patterns of disagreement emerged between authors and readers, authors tended to be more optimistic in their conclusions than the readers. Policy implications are discussed.
doi_str_mv 10.1017/s0266462301107014
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_miscellaneous_72333812</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>72333812</sourcerecordid><originalsourceid>FETCH-LOGICAL-c221t-a99f21f533b380680f82804ae69655fe3f210c427590c0f02a36f5091c5c62c53</originalsourceid><addsrcrecordid>eNpdkN9LwzAQx4Mobk7_AF-kCPpWvUvapPVlyFAnDMRfzyXLEu3ompm0Dv97s24w2NOR-3zucnwJOUe4QUBx64FynnDKABEEYHJA-pgIjDlLskPSX-N4zXvkxPs5ADLI4Zj0EEWa0Rz65PVN_5Z6VdZfUfOtI9e9_F00tqvIN86Gfuk7FMhM10oPO6YqLV0kne6YsrWqWl_a2g9PyZGRlddn2zogn48PH6NxPHl5eh7dT2JFKTaxzHND0aSMTVkGPAOT0QwSqXnO09RoFiiohIo0BwUGqGTcpJCjShWnKmUDcr3Zu3T2p9W-KRalV7qqZK1t6wtBGWMZ0iBe7olz27o63FaEL5jgIbkg4UZSznrvtCmWrlxI91cgFOuwi_f9sMPMxXZxO13o2W5im24QrraC9EpWxslalX7nhfsoCM7-AbnCg9E</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>210376701</pqid></control><display><type>article</type><title>Reviewing the reviews: How strong is the evidence? How clear are the conclusions?</title><source>MEDLINE</source><source>Cambridge University Press Journals Complete</source><creator>EZZO, Jeannette ; BAUSELL, Barker ; MOERMAN, Daniel E ; BERMAN, Brian ; HADHAZY, Victoria</creator><creatorcontrib>EZZO, Jeannette ; BAUSELL, Barker ; MOERMAN, Daniel E ; BERMAN, Brian ; HADHAZY, Victoria</creatorcontrib><description>The objectives of this paper were: a) to determine what can be learned from conclusions of systematic reviews about the evidence base of medicine; and b) to determine whether two readers draw similar conclusions from the same review, and whether these match the authors' conclusions. Three methodologists (two per review) rated 160 Cochrane systematic reviews (issue 1, 1998) using pre-established conclusion categories. Disagreements were resolved by discussion to arrive at a consensual score for each review. Reviews' authors were asked to use the same categories to designate the intended conclusion. Interrater agreements were calculated. Interrater agreement between two readers was 0.68 and 0.72, and between readers and authors, 0.32. The largest categories assigned by methodologists were "positive effect" (22.5%), "insufficient evidence" (21.3%), and "evidence of no effect" (20.0%). The largest categories assigned by authors were "insufficient evidence" (32.4%), "possibly positive" (28.6%), and "positive effect" (26.7%). The number of reviews indicating that the modern biomedical interventions show either no effect or insufficient evidence is surprisingly high. Interrater disagreements suggest a surprising degree of subjective interpretation involved in systematic reviews. Where patterns of disagreement emerged between authors and readers, authors tended to be more optimistic in their conclusions than the readers. Policy implications are discussed.</description><identifier>ISSN: 0266-4623</identifier><identifier>EISSN: 1471-6348</identifier><identifier>DOI: 10.1017/s0266462301107014</identifier><identifier>PMID: 11758290</identifier><identifier>CODEN: IJTCEK</identifier><language>eng</language><publisher>Cambridge: Cambridge University Press</publisher><subject>Agreements ; Alternative medicine ; Bibliometrics ; Biological and medical sciences ; Evidence-Based Medicine ; General aspects ; Humans ; Medical practices ; Medical sciences ; Medicine ; Meta-Analysis as Topic ; Observer Variation ; Outcome Assessment (Health Care) ; Randomized Controlled Trials as Topic - statistics &amp; numerical data ; United States</subject><ispartof>International journal of technology assessment in health care, 2001, Vol.17 (4), p.457-466</ispartof><rights>2002 INIST-CNRS</rights><rights>2001 Cambridge University Press</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,780,784,4024,27923,27924,27925</link.rule.ids><backlink>$$Uhttp://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&amp;idt=13382076$$DView record in Pascal Francis$$Hfree_for_read</backlink><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/11758290$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>EZZO, Jeannette</creatorcontrib><creatorcontrib>BAUSELL, Barker</creatorcontrib><creatorcontrib>MOERMAN, Daniel E</creatorcontrib><creatorcontrib>BERMAN, Brian</creatorcontrib><creatorcontrib>HADHAZY, Victoria</creatorcontrib><title>Reviewing the reviews: How strong is the evidence? How clear are the conclusions?</title><title>International journal of technology assessment in health care</title><addtitle>Int J Technol Assess Health Care</addtitle><description>The objectives of this paper were: a) to determine what can be learned from conclusions of systematic reviews about the evidence base of medicine; and b) to determine whether two readers draw similar conclusions from the same review, and whether these match the authors' conclusions. Three methodologists (two per review) rated 160 Cochrane systematic reviews (issue 1, 1998) using pre-established conclusion categories. Disagreements were resolved by discussion to arrive at a consensual score for each review. Reviews' authors were asked to use the same categories to designate the intended conclusion. Interrater agreements were calculated. Interrater agreement between two readers was 0.68 and 0.72, and between readers and authors, 0.32. The largest categories assigned by methodologists were "positive effect" (22.5%), "insufficient evidence" (21.3%), and "evidence of no effect" (20.0%). The largest categories assigned by authors were "insufficient evidence" (32.4%), "possibly positive" (28.6%), and "positive effect" (26.7%). The number of reviews indicating that the modern biomedical interventions show either no effect or insufficient evidence is surprisingly high. Interrater disagreements suggest a surprising degree of subjective interpretation involved in systematic reviews. Where patterns of disagreement emerged between authors and readers, authors tended to be more optimistic in their conclusions than the readers. Policy implications are discussed.</description><subject>Agreements</subject><subject>Alternative medicine</subject><subject>Bibliometrics</subject><subject>Biological and medical sciences</subject><subject>Evidence-Based Medicine</subject><subject>General aspects</subject><subject>Humans</subject><subject>Medical practices</subject><subject>Medical sciences</subject><subject>Medicine</subject><subject>Meta-Analysis as Topic</subject><subject>Observer Variation</subject><subject>Outcome Assessment (Health Care)</subject><subject>Randomized Controlled Trials as Topic - statistics &amp; numerical data</subject><subject>United States</subject><issn>0266-4623</issn><issn>1471-6348</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2001</creationdate><recordtype>article</recordtype><sourceid>EIF</sourceid><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><recordid>eNpdkN9LwzAQx4Mobk7_AF-kCPpWvUvapPVlyFAnDMRfzyXLEu3ompm0Dv97s24w2NOR-3zucnwJOUe4QUBx64FynnDKABEEYHJA-pgIjDlLskPSX-N4zXvkxPs5ADLI4Zj0EEWa0Rz65PVN_5Z6VdZfUfOtI9e9_F00tqvIN86Gfuk7FMhM10oPO6YqLV0kne6YsrWqWl_a2g9PyZGRlddn2zogn48PH6NxPHl5eh7dT2JFKTaxzHND0aSMTVkGPAOT0QwSqXnO09RoFiiohIo0BwUGqGTcpJCjShWnKmUDcr3Zu3T2p9W-KRalV7qqZK1t6wtBGWMZ0iBe7olz27o63FaEL5jgIbkg4UZSznrvtCmWrlxI91cgFOuwi_f9sMPMxXZxO13o2W5im24QrraC9EpWxslalX7nhfsoCM7-AbnCg9E</recordid><startdate>2001</startdate><enddate>2001</enddate><creator>EZZO, Jeannette</creator><creator>BAUSELL, Barker</creator><creator>MOERMAN, Daniel E</creator><creator>BERMAN, Brian</creator><creator>HADHAZY, Victoria</creator><general>Cambridge University Press</general><scope>IQODW</scope><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>3V.</scope><scope>7RV</scope><scope>7T5</scope><scope>7U5</scope><scope>7WY</scope><scope>7WZ</scope><scope>7X7</scope><scope>7XB</scope><scope>87Z</scope><scope>88C</scope><scope>88E</scope><scope>8C1</scope><scope>8FD</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>8FL</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>BENPR</scope><scope>BEZIV</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>FRNLG</scope><scope>FYUFA</scope><scope>F~G</scope><scope>GHDGH</scope><scope>H94</scope><scope>K60</scope><scope>K6~</scope><scope>K9.</scope><scope>KB0</scope><scope>L.-</scope><scope>L7M</scope><scope>M0C</scope><scope>M0S</scope><scope>M0T</scope><scope>M1P</scope><scope>NAPCQ</scope><scope>PQBIZ</scope><scope>PQBZA</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>Q9U</scope><scope>7X8</scope></search><sort><creationdate>2001</creationdate><title>Reviewing the reviews: How strong is the evidence? How clear are the conclusions?</title><author>EZZO, Jeannette ; BAUSELL, Barker ; MOERMAN, Daniel E ; BERMAN, Brian ; HADHAZY, Victoria</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c221t-a99f21f533b380680f82804ae69655fe3f210c427590c0f02a36f5091c5c62c53</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2001</creationdate><topic>Agreements</topic><topic>Alternative medicine</topic><topic>Bibliometrics</topic><topic>Biological and medical sciences</topic><topic>Evidence-Based Medicine</topic><topic>General aspects</topic><topic>Humans</topic><topic>Medical practices</topic><topic>Medical sciences</topic><topic>Medicine</topic><topic>Meta-Analysis as Topic</topic><topic>Observer Variation</topic><topic>Outcome Assessment (Health Care)</topic><topic>Randomized Controlled Trials as Topic - statistics &amp; numerical data</topic><topic>United States</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>EZZO, Jeannette</creatorcontrib><creatorcontrib>BAUSELL, Barker</creatorcontrib><creatorcontrib>MOERMAN, Daniel E</creatorcontrib><creatorcontrib>BERMAN, Brian</creatorcontrib><creatorcontrib>HADHAZY, Victoria</creatorcontrib><collection>Pascal-Francis</collection><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>ProQuest Central (Corporate)</collection><collection>Nursing &amp; Allied Health Database</collection><collection>Immunology Abstracts</collection><collection>Solid State and Superconductivity Abstracts</collection><collection>Access via ABI/INFORM (ProQuest)</collection><collection>ABI/INFORM Global (PDF only)</collection><collection>Health &amp; Medical Collection</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>ABI/INFORM Global (Alumni Edition)</collection><collection>Healthcare Administration Database (Alumni)</collection><collection>Medical Database (Alumni Edition)</collection><collection>Public Health Database</collection><collection>Technology Research Database</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ABI/INFORM Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest Central UK/Ireland</collection><collection>ProQuest Central</collection><collection>Business Premium Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>Business Premium Collection (Alumni)</collection><collection>Health Research Premium Collection</collection><collection>ABI/INFORM Global (Corporate)</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>AIDS and Cancer Research Abstracts</collection><collection>ProQuest Business Collection (Alumni Edition)</collection><collection>ProQuest Business Collection</collection><collection>ProQuest Health &amp; Medical Complete (Alumni)</collection><collection>Nursing &amp; Allied Health Database (Alumni Edition)</collection><collection>ABI/INFORM Professional Advanced</collection><collection>Advanced Technologies Database with Aerospace</collection><collection>ABI/INFORM Global</collection><collection>Health &amp; Medical Collection (Alumni Edition)</collection><collection>Healthcare Administration Database</collection><collection>Medical Database</collection><collection>Nursing &amp; Allied Health Premium</collection><collection>ProQuest One Business</collection><collection>ProQuest One Business (Alumni)</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central Basic</collection><collection>MEDLINE - Academic</collection><jtitle>International journal of technology assessment in health care</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>EZZO, Jeannette</au><au>BAUSELL, Barker</au><au>MOERMAN, Daniel E</au><au>BERMAN, Brian</au><au>HADHAZY, Victoria</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Reviewing the reviews: How strong is the evidence? How clear are the conclusions?</atitle><jtitle>International journal of technology assessment in health care</jtitle><addtitle>Int J Technol Assess Health Care</addtitle><date>2001</date><risdate>2001</risdate><volume>17</volume><issue>4</issue><spage>457</spage><epage>466</epage><pages>457-466</pages><issn>0266-4623</issn><eissn>1471-6348</eissn><coden>IJTCEK</coden><abstract>The objectives of this paper were: a) to determine what can be learned from conclusions of systematic reviews about the evidence base of medicine; and b) to determine whether two readers draw similar conclusions from the same review, and whether these match the authors' conclusions. Three methodologists (two per review) rated 160 Cochrane systematic reviews (issue 1, 1998) using pre-established conclusion categories. Disagreements were resolved by discussion to arrive at a consensual score for each review. Reviews' authors were asked to use the same categories to designate the intended conclusion. Interrater agreements were calculated. Interrater agreement between two readers was 0.68 and 0.72, and between readers and authors, 0.32. The largest categories assigned by methodologists were "positive effect" (22.5%), "insufficient evidence" (21.3%), and "evidence of no effect" (20.0%). The largest categories assigned by authors were "insufficient evidence" (32.4%), "possibly positive" (28.6%), and "positive effect" (26.7%). The number of reviews indicating that the modern biomedical interventions show either no effect or insufficient evidence is surprisingly high. Interrater disagreements suggest a surprising degree of subjective interpretation involved in systematic reviews. Where patterns of disagreement emerged between authors and readers, authors tended to be more optimistic in their conclusions than the readers. Policy implications are discussed.</abstract><cop>Cambridge</cop><pub>Cambridge University Press</pub><pmid>11758290</pmid><doi>10.1017/s0266462301107014</doi><tpages>10</tpages></addata></record>
fulltext fulltext
identifier ISSN: 0266-4623
ispartof International journal of technology assessment in health care, 2001, Vol.17 (4), p.457-466
issn 0266-4623
1471-6348
language eng
recordid cdi_proquest_miscellaneous_72333812
source MEDLINE; Cambridge University Press Journals Complete
subjects Agreements
Alternative medicine
Bibliometrics
Biological and medical sciences
Evidence-Based Medicine
General aspects
Humans
Medical practices
Medical sciences
Medicine
Meta-Analysis as Topic
Observer Variation
Outcome Assessment (Health Care)
Randomized Controlled Trials as Topic - statistics & numerical data
United States
title Reviewing the reviews: How strong is the evidence? How clear are the conclusions?
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-24T14%3A43%3A00IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Reviewing%20the%20reviews:%20How%20strong%20is%20the%20evidence?%20How%20clear%20are%20the%20conclusions?&rft.jtitle=International%20journal%20of%20technology%20assessment%20in%20health%20care&rft.au=EZZO,%20Jeannette&rft.date=2001&rft.volume=17&rft.issue=4&rft.spage=457&rft.epage=466&rft.pages=457-466&rft.issn=0266-4623&rft.eissn=1471-6348&rft.coden=IJTCEK&rft_id=info:doi/10.1017/s0266462301107014&rft_dat=%3Cproquest_cross%3E72333812%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=210376701&rft_id=info:pmid/11758290&rfr_iscdi=true