Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions
Background Systematic reviews may be compromised by selective inclusion and reporting of outcomes and analyses. Selective inclusion occurs when there are multiple effect estimates in a trial report that could be included in a particular meta‐analysis (e.g. from multiple measurement scales and time p...
Gespeichert in:
| Veröffentlicht in: | Cochrane database of systematic reviews 2014-10, Vol.2015 (1), p.MR000035-MR000035 |
|---|---|
| Hauptverfasser: | , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | MR000035 |
|---|---|
| container_issue | 1 |
| container_start_page | MR000035 |
| container_title | Cochrane database of systematic reviews |
| container_volume | 2015 |
| creator | Page, Matthew J McKenzie, Joanne E Kirkham, Jamie Dwan, Kerry Kramer, Sharon Green, Sally Forbes, Andrew Page, Matthew J |
| description | Background
Systematic reviews may be compromised by selective inclusion and reporting of outcomes and analyses. Selective inclusion occurs when there are multiple effect estimates in a trial report that could be included in a particular meta‐analysis (e.g. from multiple measurement scales and time points) and the choice of effect estimate to include in the meta‐analysis is based on the results (e.g. statistical significance, magnitude or direction of effect). Selective reporting occurs when the reporting of a subset of outcomes and analyses in the systematic review is based on the results (e.g. a protocol‐defined outcome is omitted from the published systematic review).
Objectives
To summarise the characteristics and synthesise the results of empirical studies that have investigated the prevalence of selective inclusion or reporting in systematic reviews of randomised controlled trials (RCTs), investigated the factors (e.g. statistical significance or direction of effect) associated with the prevalence and quantified the bias.
Search methods
We searched the Cochrane Methodology Register (to July 2012), Ovid MEDLINE, Ovid EMBASE, Ovid PsycINFO and ISI Web of Science (each up to May 2013), and the US Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program's Scientific Resource Center (SRC) Methods Library (to June 2013). We also searched the books of the 2011 and 2012 Cochrane Colloquia and the article alerts for methodological work in research synthesis published from 2009 to 2011 and compiled in Research Synthesis Methods.
Selection criteria
We included both published and unpublished empirical studies that investigated the prevalence and factors associated with selective inclusion or reporting, or both, in systematic reviews of RCTs of healthcare interventions. We included empirical studies assessing any type of selective inclusion or reporting, such as investigations of how frequently RCT outcome data is selectively included in systematic reviews based on the results, outcomes and analyses are discrepant between protocol and published review or non‐significant outcomes are partially reported in the full text or summary within systematic reviews.
Data collection and analysis
Two review authors independently selected empirical studies for inclusion, extracted the data and performed a risk of bias assessment. A third review author resolved any disagreements about inclusion or exclusion of empirical studies, data extraction and risk of |
| doi_str_mv | 10.1002/14651858.MR000035.pub2 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_pubme</sourceid><recordid>TN_cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_8191366</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>1619319005</sourcerecordid><originalsourceid>FETCH-LOGICAL-c4732-ad8a6bbe50d6dd0c8396b24744262281cfd4c0dc0c46c46e9665611d4d924d8f3</originalsourceid><addsrcrecordid>eNqFUd1OHCEYJY2mWttXMFx6s1tgGHbmxqSaqk1sTIy9Jix849IwsAVmzT5E31lm192oNxISfs75zoHvIHRKyZQSwr5TLmra1M309z0po6qny2HOPqHjEZiMyMGr_RH6ktLfQhMtm31GR6xmM0ra5hj9v7AqYTMAzgEncKCzXQG2Xrsh2eCx8gZHWIaYrX_EocNhyDr0kDaI8sqtUzlYj9M6ZehVtroUrCw8pZEeCy30NoHBOVrlNpcLUC4vtIqjU4a4Ap-LWfqKDrtCgW8v6wn6c_Xz4fJmcnt3_evyx-1E81nFJso0SsznUBMjjCG6qVoxZ3zGOROMNVR3hmtiNNFclAmtELWg1HDTMm6arjpB51vd0rQejC72UTm5jLZXcS2DsvIt4u1CPoaVbGhLKyGKwNmLQAz_BkhZli9qcE55CEOSVNC2oi0hdaGKLVXHkFKEbm9DiRyzlLss5S7L0ZyVwtPXj9yX7cIrhIst4ck6WEsd9KJ0Gz7QfefyDGnTtCc</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1619319005</pqid></control><display><type>article</type><title>Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions</title><source>MEDLINE</source><source>Cochrane Library</source><source>EZB-FREE-00999 freely available EZB journals</source><source>Alma/SFX Local Collection</source><creator>Page, Matthew J ; McKenzie, Joanne E ; Kirkham, Jamie ; Dwan, Kerry ; Kramer, Sharon ; Green, Sally ; Forbes, Andrew ; Page, Matthew J</creator><creatorcontrib>Page, Matthew J ; McKenzie, Joanne E ; Kirkham, Jamie ; Dwan, Kerry ; Kramer, Sharon ; Green, Sally ; Forbes, Andrew ; Page, Matthew J</creatorcontrib><description>Background
Systematic reviews may be compromised by selective inclusion and reporting of outcomes and analyses. Selective inclusion occurs when there are multiple effect estimates in a trial report that could be included in a particular meta‐analysis (e.g. from multiple measurement scales and time points) and the choice of effect estimate to include in the meta‐analysis is based on the results (e.g. statistical significance, magnitude or direction of effect). Selective reporting occurs when the reporting of a subset of outcomes and analyses in the systematic review is based on the results (e.g. a protocol‐defined outcome is omitted from the published systematic review).
Objectives
To summarise the characteristics and synthesise the results of empirical studies that have investigated the prevalence of selective inclusion or reporting in systematic reviews of randomised controlled trials (RCTs), investigated the factors (e.g. statistical significance or direction of effect) associated with the prevalence and quantified the bias.
Search methods
We searched the Cochrane Methodology Register (to July 2012), Ovid MEDLINE, Ovid EMBASE, Ovid PsycINFO and ISI Web of Science (each up to May 2013), and the US Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program's Scientific Resource Center (SRC) Methods Library (to June 2013). We also searched the books of the 2011 and 2012 Cochrane Colloquia and the article alerts for methodological work in research synthesis published from 2009 to 2011 and compiled in Research Synthesis Methods.
Selection criteria
We included both published and unpublished empirical studies that investigated the prevalence and factors associated with selective inclusion or reporting, or both, in systematic reviews of RCTs of healthcare interventions. We included empirical studies assessing any type of selective inclusion or reporting, such as investigations of how frequently RCT outcome data is selectively included in systematic reviews based on the results, outcomes and analyses are discrepant between protocol and published review or non‐significant outcomes are partially reported in the full text or summary within systematic reviews.
Data collection and analysis
Two review authors independently selected empirical studies for inclusion, extracted the data and performed a risk of bias assessment. A third review author resolved any disagreements about inclusion or exclusion of empirical studies, data extraction and risk of bias. We contacted authors of included studies for additional unpublished data. Primary outcomes included overall prevalence of selective inclusion or reporting, association between selective inclusion or reporting and the statistical significance of the effect estimate, and association between selective inclusion or reporting and the direction of the effect estimate. We combined prevalence estimates and risk ratios (RRs) using a random‐effects meta‐analysis model.
Main results
Seven studies met the inclusion criteria. No studies had investigated selective inclusion of results in systematic reviews, or discrepancies in outcomes and analyses between systematic review registry entries and published systematic reviews. Based on a meta‐analysis of four studies (including 485 Cochrane Reviews), 38% (95% confidence interval (CI) 23% to 54%) of systematic reviews added, omitted, upgraded or downgraded at least one outcome between the protocol and published systematic review. The association between statistical significance and discrepant outcome reporting between protocol and published systematic review was uncertain. The meta‐analytic estimate suggested an increased risk of adding or upgrading (i.e. changing a secondary outcome to primary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.43, 95% CI 0.71 to 2.85; two studies, n = 552 meta‐analyses). Also, the meta‐analytic estimate suggested an increased risk of downgrading (i.e. changing a primary outcome to secondary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.26, 95% CI 0.60 to 2.62; two studies, n = 484 meta‐analyses). None of the included studies had investigated whether the association between statistical significance and adding, upgrading or downgrading of outcomes was modified by the type of comparison, direction of effect or type of outcome; or whether there is an association between direction of the effect estimate and discrepant outcome reporting.
Several secondary outcomes were reported in the included studies. Two studies found that reasons for discrepant outcome reporting were infrequently reported in published systematic reviews (6% in one study and 22% in the other). One study (including 62 Cochrane Reviews) found that 32% (95% CI 21% to 45%) of systematic reviews did not report all primary outcomes in the . Another study (including 64 Cochrane and 118 non‐Cochrane reviews) found that statistically significant primary outcomes were more likely to be completely reported in the systematic review than non‐significant primary outcomes (RR 2.66, 95% CI 1.81 to 3.90). None of the studies included systematic reviews published after 2009 when reporting standards for systematic reviews (Preferred Reporting Items for Systematic reviews and Meta‐Analyses (PRISMA) Statement, and Methodological Expectations of Cochrane Intervention Reviews (MECIR)) were disseminated, so the results might not be generalisable to more recent systematic reviews.
Authors' conclusions
Discrepant outcome reporting between the protocol and published systematic review is fairly common, although the association between statistical significance and discrepant outcome reporting is uncertain. Complete reporting of outcomes in systematic review s is associated with statistical significance of the results for those outcomes. Systematic review outcomes and analysis plans should be specified prior to seeing the results of included studies to minimise post‐hoc decisions that may be based on the observed results. Modifications that occur once the review has commenced, along with their justification, should be clearly reported. Effect estimates and CIs should be reported for all systematic review outcomes regardless of the results. The lack of research on selective inclusion of results in systematic reviews needs to be addressed and studies that avoid the methodological weaknesses of existing research are also needed.</description><identifier>ISSN: 1465-1858</identifier><identifier>EISSN: 1465-1858</identifier><identifier>EISSN: 1469-493X</identifier><identifier>DOI: 10.1002/14651858.MR000035.pub2</identifier><identifier>PMID: 25271098</identifier><language>eng</language><publisher>Chichester, UK: John Wiley & Sons, Ltd</publisher><subject>Assessments of the quality of reviews ; Child health ; Medicine General & Introductory Medical Sciences ; Meta-Analysis as Topic ; META‐ANALYSIS ; Methodology ; Methods for handling missing data ; PRESENTATION OF REVIEWS ; Randomized Controlled Trials as Topic ; Review Literature as Topic ; REVIEW METHODOLOGY ; Selection Bias ; Subgroup analyses versus overall analyses ; Treatment Outcome</subject><ispartof>Cochrane database of systematic reviews, 2014-10, Vol.2015 (1), p.MR000035-MR000035</ispartof><rights>Copyright © 2015 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd.</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c4732-ad8a6bbe50d6dd0c8396b24744262281cfd4c0dc0c46c46e9665611d4d924d8f3</citedby><cites>FETCH-LOGICAL-c4732-ad8a6bbe50d6dd0c8396b24744262281cfd4c0dc0c46c46e9665611d4d924d8f3</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>230,314,780,784,885,27924,27925</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/25271098$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Page, Matthew J</creatorcontrib><creatorcontrib>McKenzie, Joanne E</creatorcontrib><creatorcontrib>Kirkham, Jamie</creatorcontrib><creatorcontrib>Dwan, Kerry</creatorcontrib><creatorcontrib>Kramer, Sharon</creatorcontrib><creatorcontrib>Green, Sally</creatorcontrib><creatorcontrib>Forbes, Andrew</creatorcontrib><creatorcontrib>Page, Matthew J</creatorcontrib><title>Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions</title><title>Cochrane database of systematic reviews</title><addtitle>Cochrane Database Syst Rev</addtitle><description>Background
Systematic reviews may be compromised by selective inclusion and reporting of outcomes and analyses. Selective inclusion occurs when there are multiple effect estimates in a trial report that could be included in a particular meta‐analysis (e.g. from multiple measurement scales and time points) and the choice of effect estimate to include in the meta‐analysis is based on the results (e.g. statistical significance, magnitude or direction of effect). Selective reporting occurs when the reporting of a subset of outcomes and analyses in the systematic review is based on the results (e.g. a protocol‐defined outcome is omitted from the published systematic review).
Objectives
To summarise the characteristics and synthesise the results of empirical studies that have investigated the prevalence of selective inclusion or reporting in systematic reviews of randomised controlled trials (RCTs), investigated the factors (e.g. statistical significance or direction of effect) associated with the prevalence and quantified the bias.
Search methods
We searched the Cochrane Methodology Register (to July 2012), Ovid MEDLINE, Ovid EMBASE, Ovid PsycINFO and ISI Web of Science (each up to May 2013), and the US Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program's Scientific Resource Center (SRC) Methods Library (to June 2013). We also searched the books of the 2011 and 2012 Cochrane Colloquia and the article alerts for methodological work in research synthesis published from 2009 to 2011 and compiled in Research Synthesis Methods.
Selection criteria
We included both published and unpublished empirical studies that investigated the prevalence and factors associated with selective inclusion or reporting, or both, in systematic reviews of RCTs of healthcare interventions. We included empirical studies assessing any type of selective inclusion or reporting, such as investigations of how frequently RCT outcome data is selectively included in systematic reviews based on the results, outcomes and analyses are discrepant between protocol and published review or non‐significant outcomes are partially reported in the full text or summary within systematic reviews.
Data collection and analysis
Two review authors independently selected empirical studies for inclusion, extracted the data and performed a risk of bias assessment. A third review author resolved any disagreements about inclusion or exclusion of empirical studies, data extraction and risk of bias. We contacted authors of included studies for additional unpublished data. Primary outcomes included overall prevalence of selective inclusion or reporting, association between selective inclusion or reporting and the statistical significance of the effect estimate, and association between selective inclusion or reporting and the direction of the effect estimate. We combined prevalence estimates and risk ratios (RRs) using a random‐effects meta‐analysis model.
Main results
Seven studies met the inclusion criteria. No studies had investigated selective inclusion of results in systematic reviews, or discrepancies in outcomes and analyses between systematic review registry entries and published systematic reviews. Based on a meta‐analysis of four studies (including 485 Cochrane Reviews), 38% (95% confidence interval (CI) 23% to 54%) of systematic reviews added, omitted, upgraded or downgraded at least one outcome between the protocol and published systematic review. The association between statistical significance and discrepant outcome reporting between protocol and published systematic review was uncertain. The meta‐analytic estimate suggested an increased risk of adding or upgrading (i.e. changing a secondary outcome to primary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.43, 95% CI 0.71 to 2.85; two studies, n = 552 meta‐analyses). Also, the meta‐analytic estimate suggested an increased risk of downgrading (i.e. changing a primary outcome to secondary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.26, 95% CI 0.60 to 2.62; two studies, n = 484 meta‐analyses). None of the included studies had investigated whether the association between statistical significance and adding, upgrading or downgrading of outcomes was modified by the type of comparison, direction of effect or type of outcome; or whether there is an association between direction of the effect estimate and discrepant outcome reporting.
Several secondary outcomes were reported in the included studies. Two studies found that reasons for discrepant outcome reporting were infrequently reported in published systematic reviews (6% in one study and 22% in the other). One study (including 62 Cochrane Reviews) found that 32% (95% CI 21% to 45%) of systematic reviews did not report all primary outcomes in the . Another study (including 64 Cochrane and 118 non‐Cochrane reviews) found that statistically significant primary outcomes were more likely to be completely reported in the systematic review than non‐significant primary outcomes (RR 2.66, 95% CI 1.81 to 3.90). None of the studies included systematic reviews published after 2009 when reporting standards for systematic reviews (Preferred Reporting Items for Systematic reviews and Meta‐Analyses (PRISMA) Statement, and Methodological Expectations of Cochrane Intervention Reviews (MECIR)) were disseminated, so the results might not be generalisable to more recent systematic reviews.
Authors' conclusions
Discrepant outcome reporting between the protocol and published systematic review is fairly common, although the association between statistical significance and discrepant outcome reporting is uncertain. Complete reporting of outcomes in systematic review s is associated with statistical significance of the results for those outcomes. Systematic review outcomes and analysis plans should be specified prior to seeing the results of included studies to minimise post‐hoc decisions that may be based on the observed results. Modifications that occur once the review has commenced, along with their justification, should be clearly reported. Effect estimates and CIs should be reported for all systematic review outcomes regardless of the results. The lack of research on selective inclusion of results in systematic reviews needs to be addressed and studies that avoid the methodological weaknesses of existing research are also needed.</description><subject>Assessments of the quality of reviews</subject><subject>Child health</subject><subject>Medicine General & Introductory Medical Sciences</subject><subject>Meta-Analysis as Topic</subject><subject>META‐ANALYSIS</subject><subject>Methodology</subject><subject>Methods for handling missing data</subject><subject>PRESENTATION OF REVIEWS</subject><subject>Randomized Controlled Trials as Topic</subject><subject>Review Literature as Topic</subject><subject>REVIEW METHODOLOGY</subject><subject>Selection Bias</subject><subject>Subgroup analyses versus overall analyses</subject><subject>Treatment Outcome</subject><issn>1465-1858</issn><issn>1465-1858</issn><issn>1469-493X</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2014</creationdate><recordtype>article</recordtype><sourceid>RWY</sourceid><sourceid>EIF</sourceid><recordid>eNqFUd1OHCEYJY2mWttXMFx6s1tgGHbmxqSaqk1sTIy9Jix849IwsAVmzT5E31lm192oNxISfs75zoHvIHRKyZQSwr5TLmra1M309z0po6qny2HOPqHjEZiMyMGr_RH6ktLfQhMtm31GR6xmM0ra5hj9v7AqYTMAzgEncKCzXQG2Xrsh2eCx8gZHWIaYrX_EocNhyDr0kDaI8sqtUzlYj9M6ZehVtroUrCw8pZEeCy30NoHBOVrlNpcLUC4vtIqjU4a4Ap-LWfqKDrtCgW8v6wn6c_Xz4fJmcnt3_evyx-1E81nFJso0SsznUBMjjCG6qVoxZ3zGOROMNVR3hmtiNNFclAmtELWg1HDTMm6arjpB51vd0rQejC72UTm5jLZXcS2DsvIt4u1CPoaVbGhLKyGKwNmLQAz_BkhZli9qcE55CEOSVNC2oi0hdaGKLVXHkFKEbm9DiRyzlLss5S7L0ZyVwtPXj9yX7cIrhIst4ck6WEsd9KJ0Gz7QfefyDGnTtCc</recordid><startdate>20141001</startdate><enddate>20141001</enddate><creator>Page, Matthew J</creator><creator>McKenzie, Joanne E</creator><creator>Kirkham, Jamie</creator><creator>Dwan, Kerry</creator><creator>Kramer, Sharon</creator><creator>Green, Sally</creator><creator>Forbes, Andrew</creator><creator>Page, Matthew J</creator><general>John Wiley & Sons, Ltd</general><scope>7PX</scope><scope>RWY</scope><scope>ZYTZH</scope><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7X8</scope><scope>5PM</scope></search><sort><creationdate>20141001</creationdate><title>Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions</title><author>Page, Matthew J ; McKenzie, Joanne E ; Kirkham, Jamie ; Dwan, Kerry ; Kramer, Sharon ; Green, Sally ; Forbes, Andrew ; Page, Matthew J</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c4732-ad8a6bbe50d6dd0c8396b24744262281cfd4c0dc0c46c46e9665611d4d924d8f3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2014</creationdate><topic>Assessments of the quality of reviews</topic><topic>Child health</topic><topic>Medicine General & Introductory Medical Sciences</topic><topic>Meta-Analysis as Topic</topic><topic>META‐ANALYSIS</topic><topic>Methodology</topic><topic>Methods for handling missing data</topic><topic>PRESENTATION OF REVIEWS</topic><topic>Randomized Controlled Trials as Topic</topic><topic>Review Literature as Topic</topic><topic>REVIEW METHODOLOGY</topic><topic>Selection Bias</topic><topic>Subgroup analyses versus overall analyses</topic><topic>Treatment Outcome</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Page, Matthew J</creatorcontrib><creatorcontrib>McKenzie, Joanne E</creatorcontrib><creatorcontrib>Kirkham, Jamie</creatorcontrib><creatorcontrib>Dwan, Kerry</creatorcontrib><creatorcontrib>Kramer, Sharon</creatorcontrib><creatorcontrib>Green, Sally</creatorcontrib><creatorcontrib>Forbes, Andrew</creatorcontrib><creatorcontrib>Page, Matthew J</creatorcontrib><collection>Wiley-Blackwell Cochrane Library</collection><collection>Cochrane Library</collection><collection>Cochrane Library (Open Aceess)</collection><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><jtitle>Cochrane database of systematic reviews</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Page, Matthew J</au><au>McKenzie, Joanne E</au><au>Kirkham, Jamie</au><au>Dwan, Kerry</au><au>Kramer, Sharon</au><au>Green, Sally</au><au>Forbes, Andrew</au><au>Page, Matthew J</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions</atitle><jtitle>Cochrane database of systematic reviews</jtitle><addtitle>Cochrane Database Syst Rev</addtitle><date>2014-10-01</date><risdate>2014</risdate><volume>2015</volume><issue>1</issue><spage>MR000035</spage><epage>MR000035</epage><pages>MR000035-MR000035</pages><issn>1465-1858</issn><eissn>1465-1858</eissn><eissn>1469-493X</eissn><abstract>Background
Systematic reviews may be compromised by selective inclusion and reporting of outcomes and analyses. Selective inclusion occurs when there are multiple effect estimates in a trial report that could be included in a particular meta‐analysis (e.g. from multiple measurement scales and time points) and the choice of effect estimate to include in the meta‐analysis is based on the results (e.g. statistical significance, magnitude or direction of effect). Selective reporting occurs when the reporting of a subset of outcomes and analyses in the systematic review is based on the results (e.g. a protocol‐defined outcome is omitted from the published systematic review).
Objectives
To summarise the characteristics and synthesise the results of empirical studies that have investigated the prevalence of selective inclusion or reporting in systematic reviews of randomised controlled trials (RCTs), investigated the factors (e.g. statistical significance or direction of effect) associated with the prevalence and quantified the bias.
Search methods
We searched the Cochrane Methodology Register (to July 2012), Ovid MEDLINE, Ovid EMBASE, Ovid PsycINFO and ISI Web of Science (each up to May 2013), and the US Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program's Scientific Resource Center (SRC) Methods Library (to June 2013). We also searched the books of the 2011 and 2012 Cochrane Colloquia and the article alerts for methodological work in research synthesis published from 2009 to 2011 and compiled in Research Synthesis Methods.
Selection criteria
We included both published and unpublished empirical studies that investigated the prevalence and factors associated with selective inclusion or reporting, or both, in systematic reviews of RCTs of healthcare interventions. We included empirical studies assessing any type of selective inclusion or reporting, such as investigations of how frequently RCT outcome data is selectively included in systematic reviews based on the results, outcomes and analyses are discrepant between protocol and published review or non‐significant outcomes are partially reported in the full text or summary within systematic reviews.
Data collection and analysis
Two review authors independently selected empirical studies for inclusion, extracted the data and performed a risk of bias assessment. A third review author resolved any disagreements about inclusion or exclusion of empirical studies, data extraction and risk of bias. We contacted authors of included studies for additional unpublished data. Primary outcomes included overall prevalence of selective inclusion or reporting, association between selective inclusion or reporting and the statistical significance of the effect estimate, and association between selective inclusion or reporting and the direction of the effect estimate. We combined prevalence estimates and risk ratios (RRs) using a random‐effects meta‐analysis model.
Main results
Seven studies met the inclusion criteria. No studies had investigated selective inclusion of results in systematic reviews, or discrepancies in outcomes and analyses between systematic review registry entries and published systematic reviews. Based on a meta‐analysis of four studies (including 485 Cochrane Reviews), 38% (95% confidence interval (CI) 23% to 54%) of systematic reviews added, omitted, upgraded or downgraded at least one outcome between the protocol and published systematic review. The association between statistical significance and discrepant outcome reporting between protocol and published systematic review was uncertain. The meta‐analytic estimate suggested an increased risk of adding or upgrading (i.e. changing a secondary outcome to primary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.43, 95% CI 0.71 to 2.85; two studies, n = 552 meta‐analyses). Also, the meta‐analytic estimate suggested an increased risk of downgrading (i.e. changing a primary outcome to secondary) when the outcome was statistically significant, although the 95% CI included no association and a decreased risk as plausible estimates (RR 1.26, 95% CI 0.60 to 2.62; two studies, n = 484 meta‐analyses). None of the included studies had investigated whether the association between statistical significance and adding, upgrading or downgrading of outcomes was modified by the type of comparison, direction of effect or type of outcome; or whether there is an association between direction of the effect estimate and discrepant outcome reporting.
Several secondary outcomes were reported in the included studies. Two studies found that reasons for discrepant outcome reporting were infrequently reported in published systematic reviews (6% in one study and 22% in the other). One study (including 62 Cochrane Reviews) found that 32% (95% CI 21% to 45%) of systematic reviews did not report all primary outcomes in the . Another study (including 64 Cochrane and 118 non‐Cochrane reviews) found that statistically significant primary outcomes were more likely to be completely reported in the systematic review than non‐significant primary outcomes (RR 2.66, 95% CI 1.81 to 3.90). None of the studies included systematic reviews published after 2009 when reporting standards for systematic reviews (Preferred Reporting Items for Systematic reviews and Meta‐Analyses (PRISMA) Statement, and Methodological Expectations of Cochrane Intervention Reviews (MECIR)) were disseminated, so the results might not be generalisable to more recent systematic reviews.
Authors' conclusions
Discrepant outcome reporting between the protocol and published systematic review is fairly common, although the association between statistical significance and discrepant outcome reporting is uncertain. Complete reporting of outcomes in systematic review s is associated with statistical significance of the results for those outcomes. Systematic review outcomes and analysis plans should be specified prior to seeing the results of included studies to minimise post‐hoc decisions that may be based on the observed results. Modifications that occur once the review has commenced, along with their justification, should be clearly reported. Effect estimates and CIs should be reported for all systematic review outcomes regardless of the results. The lack of research on selective inclusion of results in systematic reviews needs to be addressed and studies that avoid the methodological weaknesses of existing research are also needed.</abstract><cop>Chichester, UK</cop><pub>John Wiley & Sons, Ltd</pub><pmid>25271098</pmid><doi>10.1002/14651858.MR000035.pub2</doi><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1465-1858 |
| ispartof | Cochrane database of systematic reviews, 2014-10, Vol.2015 (1), p.MR000035-MR000035 |
| issn | 1465-1858 1465-1858 1469-493X |
| language | eng |
| recordid | cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_8191366 |
| source | MEDLINE; Cochrane Library; EZB-FREE-00999 freely available EZB journals; Alma/SFX Local Collection |
| subjects | Assessments of the quality of reviews Child health Medicine General & Introductory Medical Sciences Meta-Analysis as Topic META‐ANALYSIS Methodology Methods for handling missing data PRESENTATION OF REVIEWS Randomized Controlled Trials as Topic Review Literature as Topic REVIEW METHODOLOGY Selection Bias Subgroup analyses versus overall analyses Treatment Outcome |
| title | Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-02T19%3A28%3A53IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_pubme&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Bias%20due%20to%20selective%20inclusion%20and%20reporting%20of%20outcomes%20and%20analyses%20in%20systematic%20reviews%20of%20randomised%20trials%20of%20healthcare%20interventions&rft.jtitle=Cochrane%20database%20of%20systematic%20reviews&rft.au=Page,%20Matthew%20J&rft.date=2014-10-01&rft.volume=2015&rft.issue=1&rft.spage=MR000035&rft.epage=MR000035&rft.pages=MR000035-MR000035&rft.issn=1465-1858&rft.eissn=1465-1858&rft_id=info:doi/10.1002/14651858.MR000035.pub2&rft_dat=%3Cproquest_pubme%3E1619319005%3C/proquest_pubme%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1619319005&rft_id=info:pmid/25271098&rfr_iscdi=true |