On the choice of acceptance radius in free-response observer performance studies
Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acc...
Gespeichert in:
| Veröffentlicht in: | British journal of radiology 2013-01, Vol.86 (1021), p.42313554-42313554 |
|---|---|
| Hauptverfasser: | , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 42313554 |
|---|---|
| container_issue | 1021 |
| container_start_page | 42313554 |
| container_title | British journal of radiology |
| container_volume | 86 |
| creator | Haygood, T M Ryan, J Brennan, P C Li, S Marom, E M McEntee, M F Itani, M Evanoff, M Chakraborty, D |
| description | Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius.
80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of |
| doi_str_mv | 10.1259/bjr/42313554 |
| format | Article |
| fullrecord | <record><control><sourceid>proquest_pubme</sourceid><recordid>TN_cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_3615402</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>1273229088</sourcerecordid><originalsourceid>FETCH-LOGICAL-c384t-1e7a8f07d04b72b0b79f23a45571370d21a5c413c2a9697552d4dc2b56f2b37c3</originalsourceid><addsrcrecordid>eNpVkc1Lw0AQxRdRbK3ePEuOHozdnd3tJhdBil9QqAcFb8tmM7EpaTbuJgX_e1P7gZ6Gx_x4M7xHyCWjtwxkOs6WfiyAMy6lOCJDpkQSJwn9OCZDSqmKGSRyQM5CWG6kTOkpGQBIxTmFIXmd11G7wMguXGkxckVkrMWmNXWvvMnLLkRlHRUeMfYYGleHnsoC-jX6qEFfOL_6hUPb5SWGc3JSmCrgxW6OyPvjw9v0OZ7Nn16m97PY8kS0MUNlkoKqnIpMQUYzlRbAjZBSMa5oDsxIKxi3YNJJqqSEXOQWMjkpIOPK8hG52_o2XbbC3GLdelPpxpcr47-1M6X-v6nLhf50a80nTAoKvcH1zsC7rw5Dq1dlsFhVpkbXBc1AcYCUJkmP3mxR610IHovDGUb1pgTdl6D3JfT41d_XDvA-df4DKTWDlQ</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1273229088</pqid></control><display><type>article</type><title>On the choice of acceptance radius in free-response observer performance studies</title><source>MEDLINE</source><source>Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals</source><source>Oxford University Press Journals All Titles (1996-Current)</source><creator>Haygood, T M ; Ryan, J ; Brennan, P C ; Li, S ; Marom, E M ; McEntee, M F ; Itani, M ; Evanoff, M ; Chakraborty, D</creator><creatorcontrib>Haygood, T M ; Ryan, J ; Brennan, P C ; Li, S ; Marom, E M ; McEntee, M F ; Itani, M ; Evanoff, M ; Chakraborty, D</creatorcontrib><description><![CDATA[Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius.
80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of <5 pixels, <10 pixels, <20 pixels and onwards up to <200 and 200+ pixels. We counted lesion localisations in each bin and visually compared marks with the borders of nodules.
Most reader marks were tightly clustered around nodule centres, with tighter clustering for smaller than for larger nodules. At least 70% of readers' marks were placed within <10 pixels for small nodules, <20 pixels for medium nodules and <30 pixels for large nodules. Of 72 inspected marks that were less than 50 pixels from the centre of a nodule, only 1 fell outside the border of a nodule.
The acceptance radius should be based on the larger nodule sizes. For our data, an acceptance radius of 50 pixels would have captured all but 2 reader marks within the borders of a nodule, while excluding only 1 true-positive mark. The choice of an acceptance radius for FROC analysis of observer performance studies should be based on the size of larger abnormalities.]]></description><identifier>ISSN: 0007-1285</identifier><identifier>EISSN: 1748-880X</identifier><identifier>DOI: 10.1259/bjr/42313554</identifier><identifier>PMID: 22573302</identifier><language>eng</language><publisher>England: The British Institute of Radiology</publisher><subject>Humans ; MIPS special feature ; Observer Variation ; Radiographic Image Enhancement - methods ; Radiographic Image Interpretation, Computer-Assisted - methods ; Radiography, Thoracic - methods ; Reproducibility of Results ; ROC Curve ; Sensitivity and Specificity ; Solitary Pulmonary Nodule - diagnostic imaging ; Tomography, X-Ray Computed - adverse effects ; Tomography, X-Ray Computed - methods</subject><ispartof>British journal of radiology, 2013-01, Vol.86 (1021), p.42313554-42313554</ispartof><rights>2013 The British Institute of Radiology 2013</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c384t-1e7a8f07d04b72b0b79f23a45571370d21a5c413c2a9697552d4dc2b56f2b37c3</citedby><cites>FETCH-LOGICAL-c384t-1e7a8f07d04b72b0b79f23a45571370d21a5c413c2a9697552d4dc2b56f2b37c3</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>230,315,781,785,886,27929,27930</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/22573302$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><creatorcontrib>Haygood, T M</creatorcontrib><creatorcontrib>Ryan, J</creatorcontrib><creatorcontrib>Brennan, P C</creatorcontrib><creatorcontrib>Li, S</creatorcontrib><creatorcontrib>Marom, E M</creatorcontrib><creatorcontrib>McEntee, M F</creatorcontrib><creatorcontrib>Itani, M</creatorcontrib><creatorcontrib>Evanoff, M</creatorcontrib><creatorcontrib>Chakraborty, D</creatorcontrib><title>On the choice of acceptance radius in free-response observer performance studies</title><title>British journal of radiology</title><addtitle>Br J Radiol</addtitle><description><![CDATA[Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius.
80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of <5 pixels, <10 pixels, <20 pixels and onwards up to <200 and 200+ pixels. We counted lesion localisations in each bin and visually compared marks with the borders of nodules.
Most reader marks were tightly clustered around nodule centres, with tighter clustering for smaller than for larger nodules. At least 70% of readers' marks were placed within <10 pixels for small nodules, <20 pixels for medium nodules and <30 pixels for large nodules. Of 72 inspected marks that were less than 50 pixels from the centre of a nodule, only 1 fell outside the border of a nodule.
The acceptance radius should be based on the larger nodule sizes. For our data, an acceptance radius of 50 pixels would have captured all but 2 reader marks within the borders of a nodule, while excluding only 1 true-positive mark. The choice of an acceptance radius for FROC analysis of observer performance studies should be based on the size of larger abnormalities.]]></description><subject>Humans</subject><subject>MIPS special feature</subject><subject>Observer Variation</subject><subject>Radiographic Image Enhancement - methods</subject><subject>Radiographic Image Interpretation, Computer-Assisted - methods</subject><subject>Radiography, Thoracic - methods</subject><subject>Reproducibility of Results</subject><subject>ROC Curve</subject><subject>Sensitivity and Specificity</subject><subject>Solitary Pulmonary Nodule - diagnostic imaging</subject><subject>Tomography, X-Ray Computed - adverse effects</subject><subject>Tomography, X-Ray Computed - methods</subject><issn>0007-1285</issn><issn>1748-880X</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2013</creationdate><recordtype>article</recordtype><sourceid>EIF</sourceid><recordid>eNpVkc1Lw0AQxRdRbK3ePEuOHozdnd3tJhdBil9QqAcFb8tmM7EpaTbuJgX_e1P7gZ6Gx_x4M7xHyCWjtwxkOs6WfiyAMy6lOCJDpkQSJwn9OCZDSqmKGSRyQM5CWG6kTOkpGQBIxTmFIXmd11G7wMguXGkxckVkrMWmNXWvvMnLLkRlHRUeMfYYGleHnsoC-jX6qEFfOL_6hUPb5SWGc3JSmCrgxW6OyPvjw9v0OZ7Nn16m97PY8kS0MUNlkoKqnIpMQUYzlRbAjZBSMa5oDsxIKxi3YNJJqqSEXOQWMjkpIOPK8hG52_o2XbbC3GLdelPpxpcr47-1M6X-v6nLhf50a80nTAoKvcH1zsC7rw5Dq1dlsFhVpkbXBc1AcYCUJkmP3mxR610IHovDGUb1pgTdl6D3JfT41d_XDvA-df4DKTWDlQ</recordid><startdate>201301</startdate><enddate>201301</enddate><creator>Haygood, T M</creator><creator>Ryan, J</creator><creator>Brennan, P C</creator><creator>Li, S</creator><creator>Marom, E M</creator><creator>McEntee, M F</creator><creator>Itani, M</creator><creator>Evanoff, M</creator><creator>Chakraborty, D</creator><general>The British Institute of Radiology</general><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>7X8</scope><scope>5PM</scope></search><sort><creationdate>201301</creationdate><title>On the choice of acceptance radius in free-response observer performance studies</title><author>Haygood, T M ; Ryan, J ; Brennan, P C ; Li, S ; Marom, E M ; McEntee, M F ; Itani, M ; Evanoff, M ; Chakraborty, D</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c384t-1e7a8f07d04b72b0b79f23a45571370d21a5c413c2a9697552d4dc2b56f2b37c3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2013</creationdate><topic>Humans</topic><topic>MIPS special feature</topic><topic>Observer Variation</topic><topic>Radiographic Image Enhancement - methods</topic><topic>Radiographic Image Interpretation, Computer-Assisted - methods</topic><topic>Radiography, Thoracic - methods</topic><topic>Reproducibility of Results</topic><topic>ROC Curve</topic><topic>Sensitivity and Specificity</topic><topic>Solitary Pulmonary Nodule - diagnostic imaging</topic><topic>Tomography, X-Ray Computed - adverse effects</topic><topic>Tomography, X-Ray Computed - methods</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Haygood, T M</creatorcontrib><creatorcontrib>Ryan, J</creatorcontrib><creatorcontrib>Brennan, P C</creatorcontrib><creatorcontrib>Li, S</creatorcontrib><creatorcontrib>Marom, E M</creatorcontrib><creatorcontrib>McEntee, M F</creatorcontrib><creatorcontrib>Itani, M</creatorcontrib><creatorcontrib>Evanoff, M</creatorcontrib><creatorcontrib>Chakraborty, D</creatorcontrib><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><jtitle>British journal of radiology</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Haygood, T M</au><au>Ryan, J</au><au>Brennan, P C</au><au>Li, S</au><au>Marom, E M</au><au>McEntee, M F</au><au>Itani, M</au><au>Evanoff, M</au><au>Chakraborty, D</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>On the choice of acceptance radius in free-response observer performance studies</atitle><jtitle>British journal of radiology</jtitle><addtitle>Br J Radiol</addtitle><date>2013-01</date><risdate>2013</risdate><volume>86</volume><issue>1021</issue><spage>42313554</spage><epage>42313554</epage><pages>42313554-42313554</pages><issn>0007-1285</issn><eissn>1748-880X</eissn><abstract><![CDATA[Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius.
80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of <5 pixels, <10 pixels, <20 pixels and onwards up to <200 and 200+ pixels. We counted lesion localisations in each bin and visually compared marks with the borders of nodules.
Most reader marks were tightly clustered around nodule centres, with tighter clustering for smaller than for larger nodules. At least 70% of readers' marks were placed within <10 pixels for small nodules, <20 pixels for medium nodules and <30 pixels for large nodules. Of 72 inspected marks that were less than 50 pixels from the centre of a nodule, only 1 fell outside the border of a nodule.
The acceptance radius should be based on the larger nodule sizes. For our data, an acceptance radius of 50 pixels would have captured all but 2 reader marks within the borders of a nodule, while excluding only 1 true-positive mark. The choice of an acceptance radius for FROC analysis of observer performance studies should be based on the size of larger abnormalities.]]></abstract><cop>England</cop><pub>The British Institute of Radiology</pub><pmid>22573302</pmid><doi>10.1259/bjr/42313554</doi><tpages>1</tpages><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 0007-1285 |
| ispartof | British journal of radiology, 2013-01, Vol.86 (1021), p.42313554-42313554 |
| issn | 0007-1285 1748-880X |
| language | eng |
| recordid | cdi_pubmedcentral_primary_oai_pubmedcentral_nih_gov_3615402 |
| source | MEDLINE; Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals; Oxford University Press Journals All Titles (1996-Current) |
| subjects | Humans MIPS special feature Observer Variation Radiographic Image Enhancement - methods Radiographic Image Interpretation, Computer-Assisted - methods Radiography, Thoracic - methods Reproducibility of Results ROC Curve Sensitivity and Specificity Solitary Pulmonary Nodule - diagnostic imaging Tomography, X-Ray Computed - adverse effects Tomography, X-Ray Computed - methods |
| title | On the choice of acceptance radius in free-response observer performance studies |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-15T12%3A04%3A46IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_pubme&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=On%20the%20choice%20of%20acceptance%20radius%20in%20free-response%20observer%20performance%20studies&rft.jtitle=British%20journal%20of%20radiology&rft.au=Haygood,%20T%20M&rft.date=2013-01&rft.volume=86&rft.issue=1021&rft.spage=42313554&rft.epage=42313554&rft.pages=42313554-42313554&rft.issn=0007-1285&rft.eissn=1748-880X&rft_id=info:doi/10.1259/bjr/42313554&rft_dat=%3Cproquest_pubme%3E1273229088%3C/proquest_pubme%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1273229088&rft_id=info:pmid/22573302&rfr_iscdi=true |