Honest calibration assessment for binary outcome predictions

Summary Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability $x$, it should materialize with approximately that frequency, which means that the so-called calibration curve $p(\cdot)$ should equal the id...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Biometrika 2023-09, Vol.110 (3), p.663-680
Hauptverfasser: Dimitriadis, Timo, Dümbgen, Lutz, Henzi, Alexander, Puke, Marius, Ziegel, Johanna
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page 680
container_issue 3
container_start_page 663
container_title Biometrika
container_volume 110
creator Dimitriadis, Timo
Dümbgen, Lutz
Henzi, Alexander
Puke, Marius
Ziegel, Johanna
description Summary Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability $x$, it should materialize with approximately that frequency, which means that the so-called calibration curve $p(\cdot)$ should equal the identity, i.e., $p(x) = x$ for all $x$ in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid subject to only the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well-specified model. We show that our bands have a finite-sample coverage guarantee, are narrower than those of existing approaches, and adapt to the local smoothness of the calibration curve $p$ and the local variance of the binary observations. In an application to modelling predictions of an infant having low birth weight, the bounds give informative insights into model calibration.
doi_str_mv 10.1093/biomet/asac068
format Article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_3049131813</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><oup_id>10.1093/biomet/asac068</oup_id><sourcerecordid>3049131813</sourcerecordid><originalsourceid>FETCH-LOGICAL-c341t-3a36a5a4eb8d48e8d991397bc5a51d9ae2b7b616b60ae22fb2a96d8f7224ee373</originalsourceid><addsrcrecordid>eNqFkEFLxDAQhYMoWFevngOePHQ3adI0BS-yuK6w4EXPYZKmkGXb1CQ9-O_N0r17mhn43pvHQ-iRkjUlLdto5webNhDBECGvUEG54CWrKblGBSFElIxzfovuYjyeT1GLAr3s_WhjwgZOTgdIzo8YYrQxDnZMuPcBazdC-MV-TiY_wFOwnTNnMN6jmx5O0T5c5gp9796-tvvy8Pn-sX09lIZxmkoGTEAN3GrZcWll17aUtY02NdS0a8FWutGCCi1I3qteV9CKTvZNVXFrWcNW6GnxnYL_mXNcdfRzGPNLxQjPZlRSlqn1QpngYwy2V1NwQ46uKFHnhtTSkLo0lAXPi8DP03_sH-qdap0</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>3049131813</pqid></control><display><type>article</type><title>Honest calibration assessment for binary outcome predictions</title><source>Oxford University Press Journals All Titles (1996-Current)</source><creator>Dimitriadis, Timo ; Dümbgen, Lutz ; Henzi, Alexander ; Puke, Marius ; Ziegel, Johanna</creator><creatorcontrib>Dimitriadis, Timo ; Dümbgen, Lutz ; Henzi, Alexander ; Puke, Marius ; Ziegel, Johanna</creatorcontrib><description>Summary Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability $x$, it should materialize with approximately that frequency, which means that the so-called calibration curve $p(\cdot)$ should equal the identity, i.e., $p(x) = x$ for all $x$ in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid subject to only the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well-specified model. We show that our bands have a finite-sample coverage guarantee, are narrower than those of existing approaches, and adapt to the local smoothness of the calibration curve $p$ and the local variance of the binary observations. In an application to modelling predictions of an infant having low birth weight, the bounds give informative insights into model calibration.</description><identifier>ISSN: 0006-3444</identifier><identifier>EISSN: 1464-3510</identifier><identifier>DOI: 10.1093/biomet/asac068</identifier><language>eng</language><publisher>Oxford: Oxford University Press</publisher><subject>Birth weight ; Calibration ; Goodness of fit ; Isotonicity ; Low birth weight ; Machine learning ; Predictions ; Smoothness ; Statistical tests</subject><ispartof>Biometrika, 2023-09, Vol.110 (3), p.663-680</ispartof><rights>The Author(s) 2022. Published by Oxford University Press on behalf of the Biometrika Trust. All rights reserved. For permissions, please email: journals.permissions@oup.com 2022</rights><rights>The Author(s) 2022. Published by Oxford University Press on behalf of the Biometrika Trust. All rights reserved. For permissions, please email: journals.permissions@oup.com</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c341t-3a36a5a4eb8d48e8d991397bc5a51d9ae2b7b616b60ae22fb2a96d8f7224ee373</citedby><cites>FETCH-LOGICAL-c341t-3a36a5a4eb8d48e8d991397bc5a51d9ae2b7b616b60ae22fb2a96d8f7224ee373</cites><orcidid>0000-0002-8322-360X ; 0000-0002-5916-9746</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>314,776,780,1578,27901,27902</link.rule.ids></links><search><creatorcontrib>Dimitriadis, Timo</creatorcontrib><creatorcontrib>Dümbgen, Lutz</creatorcontrib><creatorcontrib>Henzi, Alexander</creatorcontrib><creatorcontrib>Puke, Marius</creatorcontrib><creatorcontrib>Ziegel, Johanna</creatorcontrib><title>Honest calibration assessment for binary outcome predictions</title><title>Biometrika</title><description>Summary Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability $x$, it should materialize with approximately that frequency, which means that the so-called calibration curve $p(\cdot)$ should equal the identity, i.e., $p(x) = x$ for all $x$ in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid subject to only the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well-specified model. We show that our bands have a finite-sample coverage guarantee, are narrower than those of existing approaches, and adapt to the local smoothness of the calibration curve $p$ and the local variance of the binary observations. In an application to modelling predictions of an infant having low birth weight, the bounds give informative insights into model calibration.</description><subject>Birth weight</subject><subject>Calibration</subject><subject>Goodness of fit</subject><subject>Isotonicity</subject><subject>Low birth weight</subject><subject>Machine learning</subject><subject>Predictions</subject><subject>Smoothness</subject><subject>Statistical tests</subject><issn>0006-3444</issn><issn>1464-3510</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><recordid>eNqFkEFLxDAQhYMoWFevngOePHQ3adI0BS-yuK6w4EXPYZKmkGXb1CQ9-O_N0r17mhn43pvHQ-iRkjUlLdto5webNhDBECGvUEG54CWrKblGBSFElIxzfovuYjyeT1GLAr3s_WhjwgZOTgdIzo8YYrQxDnZMuPcBazdC-MV-TiY_wFOwnTNnMN6jmx5O0T5c5gp9796-tvvy8Pn-sX09lIZxmkoGTEAN3GrZcWll17aUtY02NdS0a8FWutGCCi1I3qteV9CKTvZNVXFrWcNW6GnxnYL_mXNcdfRzGPNLxQjPZlRSlqn1QpngYwy2V1NwQ46uKFHnhtTSkLo0lAXPi8DP03_sH-qdap0</recordid><startdate>20230901</startdate><enddate>20230901</enddate><creator>Dimitriadis, Timo</creator><creator>Dümbgen, Lutz</creator><creator>Henzi, Alexander</creator><creator>Puke, Marius</creator><creator>Ziegel, Johanna</creator><general>Oxford University Press</general><general>Oxford Publishing Limited (England)</general><scope>AAYXX</scope><scope>CITATION</scope><scope>7QO</scope><scope>8FD</scope><scope>FR3</scope><scope>P64</scope><orcidid>https://orcid.org/0000-0002-8322-360X</orcidid><orcidid>https://orcid.org/0000-0002-5916-9746</orcidid></search><sort><creationdate>20230901</creationdate><title>Honest calibration assessment for binary outcome predictions</title><author>Dimitriadis, Timo ; Dümbgen, Lutz ; Henzi, Alexander ; Puke, Marius ; Ziegel, Johanna</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c341t-3a36a5a4eb8d48e8d991397bc5a51d9ae2b7b616b60ae22fb2a96d8f7224ee373</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Birth weight</topic><topic>Calibration</topic><topic>Goodness of fit</topic><topic>Isotonicity</topic><topic>Low birth weight</topic><topic>Machine learning</topic><topic>Predictions</topic><topic>Smoothness</topic><topic>Statistical tests</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Dimitriadis, Timo</creatorcontrib><creatorcontrib>Dümbgen, Lutz</creatorcontrib><creatorcontrib>Henzi, Alexander</creatorcontrib><creatorcontrib>Puke, Marius</creatorcontrib><creatorcontrib>Ziegel, Johanna</creatorcontrib><collection>CrossRef</collection><collection>Biotechnology Research Abstracts</collection><collection>Technology Research Database</collection><collection>Engineering Research Database</collection><collection>Biotechnology and BioEngineering Abstracts</collection><jtitle>Biometrika</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Dimitriadis, Timo</au><au>Dümbgen, Lutz</au><au>Henzi, Alexander</au><au>Puke, Marius</au><au>Ziegel, Johanna</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Honest calibration assessment for binary outcome predictions</atitle><jtitle>Biometrika</jtitle><date>2023-09-01</date><risdate>2023</risdate><volume>110</volume><issue>3</issue><spage>663</spage><epage>680</epage><pages>663-680</pages><issn>0006-3444</issn><eissn>1464-3510</eissn><abstract>Summary Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability $x$, it should materialize with approximately that frequency, which means that the so-called calibration curve $p(\cdot)$ should equal the identity, i.e., $p(x) = x$ for all $x$ in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid subject to only the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well-specified model. We show that our bands have a finite-sample coverage guarantee, are narrower than those of existing approaches, and adapt to the local smoothness of the calibration curve $p$ and the local variance of the binary observations. In an application to modelling predictions of an infant having low birth weight, the bounds give informative insights into model calibration.</abstract><cop>Oxford</cop><pub>Oxford University Press</pub><doi>10.1093/biomet/asac068</doi><tpages>18</tpages><orcidid>https://orcid.org/0000-0002-8322-360X</orcidid><orcidid>https://orcid.org/0000-0002-5916-9746</orcidid><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 0006-3444
ispartof Biometrika, 2023-09, Vol.110 (3), p.663-680
issn 0006-3444
1464-3510
language eng
recordid cdi_proquest_journals_3049131813
source Oxford University Press Journals All Titles (1996-Current)
subjects Birth weight
Calibration
Goodness of fit
Isotonicity
Low birth weight
Machine learning
Predictions
Smoothness
Statistical tests
title Honest calibration assessment for binary outcome predictions
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-30T23%3A32%3A57IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Honest%20calibration%20assessment%20for%20binary%20outcome%20predictions&rft.jtitle=Biometrika&rft.au=Dimitriadis,%20Timo&rft.date=2023-09-01&rft.volume=110&rft.issue=3&rft.spage=663&rft.epage=680&rft.pages=663-680&rft.issn=0006-3444&rft.eissn=1464-3510&rft_id=info:doi/10.1093/biomet/asac068&rft_dat=%3Cproquest_cross%3E3049131813%3C/proquest_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=3049131813&rft_id=info:pmid/&rft_oup_id=10.1093/biomet/asac068&rfr_iscdi=true