Towards Task and Architecture-Independent Generalization Gap Predictors

Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectures, on different variations of spiral datasets, and using different optimization parameters. We used this dataset to tr...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Yak, Scott, Gonzalvo, Javier, Mazzawi, Hanna
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Yak, Scott
Gonzalvo, Javier
Mazzawi, Hanna
description Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectures, on different variations of spiral datasets, and using different optimization parameters. We used this dataset to train task-independent and architecture-independent generalization gap predictors for those neural networks. We extend Jiang et al. (2018) to also use DNNs and RNNs and show that they outperform the linear model, obtaining $R^2=0.965$. We also show results for architecture-independent, task-independent, and out-of-distribution generalization gap prediction tasks. Both DNNs and RNNs consistently and significantly outperform linear models, with RNNs obtaining $R^2=0.584$.
doi_str_mv 10.48550/arxiv.1906.01550
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_1906_01550</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>1906_01550</sourcerecordid><originalsourceid>FETCH-LOGICAL-a670-18848b25f9c1c6eaa5677cf6919e1be535e752c0fa828099ad501e90f6b8a15d3</originalsourceid><addsrcrecordid>eNotz71OwzAUBWAvDKjwAEz4BZL6JrVjj1VVQqVKdMge3djXwqI4kWN-2qenFJZzpDMc6WPsAUS50lKKJabv8FmCEaoUcBluWduNX5jczDuc3zhGx9fJvoZMNn8kKnbR0USXiJm3FCnhMZwxhzHyFid-SOSCzWOa79iNx-NM9_-9YN3Ttts8F_uXdrdZ7wtUjShA65UeKumNBasIUaqmsV4ZMAQDyVpSIysrPOpKC2PQSQFkhFeDRpCuXrDHv9urpJ9SeMd06n9F_VVU_wB3PUaC</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Towards Task and Architecture-Independent Generalization Gap Predictors</title><source>arXiv.org</source><creator>Yak, Scott ; Gonzalvo, Javier ; Mazzawi, Hanna</creator><creatorcontrib>Yak, Scott ; Gonzalvo, Javier ; Mazzawi, Hanna</creatorcontrib><description>Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectures, on different variations of spiral datasets, and using different optimization parameters. We used this dataset to train task-independent and architecture-independent generalization gap predictors for those neural networks. We extend Jiang et al. (2018) to also use DNNs and RNNs and show that they outperform the linear model, obtaining $R^2=0.965$. We also show results for architecture-independent, task-independent, and out-of-distribution generalization gap prediction tasks. Both DNNs and RNNs consistently and significantly outperform linear models, with RNNs obtaining $R^2=0.584$.</description><identifier>DOI: 10.48550/arxiv.1906.01550</identifier><language>eng</language><subject>Computer Science - Learning ; Statistics - Machine Learning</subject><creationdate>2019-06</creationdate><rights>http://creativecommons.org/licenses/by-nc-sa/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,777,882</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/1906.01550$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.1906.01550$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Yak, Scott</creatorcontrib><creatorcontrib>Gonzalvo, Javier</creatorcontrib><creatorcontrib>Mazzawi, Hanna</creatorcontrib><title>Towards Task and Architecture-Independent Generalization Gap Predictors</title><description>Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectures, on different variations of spiral datasets, and using different optimization parameters. We used this dataset to train task-independent and architecture-independent generalization gap predictors for those neural networks. We extend Jiang et al. (2018) to also use DNNs and RNNs and show that they outperform the linear model, obtaining $R^2=0.965$. We also show results for architecture-independent, task-independent, and out-of-distribution generalization gap prediction tasks. Both DNNs and RNNs consistently and significantly outperform linear models, with RNNs obtaining $R^2=0.584$.</description><subject>Computer Science - Learning</subject><subject>Statistics - Machine Learning</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2019</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotz71OwzAUBWAvDKjwAEz4BZL6JrVjj1VVQqVKdMge3djXwqI4kWN-2qenFJZzpDMc6WPsAUS50lKKJabv8FmCEaoUcBluWduNX5jczDuc3zhGx9fJvoZMNn8kKnbR0USXiJm3FCnhMZwxhzHyFid-SOSCzWOa79iNx-NM9_-9YN3Ttts8F_uXdrdZ7wtUjShA65UeKumNBasIUaqmsV4ZMAQDyVpSIysrPOpKC2PQSQFkhFeDRpCuXrDHv9urpJ9SeMd06n9F_VVU_wB3PUaC</recordid><startdate>20190604</startdate><enddate>20190604</enddate><creator>Yak, Scott</creator><creator>Gonzalvo, Javier</creator><creator>Mazzawi, Hanna</creator><scope>AKY</scope><scope>EPD</scope><scope>GOX</scope></search><sort><creationdate>20190604</creationdate><title>Towards Task and Architecture-Independent Generalization Gap Predictors</title><author>Yak, Scott ; Gonzalvo, Javier ; Mazzawi, Hanna</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a670-18848b25f9c1c6eaa5677cf6919e1be535e752c0fa828099ad501e90f6b8a15d3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2019</creationdate><topic>Computer Science - Learning</topic><topic>Statistics - Machine Learning</topic><toplevel>online_resources</toplevel><creatorcontrib>Yak, Scott</creatorcontrib><creatorcontrib>Gonzalvo, Javier</creatorcontrib><creatorcontrib>Mazzawi, Hanna</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv Statistics</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Yak, Scott</au><au>Gonzalvo, Javier</au><au>Mazzawi, Hanna</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Towards Task and Architecture-Independent Generalization Gap Predictors</atitle><date>2019-06-04</date><risdate>2019</risdate><abstract>Can we use deep learning to predict when deep learning works? Our results suggest the affirmative. We created a dataset by training 13,500 neural networks with different architectures, on different variations of spiral datasets, and using different optimization parameters. We used this dataset to train task-independent and architecture-independent generalization gap predictors for those neural networks. We extend Jiang et al. (2018) to also use DNNs and RNNs and show that they outperform the linear model, obtaining $R^2=0.965$. We also show results for architecture-independent, task-independent, and out-of-distribution generalization gap prediction tasks. Both DNNs and RNNs consistently and significantly outperform linear models, with RNNs obtaining $R^2=0.584$.</abstract><doi>10.48550/arxiv.1906.01550</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.1906.01550
ispartof
issn
language eng
recordid cdi_arxiv_primary_1906_01550
source arXiv.org
subjects Computer Science - Learning
Statistics - Machine Learning
title Towards Task and Architecture-Independent Generalization Gap Predictors
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-20T12%3A59%3A10IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Towards%20Task%20and%20Architecture-Independent%20Generalization%20Gap%20Predictors&rft.au=Yak,%20Scott&rft.date=2019-06-04&rft_id=info:doi/10.48550/arxiv.1906.01550&rft_dat=%3Carxiv_GOX%3E1906_01550%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true