“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers
Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically sugg...
Gespeichert in:
| Veröffentlicht in: | ACM transactions on computer-human interaction 2023-11, Vol.31 (1), p.1-31, Article 4 |
|---|---|
| Hauptverfasser: | , , , , , , , , |
| Format: | Artikel |
| Sprache: | eng |
| Schlagworte: | |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| container_end_page | 31 |
|---|---|
| container_issue | 1 |
| container_start_page | 1 |
| container_title | ACM transactions on computer-human interaction |
| container_volume | 31 |
| creator | Prather, James Reeves, Brent N. Denny, Paul Becker, Brett A. Leinonen, Juho Luxton-Reilly, Andrew Powell, Garrett Finnie-Ansley, James Santos, Eddie Antonio |
| description | Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience. |
| doi_str_mv | 10.1145/3617367 |
| format | Article |
| fullrecord | <record><control><sourceid>acm_cross</sourceid><recordid>TN_cdi_crossref_primary_10_1145_3617367</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>3617367</sourcerecordid><originalsourceid>FETCH-LOGICAL-a1927-fe1ed348ae6f891f135e9b2aa2655e1bde173b6c3ff513ed2d6f798da190756d3</originalsourceid><addsrcrecordid>eNo9kLFOwzAQhi0EEqUgdiZvTAFfXNsNG6ooRFTA0Kpj5MQ2NUriyraouvU1kODl-iSkamG6_-6--6X7EboEcgMwYLeUg6BcHKEeMCYSQVN23GkiaEIY8FN0FsIHIQQEH_RQvd1853G7-Qp4rq1XeLqQEduIn1u36ma7Lsdz2XbMzx2eBVna2sY1lq3CeRu1l1W0rg14ZeMCj9zS1i5i4zx-cZ-20vjNu3cvm0b7cI5OjKyDvjjUPpqNH6ajp2Ty-piP7ieJhCwVidGgFR0MpeZmmIEBynRWplKmnDENpdLdhyWvqDEMqFap4kZkQ9VdE8G4on10vfetvAvBa1MsvW2kXxdAil1IxSGkjrzak7Jq_qG_5S8uDmSy</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers</title><source>ACM Digital Library</source><creator>Prather, James ; Reeves, Brent N. ; Denny, Paul ; Becker, Brett A. ; Leinonen, Juho ; Luxton-Reilly, Andrew ; Powell, Garrett ; Finnie-Ansley, James ; Santos, Eddie Antonio</creator><creatorcontrib>Prather, James ; Reeves, Brent N. ; Denny, Paul ; Becker, Brett A. ; Leinonen, Juho ; Luxton-Reilly, Andrew ; Powell, Garrett ; Finnie-Ansley, James ; Santos, Eddie Antonio</creatorcontrib><description>Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.</description><identifier>ISSN: 1073-0516</identifier><identifier>EISSN: 1557-7325</identifier><identifier>DOI: 10.1145/3617367</identifier><language>eng</language><publisher>New York, NY: ACM</publisher><subject>Applied computing ; Artificial intelligence ; Computer science education ; Computing education ; Computing methodologies ; CS1 ; Education ; Empirical studies in HCI ; Human computer interaction (HCI) ; Human-centered computing ; Natural language interfaces ; Social and professional topics ; User interface programming ; User studies</subject><ispartof>ACM transactions on computer-human interaction, 2023-11, Vol.31 (1), p.1-31, Article 4</ispartof><rights>Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-a1927-fe1ed348ae6f891f135e9b2aa2655e1bde173b6c3ff513ed2d6f798da190756d3</citedby><cites>FETCH-LOGICAL-a1927-fe1ed348ae6f891f135e9b2aa2655e1bde173b6c3ff513ed2d6f798da190756d3</cites><orcidid>0000-0001-5781-1136 ; 0000-0001-8269-2909 ; 0000-0002-4279-6284 ; 0000-0002-5150-9806 ; 0000-0001-6829-9449 ; 0000-0001-5337-715X ; 0000-0003-2807-6042 ; 0000-0003-1446-647X ; 0000-0002-3221-7015</orcidid></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://dl.acm.org/doi/pdf/10.1145/3617367$$EPDF$$P50$$Gacm$$H</linktopdf><link.rule.ids>314,780,784,2282,27924,27925,40196,76228</link.rule.ids></links><search><creatorcontrib>Prather, James</creatorcontrib><creatorcontrib>Reeves, Brent N.</creatorcontrib><creatorcontrib>Denny, Paul</creatorcontrib><creatorcontrib>Becker, Brett A.</creatorcontrib><creatorcontrib>Leinonen, Juho</creatorcontrib><creatorcontrib>Luxton-Reilly, Andrew</creatorcontrib><creatorcontrib>Powell, Garrett</creatorcontrib><creatorcontrib>Finnie-Ansley, James</creatorcontrib><creatorcontrib>Santos, Eddie Antonio</creatorcontrib><title>“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers</title><title>ACM transactions on computer-human interaction</title><addtitle>ACM TOCHI</addtitle><description>Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.</description><subject>Applied computing</subject><subject>Artificial intelligence</subject><subject>Computer science education</subject><subject>Computing education</subject><subject>Computing methodologies</subject><subject>CS1</subject><subject>Education</subject><subject>Empirical studies in HCI</subject><subject>Human computer interaction (HCI)</subject><subject>Human-centered computing</subject><subject>Natural language interfaces</subject><subject>Social and professional topics</subject><subject>User interface programming</subject><subject>User studies</subject><issn>1073-0516</issn><issn>1557-7325</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2023</creationdate><recordtype>article</recordtype><recordid>eNo9kLFOwzAQhi0EEqUgdiZvTAFfXNsNG6ooRFTA0Kpj5MQ2NUriyraouvU1kODl-iSkamG6_-6--6X7EboEcgMwYLeUg6BcHKEeMCYSQVN23GkiaEIY8FN0FsIHIQQEH_RQvd1853G7-Qp4rq1XeLqQEduIn1u36ma7Lsdz2XbMzx2eBVna2sY1lq3CeRu1l1W0rg14ZeMCj9zS1i5i4zx-cZ-20vjNu3cvm0b7cI5OjKyDvjjUPpqNH6ajp2Ty-piP7ieJhCwVidGgFR0MpeZmmIEBynRWplKmnDENpdLdhyWvqDEMqFap4kZkQ9VdE8G4on10vfetvAvBa1MsvW2kXxdAil1IxSGkjrzak7Jq_qG_5S8uDmSy</recordid><startdate>20231129</startdate><enddate>20231129</enddate><creator>Prather, James</creator><creator>Reeves, Brent N.</creator><creator>Denny, Paul</creator><creator>Becker, Brett A.</creator><creator>Leinonen, Juho</creator><creator>Luxton-Reilly, Andrew</creator><creator>Powell, Garrett</creator><creator>Finnie-Ansley, James</creator><creator>Santos, Eddie Antonio</creator><general>ACM</general><scope>AAYXX</scope><scope>CITATION</scope><orcidid>https://orcid.org/0000-0001-5781-1136</orcidid><orcidid>https://orcid.org/0000-0001-8269-2909</orcidid><orcidid>https://orcid.org/0000-0002-4279-6284</orcidid><orcidid>https://orcid.org/0000-0002-5150-9806</orcidid><orcidid>https://orcid.org/0000-0001-6829-9449</orcidid><orcidid>https://orcid.org/0000-0001-5337-715X</orcidid><orcidid>https://orcid.org/0000-0003-2807-6042</orcidid><orcidid>https://orcid.org/0000-0003-1446-647X</orcidid><orcidid>https://orcid.org/0000-0002-3221-7015</orcidid></search><sort><creationdate>20231129</creationdate><title>“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers</title><author>Prather, James ; Reeves, Brent N. ; Denny, Paul ; Becker, Brett A. ; Leinonen, Juho ; Luxton-Reilly, Andrew ; Powell, Garrett ; Finnie-Ansley, James ; Santos, Eddie Antonio</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a1927-fe1ed348ae6f891f135e9b2aa2655e1bde173b6c3ff513ed2d6f798da190756d3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2023</creationdate><topic>Applied computing</topic><topic>Artificial intelligence</topic><topic>Computer science education</topic><topic>Computing education</topic><topic>Computing methodologies</topic><topic>CS1</topic><topic>Education</topic><topic>Empirical studies in HCI</topic><topic>Human computer interaction (HCI)</topic><topic>Human-centered computing</topic><topic>Natural language interfaces</topic><topic>Social and professional topics</topic><topic>User interface programming</topic><topic>User studies</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Prather, James</creatorcontrib><creatorcontrib>Reeves, Brent N.</creatorcontrib><creatorcontrib>Denny, Paul</creatorcontrib><creatorcontrib>Becker, Brett A.</creatorcontrib><creatorcontrib>Leinonen, Juho</creatorcontrib><creatorcontrib>Luxton-Reilly, Andrew</creatorcontrib><creatorcontrib>Powell, Garrett</creatorcontrib><creatorcontrib>Finnie-Ansley, James</creatorcontrib><creatorcontrib>Santos, Eddie Antonio</creatorcontrib><collection>CrossRef</collection><jtitle>ACM transactions on computer-human interaction</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Prather, James</au><au>Reeves, Brent N.</au><au>Denny, Paul</au><au>Becker, Brett A.</au><au>Leinonen, Juho</au><au>Luxton-Reilly, Andrew</au><au>Powell, Garrett</au><au>Finnie-Ansley, James</au><au>Santos, Eddie Antonio</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>“It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers</atitle><jtitle>ACM transactions on computer-human interaction</jtitle><stitle>ACM TOCHI</stitle><date>2023-11-29</date><risdate>2023</risdate><volume>31</volume><issue>1</issue><spage>1</spage><epage>31</epage><pages>1-31</pages><artnum>4</artnum><issn>1073-0516</issn><eissn>1557-7325</eissn><abstract>Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.</abstract><cop>New York, NY</cop><pub>ACM</pub><doi>10.1145/3617367</doi><tpages>31</tpages><orcidid>https://orcid.org/0000-0001-5781-1136</orcidid><orcidid>https://orcid.org/0000-0001-8269-2909</orcidid><orcidid>https://orcid.org/0000-0002-4279-6284</orcidid><orcidid>https://orcid.org/0000-0002-5150-9806</orcidid><orcidid>https://orcid.org/0000-0001-6829-9449</orcidid><orcidid>https://orcid.org/0000-0001-5337-715X</orcidid><orcidid>https://orcid.org/0000-0003-2807-6042</orcidid><orcidid>https://orcid.org/0000-0003-1446-647X</orcidid><orcidid>https://orcid.org/0000-0002-3221-7015</orcidid><oa>free_for_read</oa></addata></record> |
| fulltext | fulltext |
| identifier | ISSN: 1073-0516 |
| ispartof | ACM transactions on computer-human interaction, 2023-11, Vol.31 (1), p.1-31, Article 4 |
| issn | 1073-0516 1557-7325 |
| language | eng |
| recordid | cdi_crossref_primary_10_1145_3617367 |
| source | ACM Digital Library |
| subjects | Applied computing Artificial intelligence Computer science education Computing education Computing methodologies CS1 Education Empirical studies in HCI Human computer interaction (HCI) Human-centered computing Natural language interfaces Social and professional topics User interface programming User studies |
| title | “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers |
| url | https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-03T00%3A57%3A48IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-acm_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=%E2%80%9CIt%E2%80%99s%20Weird%20That%20it%20Knows%20What%20I%20Want%E2%80%9D:%20Usability%20and%20Interactions%20with%20Copilot%20for%20Novice%20Programmers&rft.jtitle=ACM%20transactions%20on%20computer-human%20interaction&rft.au=Prather,%20James&rft.date=2023-11-29&rft.volume=31&rft.issue=1&rft.spage=1&rft.epage=31&rft.pages=1-31&rft.artnum=4&rft.issn=1073-0516&rft.eissn=1557-7325&rft_id=info:doi/10.1145/3617367&rft_dat=%3Cacm_cross%3E3617367%3C/acm_cross%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true |