Reconstructing speech from human auditory cortex

How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstru...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:PLoS biology 2012-01, Vol.10 (1), p.e1001251-e1001251
Hauptverfasser: Pasley, Brian N, David, Stephen V, Mesgarani, Nima, Flinker, Adeen, Shamma, Shihab A, Crone, Nathan E, Knight, Robert T, Chang, Edward F
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page e1001251
container_issue 1
container_start_page e1001251
container_title PLoS biology
container_volume 10
creator Pasley, Brian N
David, Stephen V
Mesgarani, Nima
Flinker, Adeen
Shamma, Shihab A
Crone, Nathan E
Knight, Robert T
Chang, Edward F
description How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.
doi_str_mv 10.1371/journal.pbio.1001251
format Article
fullrecord <record><control><sourceid>gale_plos_</sourceid><recordid>TN_cdi_plos_journals_1303361270</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><galeid>A279722281</galeid><doaj_id>oai_doaj_org_article_825de0b5af424181b4ccf7bb930283b0</doaj_id><sourcerecordid>A279722281</sourcerecordid><originalsourceid>FETCH-LOGICAL-c760t-7c7f752c2e634c9ebb76c510395e12fb2019cfa3ff481ed834b647c1d85054293</originalsourceid><addsrcrecordid>eNqVkltrFDEUx4Motq5-A9EFH8SHXXOdJC9CKV4WioV6eQ1JJpnNMjNZkxmx396sOy0d6YOSh4ST3_mfKwDPEVwjwtHbXRxTr9v13oS4RhAizNADcIoYZSsuBHt4530CnuS8gxBjicVjcIIxgQQLdArglbOxz0Ma7RD6Zpn3ztnt0qfYLbdjp_ulHuswxHS9tDEN7tdT8MjrNrtn070A3z68_3r-aXVx-XFzfnaxsryCw4pb7jnDFruKUCudMbyyDEEimUPYGwyRtF4T76lArhaEmopyi2rBIKNYkgV4edTdtzGrqdisUEmcVAhzWIjNkaij3ql9Cp1O1yrqoP4YYmqUTkOwrVMCs9pBw7SnmCKBDLXWc2MkgVgQc9B6N0UbTedq6_oh6XYmOv_pw1Y18aciuJK0tHMBXk8CKf4YXR5UF7J1bat7F8esJJKSMQFZIV_9Rd5f3EQ1uuQfeh9LWHvQVGeYS45xGV-h1vdQ5dSuC2Wuzodinzm8mTkUpsx0aPSYs9p8ufoP9vO_s5ff5yw9sjbFnJPzt21GUB1W-6Yh6rDaalrt4vbi7ohunW52mfwG0ufxnw</addsrcrecordid><sourcetype>Open Website</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>1303361270</pqid></control><display><type>article</type><title>Reconstructing speech from human auditory cortex</title><source>MEDLINE</source><source>DOAJ Directory of Open Access Journals</source><source>Public Library of Science (PLoS)</source><source>EZB-FREE-00999 freely available EZB journals</source><source>PubMed Central</source><creator>Pasley, Brian N ; David, Stephen V ; Mesgarani, Nima ; Flinker, Adeen ; Shamma, Shihab A ; Crone, Nathan E ; Knight, Robert T ; Chang, Edward F</creator><contributor>Zatorre, Robert</contributor><creatorcontrib>Pasley, Brian N ; David, Stephen V ; Mesgarani, Nima ; Flinker, Adeen ; Shamma, Shihab A ; Crone, Nathan E ; Knight, Robert T ; Chang, Edward F ; Zatorre, Robert</creatorcontrib><description>How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.</description><identifier>ISSN: 1545-7885</identifier><identifier>ISSN: 1544-9173</identifier><identifier>EISSN: 1545-7885</identifier><identifier>DOI: 10.1371/journal.pbio.1001251</identifier><identifier>PMID: 22303281</identifier><language>eng</language><publisher>United States: Public Library of Science</publisher><subject>Algorithms ; Auditory Cortex - physiology ; Biology ; Brain ; Brain Mapping ; Computer Simulation ; Ears &amp; hearing ; Electrodes, Implanted ; Electroencephalography ; Engineering ; Female ; Health aspects ; Humans ; Linear Models ; Male ; Models, Biological ; Neurology ; Phonetics ; Population ; Regression analysis ; Social and Behavioral Sciences ; Sound ; Speech ; Speech Acoustics ; Speech perception ; Studies ; Surgery</subject><ispartof>PLoS biology, 2012-01, Vol.10 (1), p.e1001251-e1001251</ispartof><rights>COPYRIGHT 2012 Public Library of Science</rights><rights>2012 Pasley et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited: Pasley BN, David SV, Mesgarani N, Flinker A, Shamma SA, et al. (2012) Reconstructing Speech from Human Auditory Cortex. PLoS Biol 10(1): e1001251. doi:10.1371/journal.pbio.1001251</rights><rights>Pasley et al. 2012</rights><lds50>peer_reviewed</lds50><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c760t-7c7f752c2e634c9ebb76c510395e12fb2019cfa3ff481ed834b647c1d85054293</citedby><cites>FETCH-LOGICAL-c760t-7c7f752c2e634c9ebb76c510395e12fb2019cfa3ff481ed834b647c1d85054293</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC3269422/pdf/$$EPDF$$P50$$Gpubmedcentral$$Hfree_for_read</linktopdf><linktohtml>$$Uhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC3269422/$$EHTML$$P50$$Gpubmedcentral$$Hfree_for_read</linktohtml><link.rule.ids>230,314,724,777,781,861,882,2096,2915,23847,27905,27906,53772,53774,79349,79350</link.rule.ids><backlink>$$Uhttps://www.ncbi.nlm.nih.gov/pubmed/22303281$$D View this record in MEDLINE/PubMed$$Hfree_for_read</backlink></links><search><contributor>Zatorre, Robert</contributor><creatorcontrib>Pasley, Brian N</creatorcontrib><creatorcontrib>David, Stephen V</creatorcontrib><creatorcontrib>Mesgarani, Nima</creatorcontrib><creatorcontrib>Flinker, Adeen</creatorcontrib><creatorcontrib>Shamma, Shihab A</creatorcontrib><creatorcontrib>Crone, Nathan E</creatorcontrib><creatorcontrib>Knight, Robert T</creatorcontrib><creatorcontrib>Chang, Edward F</creatorcontrib><title>Reconstructing speech from human auditory cortex</title><title>PLoS biology</title><addtitle>PLoS Biol</addtitle><description>How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.</description><subject>Algorithms</subject><subject>Auditory Cortex - physiology</subject><subject>Biology</subject><subject>Brain</subject><subject>Brain Mapping</subject><subject>Computer Simulation</subject><subject>Ears &amp; hearing</subject><subject>Electrodes, Implanted</subject><subject>Electroencephalography</subject><subject>Engineering</subject><subject>Female</subject><subject>Health aspects</subject><subject>Humans</subject><subject>Linear Models</subject><subject>Male</subject><subject>Models, Biological</subject><subject>Neurology</subject><subject>Phonetics</subject><subject>Population</subject><subject>Regression analysis</subject><subject>Social and Behavioral Sciences</subject><subject>Sound</subject><subject>Speech</subject><subject>Speech Acoustics</subject><subject>Speech perception</subject><subject>Studies</subject><subject>Surgery</subject><issn>1545-7885</issn><issn>1544-9173</issn><issn>1545-7885</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2012</creationdate><recordtype>article</recordtype><sourceid>EIF</sourceid><sourceid>ABUWG</sourceid><sourceid>AFKRA</sourceid><sourceid>AZQEC</sourceid><sourceid>BENPR</sourceid><sourceid>CCPQU</sourceid><sourceid>DWQXO</sourceid><sourceid>GNUQQ</sourceid><sourceid>DOA</sourceid><recordid>eNqVkltrFDEUx4Motq5-A9EFH8SHXXOdJC9CKV4WioV6eQ1JJpnNMjNZkxmx396sOy0d6YOSh4ST3_mfKwDPEVwjwtHbXRxTr9v13oS4RhAizNADcIoYZSsuBHt4530CnuS8gxBjicVjcIIxgQQLdArglbOxz0Ma7RD6Zpn3ztnt0qfYLbdjp_ulHuswxHS9tDEN7tdT8MjrNrtn070A3z68_3r-aXVx-XFzfnaxsryCw4pb7jnDFruKUCudMbyyDEEimUPYGwyRtF4T76lArhaEmopyi2rBIKNYkgV4edTdtzGrqdisUEmcVAhzWIjNkaij3ql9Cp1O1yrqoP4YYmqUTkOwrVMCs9pBw7SnmCKBDLXWc2MkgVgQc9B6N0UbTedq6_oh6XYmOv_pw1Y18aciuJK0tHMBXk8CKf4YXR5UF7J1bat7F8esJJKSMQFZIV_9Rd5f3EQ1uuQfeh9LWHvQVGeYS45xGV-h1vdQ5dSuC2Wuzodinzm8mTkUpsx0aPSYs9p8ufoP9vO_s5ff5yw9sjbFnJPzt21GUB1W-6Yh6rDaalrt4vbi7ohunW52mfwG0ufxnw</recordid><startdate>20120101</startdate><enddate>20120101</enddate><creator>Pasley, Brian N</creator><creator>David, Stephen V</creator><creator>Mesgarani, Nima</creator><creator>Flinker, Adeen</creator><creator>Shamma, Shihab A</creator><creator>Crone, Nathan E</creator><creator>Knight, Robert T</creator><creator>Chang, Edward F</creator><general>Public Library of Science</general><general>Public Library of Science (PLoS)</general><scope>CGR</scope><scope>CUY</scope><scope>CVF</scope><scope>ECM</scope><scope>EIF</scope><scope>NPM</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>IOV</scope><scope>ISN</scope><scope>ISR</scope><scope>3V.</scope><scope>7QG</scope><scope>7QL</scope><scope>7SN</scope><scope>7SS</scope><scope>7T5</scope><scope>7TK</scope><scope>7TM</scope><scope>7X7</scope><scope>7XB</scope><scope>88E</scope><scope>8FD</scope><scope>8FE</scope><scope>8FH</scope><scope>8FI</scope><scope>8FJ</scope><scope>8FK</scope><scope>ABUWG</scope><scope>AEUYN</scope><scope>AFKRA</scope><scope>ATCPS</scope><scope>AZQEC</scope><scope>BBNVY</scope><scope>BENPR</scope><scope>BHPHI</scope><scope>C1K</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>FR3</scope><scope>FYUFA</scope><scope>GHDGH</scope><scope>GNUQQ</scope><scope>H94</scope><scope>HCIFZ</scope><scope>K9.</scope><scope>LK8</scope><scope>M0S</scope><scope>M1P</scope><scope>M7N</scope><scope>M7P</scope><scope>P64</scope><scope>PATMY</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PYCSY</scope><scope>RC3</scope><scope>7X8</scope><scope>5PM</scope><scope>DOA</scope><scope>CZG</scope></search><sort><creationdate>20120101</creationdate><title>Reconstructing speech from human auditory cortex</title><author>Pasley, Brian N ; David, Stephen V ; Mesgarani, Nima ; Flinker, Adeen ; Shamma, Shihab A ; Crone, Nathan E ; Knight, Robert T ; Chang, Edward F</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c760t-7c7f752c2e634c9ebb76c510395e12fb2019cfa3ff481ed834b647c1d85054293</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2012</creationdate><topic>Algorithms</topic><topic>Auditory Cortex - physiology</topic><topic>Biology</topic><topic>Brain</topic><topic>Brain Mapping</topic><topic>Computer Simulation</topic><topic>Ears &amp; hearing</topic><topic>Electrodes, Implanted</topic><topic>Electroencephalography</topic><topic>Engineering</topic><topic>Female</topic><topic>Health aspects</topic><topic>Humans</topic><topic>Linear Models</topic><topic>Male</topic><topic>Models, Biological</topic><topic>Neurology</topic><topic>Phonetics</topic><topic>Population</topic><topic>Regression analysis</topic><topic>Social and Behavioral Sciences</topic><topic>Sound</topic><topic>Speech</topic><topic>Speech Acoustics</topic><topic>Speech perception</topic><topic>Studies</topic><topic>Surgery</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Pasley, Brian N</creatorcontrib><creatorcontrib>David, Stephen V</creatorcontrib><creatorcontrib>Mesgarani, Nima</creatorcontrib><creatorcontrib>Flinker, Adeen</creatorcontrib><creatorcontrib>Shamma, Shihab A</creatorcontrib><creatorcontrib>Crone, Nathan E</creatorcontrib><creatorcontrib>Knight, Robert T</creatorcontrib><creatorcontrib>Chang, Edward F</creatorcontrib><collection>Medline</collection><collection>MEDLINE</collection><collection>MEDLINE (Ovid)</collection><collection>MEDLINE</collection><collection>MEDLINE</collection><collection>PubMed</collection><collection>CrossRef</collection><collection>Gale In Context: Opposing Viewpoints</collection><collection>Gale In Context: Canada</collection><collection>Gale In Context: Science</collection><collection>ProQuest Central (Corporate)</collection><collection>Animal Behavior Abstracts</collection><collection>Bacteriology Abstracts (Microbiology B)</collection><collection>Ecology Abstracts</collection><collection>Entomology Abstracts (Full archive)</collection><collection>Immunology Abstracts</collection><collection>Neurosciences Abstracts</collection><collection>Nucleic Acids Abstracts</collection><collection>Health &amp; Medical Collection</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Medical Database (Alumni Edition)</collection><collection>Technology Research Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Natural Science Collection</collection><collection>Hospital Premium Collection</collection><collection>Hospital Premium Collection (Alumni Edition)</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>ProQuest Central (Alumni Edition)</collection><collection>ProQuest One Sustainability</collection><collection>ProQuest Central UK/Ireland</collection><collection>Agricultural &amp; Environmental Science Collection</collection><collection>ProQuest Central Essentials</collection><collection>Biological Science Collection</collection><collection>ProQuest Central</collection><collection>Natural Science Collection</collection><collection>Environmental Sciences and Pollution Management</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>Engineering Research Database</collection><collection>Health Research Premium Collection</collection><collection>Health Research Premium Collection (Alumni)</collection><collection>ProQuest Central Student</collection><collection>AIDS and Cancer Research Abstracts</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Health &amp; Medical Complete (Alumni)</collection><collection>ProQuest Biological Science Collection</collection><collection>Health &amp; Medical Collection (Alumni Edition)</collection><collection>Medical Database</collection><collection>Algology Mycology and Protozoology Abstracts (Microbiology C)</collection><collection>Biological Science Database</collection><collection>Biotechnology and BioEngineering Abstracts</collection><collection>Environmental Science Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Environmental Science Collection</collection><collection>Genetics Abstracts</collection><collection>MEDLINE - Academic</collection><collection>PubMed Central (Full Participant titles)</collection><collection>DOAJ Directory of Open Access Journals</collection><collection>PLoS Biology</collection><jtitle>PLoS biology</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Pasley, Brian N</au><au>David, Stephen V</au><au>Mesgarani, Nima</au><au>Flinker, Adeen</au><au>Shamma, Shihab A</au><au>Crone, Nathan E</au><au>Knight, Robert T</au><au>Chang, Edward F</au><au>Zatorre, Robert</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Reconstructing speech from human auditory cortex</atitle><jtitle>PLoS biology</jtitle><addtitle>PLoS Biol</addtitle><date>2012-01-01</date><risdate>2012</risdate><volume>10</volume><issue>1</issue><spage>e1001251</spage><epage>e1001251</epage><pages>e1001251-e1001251</pages><issn>1545-7885</issn><issn>1544-9173</issn><eissn>1545-7885</eissn><abstract>How the human auditory system extracts perceptually relevant acoustic features of speech is unknown. To address this question, we used intracranial recordings from nonprimary auditory cortex in the human superior temporal gyrus to determine what acoustic information in speech sounds can be reconstructed from population neural activity. We found that slow and intermediate temporal fluctuations, such as those corresponding to syllable rate, were accurately reconstructed using a linear model based on the auditory spectrogram. However, reconstruction of fast temporal fluctuations, such as syllable onsets and offsets, required a nonlinear sound representation based on temporal modulation energy. Reconstruction accuracy was highest within the range of spectro-temporal fluctuations that have been found to be critical for speech intelligibility. The decoded speech representations allowed readout and identification of individual words directly from brain activity during single trial sound presentations. These findings reveal neural encoding mechanisms of speech acoustic parameters in higher order human auditory cortex.</abstract><cop>United States</cop><pub>Public Library of Science</pub><pmid>22303281</pmid><doi>10.1371/journal.pbio.1001251</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext
identifier ISSN: 1545-7885
ispartof PLoS biology, 2012-01, Vol.10 (1), p.e1001251-e1001251
issn 1545-7885
1544-9173
1545-7885
language eng
recordid cdi_plos_journals_1303361270
source MEDLINE; DOAJ Directory of Open Access Journals; Public Library of Science (PLoS); EZB-FREE-00999 freely available EZB journals; PubMed Central
subjects Algorithms
Auditory Cortex - physiology
Biology
Brain
Brain Mapping
Computer Simulation
Ears & hearing
Electrodes, Implanted
Electroencephalography
Engineering
Female
Health aspects
Humans
Linear Models
Male
Models, Biological
Neurology
Phonetics
Population
Regression analysis
Social and Behavioral Sciences
Sound
Speech
Speech Acoustics
Speech perception
Studies
Surgery
title Reconstructing speech from human auditory cortex
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-19T20%3A54%3A13IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-gale_plos_&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Reconstructing%20speech%20from%20human%20auditory%20cortex&rft.jtitle=PLoS%20biology&rft.au=Pasley,%20Brian%20N&rft.date=2012-01-01&rft.volume=10&rft.issue=1&rft.spage=e1001251&rft.epage=e1001251&rft.pages=e1001251-e1001251&rft.issn=1545-7885&rft.eissn=1545-7885&rft_id=info:doi/10.1371/journal.pbio.1001251&rft_dat=%3Cgale_plos_%3EA279722281%3C/gale_plos_%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_pqid=1303361270&rft_id=info:pmid/22303281&rft_galeid=A279722281&rft_doaj_id=oai_doaj_org_article_825de0b5af424181b4ccf7bb930283b0&rfr_iscdi=true