Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence

Findings of ACL 2021 Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science settin...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Halterman, Andrew, Keith, Katherine A, Sarwar, Sheikh Muhammad, O'Connor, Brendan
Format: Artikel
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Halterman, Andrew
Keith, Katherine A
Sarwar, Sheikh Muhammad
O'Connor, Brendan
description Findings of ACL 2021 Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage retrieval tasks. Our novel corpus-level evaluations and annotation approach can guide creation of similar social-science-oriented resources in the future.
doi_str_mv 10.48550/arxiv.2105.12936
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2105_12936</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2105_12936</sourcerecordid><originalsourceid>FETCH-LOGICAL-a676-e3ca6a194330f90620b9f93af6c7821329e6bb2659788aee79c79224eada68243</originalsourceid><addsrcrecordid>eNotj81KxDAUhbNxIaMP4Mq8QGt60yaNu6GM40BhFIrbctu50UhthvQHfXtrx9XhcH7gY-wuEXGaZ5l4wPDt5hgSkcUJGKmuGRU-nKchKmmmju9m7CYcne-59WGx1I_8dfvIqw_ih_7k8MV3rqU1GPhlu8hMwfXvfFxaIATw_fSJAUf-5nxHfUs37MpiN9Dtv25Y9bSriueoPO4PxbaMUGkVkWxRYWJSKYU1QoFojDUSrWp1DokEQ6ppQGVG5zkSadNqA5ASnlDlkMoNu7_crpz1ObgvDD_1H2-98spfliJPWA</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence</title><source>arXiv.org</source><creator>Halterman, Andrew ; Keith, Katherine A ; Sarwar, Sheikh Muhammad ; O'Connor, Brendan</creator><creatorcontrib>Halterman, Andrew ; Keith, Katherine A ; Sarwar, Sheikh Muhammad ; O'Connor, Brendan</creatorcontrib><description>Findings of ACL 2021 Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage retrieval tasks. Our novel corpus-level evaluations and annotation approach can guide creation of similar social-science-oriented resources in the future.</description><identifier>DOI: 10.48550/arxiv.2105.12936</identifier><language>eng</language><subject>Computer Science - Computation and Language</subject><creationdate>2021-05</creationdate><rights>http://creativecommons.org/licenses/by/4.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,780,885</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2105.12936$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2105.12936$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Halterman, Andrew</creatorcontrib><creatorcontrib>Keith, Katherine A</creatorcontrib><creatorcontrib>Sarwar, Sheikh Muhammad</creatorcontrib><creatorcontrib>O'Connor, Brendan</creatorcontrib><title>Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence</title><description>Findings of ACL 2021 Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage retrieval tasks. Our novel corpus-level evaluations and annotation approach can guide creation of similar social-science-oriented resources in the future.</description><subject>Computer Science - Computation and Language</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2021</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj81KxDAUhbNxIaMP4Mq8QGt60yaNu6GM40BhFIrbctu50UhthvQHfXtrx9XhcH7gY-wuEXGaZ5l4wPDt5hgSkcUJGKmuGRU-nKchKmmmju9m7CYcne-59WGx1I_8dfvIqw_ih_7k8MV3rqU1GPhlu8hMwfXvfFxaIATw_fSJAUf-5nxHfUs37MpiN9Dtv25Y9bSriueoPO4PxbaMUGkVkWxRYWJSKYU1QoFojDUSrWp1DokEQ6ppQGVG5zkSadNqA5ASnlDlkMoNu7_crpz1ObgvDD_1H2-98spfliJPWA</recordid><startdate>20210527</startdate><enddate>20210527</enddate><creator>Halterman, Andrew</creator><creator>Keith, Katherine A</creator><creator>Sarwar, Sheikh Muhammad</creator><creator>O'Connor, Brendan</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20210527</creationdate><title>Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence</title><author>Halterman, Andrew ; Keith, Katherine A ; Sarwar, Sheikh Muhammad ; O'Connor, Brendan</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a676-e3ca6a194330f90620b9f93af6c7821329e6bb2659788aee79c79224eada68243</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2021</creationdate><topic>Computer Science - Computation and Language</topic><toplevel>online_resources</toplevel><creatorcontrib>Halterman, Andrew</creatorcontrib><creatorcontrib>Keith, Katherine A</creatorcontrib><creatorcontrib>Sarwar, Sheikh Muhammad</creatorcontrib><creatorcontrib>O'Connor, Brendan</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Halterman, Andrew</au><au>Keith, Katherine A</au><au>Sarwar, Sheikh Muhammad</au><au>O'Connor, Brendan</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence</atitle><date>2021-05-27</date><risdate>2021</risdate><abstract>Findings of ACL 2021 Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage retrieval tasks. Our novel corpus-level evaluations and annotation approach can guide creation of similar social-science-oriented resources in the future.</abstract><doi>10.48550/arxiv.2105.12936</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2105.12936
ispartof
issn
language eng
recordid cdi_arxiv_primary_2105_12936
source arXiv.org
subjects Computer Science - Computation and Language
title Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T10%3A45%3A40IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Corpus-Level%20Evaluation%20for%20Event%20QA:%20The%20IndiaPoliceEvents%20Corpus%20Covering%20the%202002%20Gujarat%20Violence&rft.au=Halterman,%20Andrew&rft.date=2021-05-27&rft_id=info:doi/10.48550/arxiv.2105.12936&rft_dat=%3Carxiv_GOX%3E2105_12936%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true