Structured Query-Based Image Retrieval Using Scene Graphs

Structured Query-Based Image Retrieval Using Scene Graphs

A structured query can capture the complexity of object interactions (e.g. 'woman rides motorcycle') unlike single objects (e.g. 'woman' or 'motorcycle'). Retrieval using structured queries therefore is much more useful than single object retrieval, but a much more chal...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Hauptverfasser: Schroeder, Brigit, Tripathi, Subarna
Format: Artikel
Sprache:eng
Schlagworte:
Computer Science - Computer Vision and Pattern Recognition
Computer Science - Learning
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
container_end_page
container_issue
container_start_page
container_title
container_volume
creator Schroeder, Brigit
Tripathi, Subarna
description A structured query can capture the complexity of object interactions (e.g. 'woman rides motorcycle') unlike single objects (e.g. 'woman' or 'motorcycle'). Retrieval using structured queries therefore is much more useful than single object retrieval, but a much more challenging problem. In this paper we present a method which uses scene graph embeddings as the basis for an approach to image retrieval. We examine how visual relationships, derived from scene graphs, can be used as structured queries. The visual relationships are directed subgraphs of the scene graph with a subject and object as nodes connected by a predicate relationship. Notably, we are able to achieve high recall even on low to medium frequency objects found in the long-tailed COCO-Stuff dataset, and find that adding a visual relationship-inspired loss boosts our recall by 10% in the best case.
doi_str_mv 10.48550/arxiv.2005.06653
format Article
fullrecord <record><control><sourceid>arxiv_GOX</sourceid><recordid>TN_cdi_arxiv_primary_2005_06653</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2005_06653</sourcerecordid><originalsourceid>FETCH-LOGICAL-a673-85f0cc8111892b39f677050b69eacde2340a939fb86d7040ef5ad2fd8fe02a5a3</originalsourceid><addsrcrecordid>eNotj8tuwjAURL1hUQEfwAr_QNIbO3bsJaBCkZCq8lhHN_Y1RAKEnATB30OhqxnN4mgOY6MM0twoBZ8Yb_U1FQAqBa2V_GB208bOtV0kz387ivdkis2zL0-4J76mNtZ0xSPfNfV5zzeOzsQXES-HZsB6AY8NDf-zz7bzr-3sO1n9LJazySpBXcjEqADOmSzLjBWVtEEXBSiotCV0noTMAe1zroz2BeRAQaEXwZtAIFCh7LPxG_v6Xl5ifcJ4L_8cypeDfACfgUDs</addsrcrecordid><sourcetype>Open Access Repository</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype></control><display><type>article</type><title>Structured Query-Based Image Retrieval Using Scene Graphs</title><source>arXiv.org</source><creator>Schroeder, Brigit ; Tripathi, Subarna</creator><creatorcontrib>Schroeder, Brigit ; Tripathi, Subarna</creatorcontrib><description>A structured query can capture the complexity of object interactions (e.g. 'woman rides motorcycle') unlike single objects (e.g. 'woman' or 'motorcycle'). Retrieval using structured queries therefore is much more useful than single object retrieval, but a much more challenging problem. In this paper we present a method which uses scene graph embeddings as the basis for an approach to image retrieval. We examine how visual relationships, derived from scene graphs, can be used as structured queries. The visual relationships are directed subgraphs of the scene graph with a subject and object as nodes connected by a predicate relationship. Notably, we are able to achieve high recall even on low to medium frequency objects found in the long-tailed COCO-Stuff dataset, and find that adding a visual relationship-inspired loss boosts our recall by 10% in the best case.</description><identifier>DOI: 10.48550/arxiv.2005.06653</identifier><language>eng</language><subject>Computer Science - Computer Vision and Pattern Recognition ; Computer Science - Learning</subject><creationdate>2020-05</creationdate><rights>http://arxiv.org/licenses/nonexclusive-distrib/1.0</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><link.rule.ids>228,230,776,881</link.rule.ids><linktorsrc>$$Uhttps://arxiv.org/abs/2005.06653$$EView_record_in_Cornell_University$$FView_record_in_$$GCornell_University$$Hfree_for_read</linktorsrc><backlink>$$Uhttps://doi.org/10.48550/arXiv.2005.06653$$DView paper in arXiv$$Hfree_for_read</backlink></links><search><creatorcontrib>Schroeder, Brigit</creatorcontrib><creatorcontrib>Tripathi, Subarna</creatorcontrib><title>Structured Query-Based Image Retrieval Using Scene Graphs</title><description>A structured query can capture the complexity of object interactions (e.g. 'woman rides motorcycle') unlike single objects (e.g. 'woman' or 'motorcycle'). Retrieval using structured queries therefore is much more useful than single object retrieval, but a much more challenging problem. In this paper we present a method which uses scene graph embeddings as the basis for an approach to image retrieval. We examine how visual relationships, derived from scene graphs, can be used as structured queries. The visual relationships are directed subgraphs of the scene graph with a subject and object as nodes connected by a predicate relationship. Notably, we are able to achieve high recall even on low to medium frequency objects found in the long-tailed COCO-Stuff dataset, and find that adding a visual relationship-inspired loss boosts our recall by 10% in the best case.</description><subject>Computer Science - Computer Vision and Pattern Recognition</subject><subject>Computer Science - Learning</subject><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2020</creationdate><recordtype>article</recordtype><sourceid>GOX</sourceid><recordid>eNotj8tuwjAURL1hUQEfwAr_QNIbO3bsJaBCkZCq8lhHN_Y1RAKEnATB30OhqxnN4mgOY6MM0twoBZ8Yb_U1FQAqBa2V_GB208bOtV0kz387ivdkis2zL0-4J76mNtZ0xSPfNfV5zzeOzsQXES-HZsB6AY8NDf-zz7bzr-3sO1n9LJazySpBXcjEqADOmSzLjBWVtEEXBSiotCV0noTMAe1zroz2BeRAQaEXwZtAIFCh7LPxG_v6Xl5ifcJ4L_8cypeDfACfgUDs</recordid><startdate>20200513</startdate><enddate>20200513</enddate><creator>Schroeder, Brigit</creator><creator>Tripathi, Subarna</creator><scope>AKY</scope><scope>GOX</scope></search><sort><creationdate>20200513</creationdate><title>Structured Query-Based Image Retrieval Using Scene Graphs</title><author>Schroeder, Brigit ; Tripathi, Subarna</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-a673-85f0cc8111892b39f677050b69eacde2340a939fb86d7040ef5ad2fd8fe02a5a3</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2020</creationdate><topic>Computer Science - Computer Vision and Pattern Recognition</topic><topic>Computer Science - Learning</topic><toplevel>online_resources</toplevel><creatorcontrib>Schroeder, Brigit</creatorcontrib><creatorcontrib>Tripathi, Subarna</creatorcontrib><collection>arXiv Computer Science</collection><collection>arXiv.org</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Schroeder, Brigit</au><au>Tripathi, Subarna</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Structured Query-Based Image Retrieval Using Scene Graphs</atitle><date>2020-05-13</date><risdate>2020</risdate><abstract>A structured query can capture the complexity of object interactions (e.g. 'woman rides motorcycle') unlike single objects (e.g. 'woman' or 'motorcycle'). Retrieval using structured queries therefore is much more useful than single object retrieval, but a much more challenging problem. In this paper we present a method which uses scene graph embeddings as the basis for an approach to image retrieval. We examine how visual relationships, derived from scene graphs, can be used as structured queries. The visual relationships are directed subgraphs of the scene graph with a subject and object as nodes connected by a predicate relationship. Notably, we are able to achieve high recall even on low to medium frequency objects found in the long-tailed COCO-Stuff dataset, and find that adding a visual relationship-inspired loss boosts our recall by 10% in the best case.</abstract><doi>10.48550/arxiv.2005.06653</doi><oa>free_for_read</oa></addata></record>
fulltext fulltext_linktorsrc
identifier DOI: 10.48550/arxiv.2005.06653
ispartof
issn
language eng
recordid cdi_arxiv_primary_2005_06653
source arXiv.org
subjects Computer Science - Computer Vision and Pattern Recognition
Computer Science - Learning
title Structured Query-Based Image Retrieval Using Scene Graphs
url https://sfx.bib-bvb.de/sfx_tum?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-02-08T17%3A50%3A39IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-arxiv_GOX&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Structured%20Query-Based%20Image%20Retrieval%20Using%20Scene%20Graphs&rft.au=Schroeder,%20Brigit&rft.date=2020-05-13&rft_id=info:doi/10.48550/arxiv.2005.06653&rft_dat=%3Carxiv_GOX%3E2005_06653%3C/arxiv_GOX%3E%3Curl%3E%3C/url%3E&disable_directlink=true&sfx.directlink=off&sfx.report_link=0&rft_id=info:oai/&rft_id=info:pmid/&rfr_iscdi=true