Unleashing the Power of LLMs as Multi-Modal Encoders for Text and Graph-Structured Data
Graph-structured information offers rich contextual information that can enhance language models by providing structured relationships and hierarchies, leading to more expressive embeddings for various applications such as retrieval, question answering, and classification. However, existing methods...
Gespeichert in:
Hauptverfasser: | , , , , , , , , , , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Graph-structured information offers rich contextual information that can
enhance language models by providing structured relationships and hierarchies,
leading to more expressive embeddings for various applications such as
retrieval, question answering, and classification. However, existing methods
for integrating graph and text embeddings, often based on Multi-layer
Perceptrons (MLPs) or shallow transformers, are limited in their ability to
fully exploit the heterogeneous nature of these modalities. To overcome this,
we propose Janus, a simple yet effective framework that leverages Large
Language Models (LLMs) to jointly encode text and graph data. Specifically,
Janus employs an MLP adapter to project graph embeddings into the same space as
text embeddings, allowing the LLM to process both modalities jointly. Unlike
prior work, we also introduce contrastive learning to align the graph and text
spaces more effectively, thereby improving the quality of learned joint
embeddings. Empirical results across six datasets spanning three tasks,
knowledge graph-contextualized question answering, graph-text pair
classification, and retrieval, demonstrate that Janus consistently outperforms
existing baselines, achieving significant improvements across multiple
datasets, with gains of up to 11.4% in QA tasks. These results highlight
Janus's effectiveness in integrating graph and text data. Ablation studies
further validate the effectiveness of our method. |
---|---|
DOI: | 10.48550/arxiv.2410.11235 |