Empirical Analysis of Image Caption Generation using Deep Learning
Automated image captioning is one of the applications of Deep Learning which involves fusion of work done in computer vision and natural language processing, and it is typically performed using Encoder-Decoder architectures. In this project, we have implemented and experimented with various flavors...
Gespeichert in:
Hauptverfasser: | , , |
---|---|
Format: | Artikel |
Sprache: | eng |
Schlagworte: | |
Online-Zugang: | Volltext bestellen |
Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
Zusammenfassung: | Automated image captioning is one of the applications of Deep Learning which
involves fusion of work done in computer vision and natural language
processing, and it is typically performed using Encoder-Decoder architectures.
In this project, we have implemented and experimented with various flavors of
multi-modal image captioning networks where ResNet101, DenseNet121 and VGG19
based CNN Encoders and Attention based LSTM Decoders were explored. We have
studied the effect of beam size and the use of pretrained word embeddings and
compared them to baseline CNN encoder and RNN decoder architecture. The goal is
to analyze the performance of each approach using various evaluation metrics
including BLEU, CIDEr, ROUGE and METEOR. We have also explored model
explainability using Visual Attention Maps (VAM) to highlight parts of the
images which has maximum contribution for predicting each word of the generated
caption. |
---|---|
DOI: | 10.48550/arxiv.2105.09906 |