Building Computer Vision Models for Social Good: Image Captioning for the Visually Impaired

Computer vision has made phenomenal strides over the past decade, yet bridging the gap between raw object detection and meaningful human assistance remains an inspiring challenge.

During my undergraduate thesis at the Islamic University of Technology, my research focused on creating an automated image captioning model tailored for visually impaired individuals. Here is a breakdown of the architecture, the dataset engineering process, and our work developing state-of-the-art Bengali image captioning.

The Problem: Beyond Object Detection

Standard object detection models output bounding boxes and labels:

“chair: 0.88, door: 0.94, table: 0.72”

For a visually impaired individual navigating an indoor or outdoor environment, disconnected labels fail to provide context. The critical information is the spatial relationship and hazard awareness:

“A wooden chair is blocking the doorway approximately two meters ahead.”

The Architecture: Encoder-Decoder with Bahdanau Attention

We formulated the problem using an Encoder-Decoder deep learning topology:

    [Input Image]
          │
          ▼
   [CNN Encoder: ResNet / Inception]
          │
          ▼
    [Feature Map] ◄───┐
          │            │
          ▼            │
 [Bahdanau Attention]  │ (Spatial Weights)
          │            │
          ▼            │
   [LSTM / GRU Decoder]┘
          │
          ▼
 [Generated Assistive Sentence]

1. CNN Feature Extractor (Encoder)

We leveraged pre-trained CNN backbones (ResNet-101 and Inception-V3) fine-tuned on assistive datasets. Instead of extracting the final fully connected classification vector, we took the spatial feature grid prior to pooling, preserving 2D positional references.

2. Attention Layer

At each time step $t$ in the text generation decoder, the attention mechanism computes a distribution over spatial locations:

$$\alpha_{t,i} = \frac{\exp(e_{t,i})}{\sum_{k=1}^P \exp(e_{t,k})}$$

This allows the decoder to “look” at the specific obstacle or landmark it is verbalizing.

import torch
import torch.nn as nn

class Attention(nn.Module):
    def __init__(self, encoder_dim, decoder_dim, attention_dim):
        super(Attention, self).__init__()
        self.encoder_att = nn.Linear(encoder_dim, attention_dim)
        self.decoder_att = nn.Linear(decoder_dim, attention_dim)
        self.full_att = nn.Linear(attention_dim, 1)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, encoder_out, decoder_hidden):
        # encoder_out: (batch_size, num_pixels, encoder_dim)
        # decoder_hidden: (batch_size, decoder_dim)
        att1 = self.encoder_att(encoder_out)
        att2 = self.decoder_att(decoder_hidden)
        att = self.full_att(self.relu(att1 + att2.unsqueeze(1))).squeeze(2)
        alpha = self.softmax(att)
        attention_weighted_encoding = (encoder_out * alpha.unsqueeze(2)).sum(dim=1)
        return attention_weighted_encoding, alpha

Advancing Bengali Image Captioning

While English has extensive benchmarks like MS-COCO and Flickr30k, low-resource languages lack annotated visual descriptions.

To overcome this:

1. Bilingual Corpus Alignment: We created a curated Bengali dataset with human-annotated captions reflecting local contextual vocabulary.
2. Morphological Tokenization: Bengali features rich inflectional morphology. Standard whitespace tokenization creates vocabulary explosion, so we implemented subword BPE tokenization.
3. Evaluation: Our model achieved benchmark BLEU-4 and CIDEr scores, setting state-of-the-art results for Bengali image captioning at the time of publication.

Conclusion

Assistive AI demonstrates the transformative potential of deep learning. When algorithms are paired with thoughtful product design and rigorous evaluation, machine learning moves from abstract math to life-changing accessibility.