Proposing interpretability techniques for BERT-based multi-label classification of biomedical texts — M.Sc. Thesis (Grade: 9.5/10.0)
Problem statement and research motivation
The biomedical research community publishes a massive volume of unstructured data daily, necessitating automated tools for indexing, topic categorization, and multi-label classification. While deep learning models like Transformers (BERT, blue-BERT) achieve state-of-the-art predictive performance, they function as "black boxes" — their deep, complex representations are difficult to interpret.
In domains like healthcare and biomedicine, simply providing an automated prediction is rarely enough. Users need to understand why a model made a specific decision. This project focused on adapting advanced interpretability algorithms to a multi-label format and transforming their outputs into readable, sentence-level rationales.
Two manually annotated biomedical corpora with sentence-level rationale ground truth
Four interpretability approaches adapted for multi-label text classification
Gradient-based attribution method modified to solve saturation issues in deep networks.
Propagation-based method delivering 20x faster inference than IG.
Supervised framework jointly learning label prediction and rationale extraction.
Leveraging raw attention scores across Transformer layers and aggregating into sentences.
Custom metrics designed for interpretability assessment
Standard evaluation metrics fall short for interpretability tasks. This project designed custom evaluation metrics to rigorously assess the quality of extracted rationales against human-annotated ground truth.
Measures explanation precision and recall by comparing extracted rationale sentences against human-annotated ground truth sentences at the sentence level.
Measures the drop in predictive confidence when the model's selected "important" sentences (rationales) are removed from the input. Variants:
Quantitative performance across methods, models, and datasets
Sentence-Level F₁ Score — Multi-Task Learning (HoC)
Sentence-Level F₁ scores across methods and datasets
| Method | HoC (BERT) | HoC (blue-BERT) | CEI (BERT) | CEI (blue-BERT) |
|---|---|---|---|---|
| Multi-Task Learning | 0.7421 | — | 0.7255 | — |
| LRP-POS | 0.6728 | 0.6218 | — | — |
| Integrated Gradients | 0.6125 | 0.6211 | — | — |
The Multi-Task Learning approach yielded the highest quality explanations across both datasets. Among fully unsupervised methods, LRP-POS utilizing standard BERT achieved the best results. Notably, LRP produced high-quality explanations with an inference time ~20 times faster than Integrated Gradients (~1 sec vs. ~20 sec per instance).
Interestingly, the original general-purpose BERT frequently produced more meaningful explanations than the domain-specific blue-BERT, as verified by the Faithfulness-1 metric.
Languages, frameworks, and environment
Papers used in methodology design