PCNN: Probable-Class Nearest-Neighbor Explanations Improve
Fine-Grained Image Classification Accuracy for AIs and Humans
Giang Nguyen1*, Valerie Chen2, Mohammad Reza Taesiri3, Anh Nguyen1
*Corresponding author · 1Auburn University · 2Carnegie Mellon University · 3University of Alberta
Nearest neighbors are traditionally used either to make final predictions (e.g. k-NN, SVMs) or to provide supporting evidence for a model’s decision.
PCNN takes a new path: we keep a pretrained image classifier C intact, then let an image comparator S
- compare the query image against nearest neighbors drawn from each of the top-most probable classes, and
- re-weight C’s logits in a Product-of-Experts style.
This simple plug-in consistently boosts accuracy on CUB-200, Cars-196, and Dogs-120.
A user study further shows that lay users make more accurate decisions when viewing PCNN’s probable-class neighbors versus seeing only top‑1‑class examples (prior work).
- Pre‑requisites & Pretrained Models
- Training Image‑Comparator Networks
- Testing — Binary Classification & Re-ranking
- Qualitative Visualizations
- Human‑Study Data
- Citation
- Clone this repo and install the usual PyTorch / torchvision stack (CUDA 11+ recommended).
- Download our pretrained backbones for CUB‑200, Cars‑196, and Dogs‑120
👉 https://drive.google.com/drive/folders/1pC_5bEi5DryDZCaKb51dzCE984r8EnqW
- In
params.pysetglobal_training_type = "CUB" self.set = "train"
- Launch:
sh train_cub.sh
Same recipe; just set
global_training_type = "CARS" or "DOGS".
- Edit
params.py:global_training_type = "CUB" self.set = "test"
- Run:
sh test_cub.sh
Swap global_training_type to "CARS" or "DOGS" and execute the matching script to reproduce the main‑paper table:
The following commands assume CUB‑200; substitute "CARS" / "DOGS" as needed.
| What you’ll see | How to run |
|---|---|
| Corrections made by S | python cub_visualize_corrections.py |
| Training pairs used by S | python cub_visualize_training_nns.py |
| Failure cases of S (set VISUALIZE_COMPARATOR_CORRECTNESS=True) |
python cub_infer.py |
| Attention heat‑maps (set VISUALIZE_COMPARATOR_HEATMAPS=True) |
python cub_infer.py |
All stimuli for our CUB‑200 and Cars‑196 user studies are available at:
https://drive.google.com/drive/folders/1yNIOfypfy1vvI3Q3MAq9LNIVlyQ3WY-V
@article{
nguyen2024pcnn,
title={{PCNN}: Probable-Class Nearest-Neighbor Explanations Improve Fine-Grained Image Classification Accuracy for {AI}s and Humans},
author={Giang Nguyen and Valerie Chen and Mohammad Reza Taesiri and Anh Nguyen},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2024},
url={https://openreview.net/forum?id=OcFjqiJ98b},
note={}
}🖼️ Questions? Open an issue or email nguyengiangbkhn@gmail.com 🖼️