A visual, hands-on toolkit for exploring the inner workings of vision neural networks, from edges and textures to object parts, concepts, neurons, and circuits.
Built by the ML Understanding group at Cohere Labs.
This work is based on the prior work done by Chris Olah and collaborators for the Distill Circuits Thread and OpenAI Microscope page, which has been unoperational for a few months now. We believe an interactive space is important to grasp the very influential Circuits thread, so we have built InceptionV1 Microscope, a Hugging Face space based on the notebooks shared here. Happy Exploring!
Neural networks can feel like black boxes.
But with vision models, we have a unique opportunity: we can visualize what happens inside them.
The Vision Interpretability Toolkit brings together interactive tools and reproducible Colab notebooks for investigating how vision models represent and process images.
Use it to:
- 🌀 Generate activation maximization visualizations
- 🖼️ Find dataset examples that most strongly activate a neuron
- 🔎 Locate the regions within an image driving those activations
- 🧩 Follow representations from edges → textures → shapes → object parts → concepts
- 🧠 Investigate phenomena such as polysemanticity and superposition
- 🧪 Run the experiments on your own images
- 🔌 Adapt the workflow to other models and datasets
No interpretability expertise required — curiosity is enough.
One of the best ways to understand these ideas is simply to start looking.
Our InceptionV1 Microscope provides an interactive window into InceptionV1 (GoogLeNet), trained on ImageNet.
Choose a layer.
Choose a channel.
Then investigate what makes that neuron activate.
A synthesized image optimized to strongly activate the selected neuron.
Rather than telling us exactly what a neuron means, it gives us a visual hypothesis about the kinds of patterns that excite it.
See real ImageNet images that strongly activate the same neuron.
Comparing these examples with the activation-maximization visualization lets us ask:
Does the behavior of the neuron on real images support our hypothesis?
Zoom into the regions of those images associated with strong activation.
Sometimes the connection is immediately obvious.
Sometimes it's surprising.
And sometimes things get weird.
That's where the fun begins.
Move through InceptionV1 layer by layer and you can observe a rough progression in the complexity of its learned representations.
IMAGE
│
▼
Edges
│
▼
Textures & Patterns
│
▼
Shapes
│
▼
Object Parts
│
▼
Higher-level Concepts
Early layers often respond to relatively simple visual features such as orientations, colors, edges, and textures.
Deeper layers combine these into increasingly complex patterns — curves, repeated structures, object parts, and eventually more semantic features.
Looking across layers turns an abstract neural network into something you can visually investigate.
Some neurons don't appear to correspond neatly to one human-interpretable concept.
A neuron might respond strongly to several seemingly unrelated visual patterns.
This provides an intuitive doorway into concepts such as polysemanticity and, more broadly, superposition — where neural networks can represent more features than there are individual dimensions available to represent them.
Instead of beginning with the mathematics, this toolkit lets you encounter these phenomena visually first.
See something strange → form a hypothesis → test it.
That's a beginner's path to mechanistic interpretability.
The Microscope lets you investigate our generated visualizations.
The Vision Interpretability Toolkit lets you create them yourself.
The included Colab notebooks walk through the techniques behind the visualizations so that you can:
Choose an image
↓
Choose a model
↓
Inspect its layers
↓
Visualize activations
↓
Generate feature visualizations
↓
Compare against dataset examples
↓
Form hypotheses
↓
Test them
Try a photograph of your dog.
Try an unusual texture.
Try something completely out of distribution.
Then ask:
What changed inside the network?
The easiest way to begin is through the notebooks in this repository.
Start by browsing neurons across different InceptionV1 layers.
Look for patterns.
Find a neuron that catches your attention.
Run the experiments yourself without needing to configure a local environment.
Replace the example input with something you're curious about and follow its representations through the network.
Use the notebooks as a starting point for experiments with other vision models, datasets, and interpretability questions.
InceptionV1 has an important place in the history of neural-network interpretability.
Work including the Distill Circuits thread by Chris Olah and collaborators demonstrated how studying features and their interactions in vision models could reveal surprisingly interpretable structure inside neural networks.
OpenAI's Microscope made many of these visualizations accessible through an interactive interface.
This project is, in part, an attempt to bring that style of exploration back — while also giving learners notebooks they can modify, break, extend, and make their own.
Once you've got the notebooks running, try asking your own questions:
- What changes between neighboring layers?
- Do similar neurons appear at different depths?
- Can you find neurons selective for particular textures?
- Can you find apparent object-part detectors?
- Which dataset examples activate a neuron unexpectedly?
- Can you find a clearly polysemantic neuron?
- How stable is your interpretation across different input images?
- Do different models learn similar visual features?
- What happens when you repeat the experiment on another dataset?
There isn't always a clean answer.
That's the point.
Mechanistic interpretability can quickly become mathematically and conceptually intimidating.
Vision gives us another way in.
We can look first, build intuition, ask questions, and then dig deeper.
We hope this toolkit makes it easier for students, researchers, engineers, and curious newcomers to take that first step.
This project was developed as part of the ML Understanding group at Cohere Labs. The prior project repo can be found here
It draws inspiration from the pioneering vision interpretability work of Chris Olah and collaborators, the Distill Circuits thread, and OpenAI Microscope.
Contributors:
Akanksha Devkar (akankshanc)
Mayank Bhaskar (cataluna84)
Joyce Jiang (JoyceJiang73)
Anu Adesina (nuna-aa)
Ayesha Imran (Ayesha-Imr)
Almustapha Yusuf (1mustyz)
Joana da Matta (jodamatta)