CE-Bench: Towards a Reliable Contrastive Evaluation Benchmark of Interpretability of Sparse Autoencoders

TL;DR

CE-Bench is a benchmark that scores how interpretable a sparse autoencoder is without asking another LLM to judge it. Yusen Peng, Sachin Kumar, and I built it at Ohio State and published it at the BlackboxNLP workshop at EMNLP 2025. Its rankings agree with the LLM-judged SAEBench at over 70% Spearman correlation.

Problem

Sparse autoencoders (SAEs) take the activations inside an LLM and split them into features that are supposed to mean something. The question is how to check that they do. Most automated evaluations rely on an external LLM to judge the features. That makes the scores non-deterministic, possibly biased, and hard to reproduce.

Idea

Our hypothesis: if a feature means something, it should react when the meaning changes. So we built a dataset of contrastive stories: 5,000 pairs of short stories about 1,000 subjects picked from WikiData. Each pair tells the same subject twice, once at its extreme and once as its opposite. For “computer”, one story follows a machine that cracks cryptography and maps the global weather at the same time. The other is about the “Null Device”, which cannot add two numbers.

How it works

We average how much each SAE feature fires on each story and compute two scores.

The contrastive score is the biggest gap between the two stories. Is there a feature that flips when the take flips?

The independence score is the biggest gap between this pair and the average over the whole dataset. Is there a feature that cares about this subject in particular?

Add the two up, subtract a penalty based on the SAE’s sparsity, and you have one interpretability score. No judge and no prompts at evaluation time, only arithmetic on activations.

Results

Those numbers come from 48 pretrained SAEs that SAEBench had already graded. We also used CE-Bench to compare SAEs with each other. The clearest trend is width: with the architecture and the layer held fixed, 65k-wide SAEs scored above 16k-wide ones, and those above 4k-wide ones, at every sparsity level.

Here are the CE-Bench scores of 36 of those SAEs next to their autointerp scores from SAEBench: six architectures at six sparsity levels each, all 65k wide and reading layer 12 of Gemma-2-2B. Click a column to sort by it.

One catch: the stories themselves were written by GPT-4.1, so the benchmark may favor models that are at home in GPT-style text.

Cite

If CE-Bench is useful in your work, please cite the paper:

@inproceedings{gulko-etal-2025-ce,
    title = "{CE}-Bench: Towards a Reliable Contrastive Evaluation Benchmark of Interpretability of Sparse Autoencoders",
    author = "Gulko, Alex  and
      Peng, Yusen  and
      Kumar, Sachin",
    editor = "Belinkov, Yonatan  and
      Mueller, Aaron  and
      Kim, Najoung  and
      Mohebbi, Hosein  and
      Chen, Hanjie  and
      Arad, Dana  and
      Sarti, Gabriele",
    booktitle = "Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.blackboxnlp-1.1/",
    pages = "1--15",
    ISBN = "979-8-89176-346-3"
}

See more