SuperintelligenceMONITOR
Published by Center for Artificial WisdomLatest newsletter: Issue #1 · draft v1.11

Explanatory draft revision record

What changed in the benchmark explainer

October 17, 2026 · draft v1.0 to v1.1

Adds earlier methodological research as context. The three original cases, their claims and source cutoff are unchanged. This is not an independent replication of the cases.

The earlier version had no counterpart to the two sections below. Their exact added wording and sources follow. The original case sections, opening, conclusion and disclosure are unchanged.

Added in v1.1 · October 17, 2026

Ask whether the test measures the claim

In a 2021 NeurIPS position paper, Inioluwa Deborah Raji and coauthors examined the leap from a defined benchmark task to a claim about general ability. They argued for evaluations tied to specific uses and contexts. Their account gives us a question to ask here: does success at this game measure the research ability we want to understand?

This is methodological criticism, not a new test of EBR-bench. The authors included researchers at Mozilla, universities and Google Research. Their institutional affiliations and the paper’s status matter; a conference paper does not settle how much this particular game predicts scientific work.

Added in v1.1 · October 17, 2026

A ranking also depends on the tasks chosen

A separate 2021 study by Mostafa Dehghani and coauthors at Google Research and DeepMind showed that changing the benchmark tasks could substantially alter the relative performance of methods in the settings they examined. It is another reason to inspect a comparison before extending its conclusion.

That finding concerns the study’s own benchmark setups. It neither reproduces Epoch’s game experiment nor shows that its revised scores are wrong. Our narrower conclusion is that a credible broader claim needs tests selected for that claim, with relevant conditions and failures reported.

Read the complete earlier draft · Current draft