Saved daily version · 2026-11-05
Daily evidence notes
OpenAI reports a pipeline that produced exploits and proposed fixes for 948 Linux flaws; maintainer review and deployment remain separate steps. Anthropic reports a Claude-assisted protein-structure experiment with substantial human direction and documented errors. Epoch revises how its capability index represents uncertainty and estimates a rise in TPU shipments. An independent audit also shows how grading software can move agent rankings. These reports strengthen the case for checking both AI performance and the instruments used to measure it.
Sources checked and limits of this review
Reviewed since the November 4 briefing using primary pages from METR, ARC Prize, Epoch AI, Google DeepMind, OpenAI and Anthropic, plus independent-research searches. Cybersecurity reporting supplied leads checked against OpenAI’s original account. Selected material comprises four November 4 announcements/analyses and a November 3 independent preprint newly reviewed here. The Linux run occurred in mid-October; Epoch’s shipment estimates concern fiscal Q3, not chips shipped today. No new selected capability result was identified on the METR, ARC Prize or DeepMind pages checked; ARC competition closure is not a result announcement. Anthropic’s linked preprint could not be retrieved, so that note relies on its detailed original research announcement and states the verification limit. Coverage is bounded by accessible sources and indexing. Findings, estimates, forecasts and editorial interpretations remain distinct; no overall wisdom verdict or human editorial approval is claimed.
Evidence note 1 · Lab-reported security research; SI relevance is editorial interpretation
OpenAI reports 948 Linux flaws with working exploits and proposed patches
OpenAI says an upcoming cyber model’s pipeline scanned selected Linux code, reproduced vulnerabilities, developed exploits and generated fixes. Its mid-October run yielded 948 distinct local privilege-escalation findings at about $1.21 million in API credits. Human researchers review outputs before reporting them to maintainers.
Our interpretation. Editorial interpretation: automating discovery and validation could help defenders audit large systems, while increasing attackers’ capabilities. Reduced harm depends on validated fixes reaching users, not the count of generated patches.
Main limitation. Lab-reported controlled experiment requiring an existing local foothold, not remote compromise of every Linux system. Candidate patches are not all accepted or deployed. Early exploits sometimes used the wrong flaw; initial patches sometimes missed variants or introduced tradeoffs. API cost excludes other project costs.
Next evidence. Maintainer confirmations, accepted fixes and distribution updates, outside replication, and whether patching keeps pace with discovery.
Evidence note 2 · Lab-reported experimental biology; SI relevance is editorial interpretation
Anthropic reports a protein structure found with Claude and substantial human direction
Anthropic reports producing vault-like protein shells in E. coli and mapping a bacteriophage-derived shell using cryo-electron microscopy. Claude assisted sequence searches, DNA designs, purification and data processing. Scientists selected candidates and directed analysis; a first structural model took 34 hours.
Our interpretation. Editorial interpretation: useful research assistance links computational proposals to physical experiments. Documenting human interventions and failures is essential to assess how much AI actually accelerates discovery.
Main limitation. Lab report, not an independent productivity comparison. Claude errors caused a two-week delay and a flawed early map. The particles’ function and formation in their natural hosts remain untested. The cryo-EM work used hundreds of GPUs. The linked preprint was inaccessible during this review.
Next evidence. Accessible methods and structural data, external replication, natural-host experiments and comparisons including correction time and compute.
Evidence note 3 · Independent statistical methodology update; SI relevance is editorial interpretation
Epoch changes its capability index’s uncertainty model, not the underlying AI systems
Epoch replaced the statistical fit behind its Epoch Capabilities Index with a Bayesian model that estimates capability distributions jointly. It now reports 90% credible intervals and probabilities for differences between models. Frontier estimates remain close to their previous values.
Our interpretation. Editorial interpretation: a ranking change can reflect measurement choices rather than a capability breakthrough. Explicit uncertainty can discourage treating a narrow score lead as a settled fact about general intelligence.
Main limitation. Independent measurement-method update, not a new model evaluation. Estimates depend on benchmark coverage and a single-capability-dimension assumption. The changelog says domain-specific indices have not yet received the full Bayesian refit. Statistical uncertainty does not cover every weakness in benchmark validity.
Next evidence. Independent checks of the fit, sensitivity to benchmark selection, domain-index updates and disclosure of methodology versions beside comparisons.
Evidence note 4 · Independent infrastructure estimate; SI relevance is editorial interpretation
Epoch estimates a doubling of Broadcom TPU shipments measured by compute
Epoch estimates Broadcom shipped nearly 2 million H100-equivalents of TPU compute in fiscal Q3 2026, about twice fiscal Q2. Estimated physical unit volume rose about 65% to roughly 800,000 chips; a shift toward more powerful chips explains the larger compute increase.
Our interpretation. Editorial interpretation: more accelerators could expand training and inference capacity, but shipment totals cannot establish what research they support or when usable capacity comes online.
Main limitation. Independent estimate derived from revenue, assumed chip mix and modeled prices—not a direct chip census. The reported 90% interval is roughly 1.57–2.35 million H100-equivalents. The quarter covers May to early August; estimates include Google and Anthropic shipments. H100-equivalents are a comparison measure, not literal Nvidia chips.
Next evidence. Revised revenue and chip-mix evidence, installation and power availability, actual utilization and allocation between research and customer inference.
Evidence note 5 · Independent artifact audit preprint, newly reviewed; SI relevance is editorial interpretation
Newly reviewed: grading software can change agent rankings without changing agents
Yangze Liu audited public HAL and tau2-bench records. In one comparison, grading previously ungraded episodes closed a pooled 13-point gap between agent setups. Other cases involved duplicated costs, grader actions logged as agent actions and evaluator changes that reordered results.
Our interpretation. Editorial interpretation: evidence of progress depends on the measurement pipeline as well as the model. A plausible score can misrepresent performance when grading, logging or versioning fails.
Main limitation. Independent preprint auditing two leaderboards, not a survey of all evaluations. No agents were rerun; one replay checks database and output criteria with the user simulator stubbed. The author reports HAL maintainer confirmation of three issues; SIM has not independently reproduced the calculations.
Next evidence. Published corrected archives, outside reproduction, graded-episode counts, action provenance and version-pinned before/after score comparisons.