Glasshouse ships an agent-memory benchmark with no headline number

Share
Glasshouse ships an agent-memory benchmark with no headline number

A memory-infrastructure vendor has published the test that grades its own industry — and written down, in advance, the ways it could cheat it.

Wontopos has put the files for glasshouse v0.1 into the open: a long-term memory benchmark for AI systems whose central design decision is that it will not produce a single score. The corpus is 2,847 questions over a conversation running to 1.97 million tokens, in 10 languages, with 50 uncaptioned photographs, and every axis is reported on its own because a system can be excellent on one and useless on another. The motive is a number that never settled: Zep reported 84% on LoCoMo, mem0 recomputed the same system and got 58.44%, and Zep returned with 75.14% — one system, one benchmark, three answers. LongMemEval and BEAM each ship a dataset and a script with nowhere to submit, so every vendor reports its own figure. The repository's own README still calls the version a draft.

The answer to that is receipts: a submission is a manifest plus the per-question record, and if the record does not reproduce the number, it does not go in. Seventeen axes cover ordinary retrieval, but also the failures memory vendors rarely publish — a fact that changed and can no longer be found, two stored facts that contradict with nothing in the history to settle them, a photograph stored with no caption, and 500 probes that assert something the conversation never said. One scoring rule runs through all of it: an incomplete answer whose gap is visible earns half credit, an incomplete answer that reads as current earns zero, because the second stops the user from asking again. The test ships as four nested tiers and the standard run is two of them — core at 1,882 turns with no filler, full at 103,572 turns. Only one variable separates the runs, so a score that falls between them has exactly one explanation: retrieval, not capability. The authors also measured their own instrument: asked with no memory at all the floor is 2.4%, handed the turns holding the answer the ceiling is 83.6% — a gap of 81.2 points, measured on 1,348 questions for $1.61 of model spend.

The two baselines are the part that will annoy vendors. On the medium tier at ten retrieved turns, BM25 scores 34.7% and an embedding model 35.8% — level, in one language. Asked in another language, both collapse to 1.8% and 4.6%, while the same 390 questions asked in English score 23.1% and 28.2%: the vector retriever answers in the language of the question rather than its meaning, returning turns in that language 74% of the time when such turns are 9% of the corpus. Neither baseline retrieves a single one of the 150 photograph answers, which is the control working as designed — no text retriever can find a caption that was never written.

Wontopos sells memory infrastructure, which means it competes in the thing it administers, and it says so on the front page rather than in a footnote. Its own submissions go through the same approval as everyone else's, the harness has to be one a customer could use rather than a private path, three runs is a result and five is certified, and the host publishes nothing on a new version for fourteen days so that hosting is not a head start. Everything is Apache 2.0, so if the arrangement fails the benchmark can be taken and run elsewhere. The submissions/ directory is empty — the rules went up before any results did — and three of the six Wontopos rows in its public claims register lack a published reader and judge, which by its own contribution guide would not be accepted as a submission.

That candour is also the limit. A run rigged from the start is not caught by any of this, and the README says so plainly; the only real defence is a third party running it on their own key. It is the same integrity problem we walked through in AI 101 — What is an AI benchmark?, and the self-audit instinct the site covered when IFM audited its own benchmark scores and found them 3.4 points too high.

What to watch: whether a rival memory vendor submits a result — the first non-Wontopos number is the one that decides whether this becomes a standard or stays a vendor's homework — and whether the judge-scored baselines, still unrun, ever publish.

A benchmark with no headline number is harder to market and harder to fake. Does that trade hold, or does the industry keep quoting LoCoMo? Write to us via our contact page.

Sources: glasshouse (GitHub) · glasshouse v0.1 README · glasshouse submission rules · glasshouse CREDITS.md

Get SiliconNoon in your inbox

One email a day with the AI stories that matter — what happened, the numbers, and why it matters. No spam, unsubscribe in one click.

Read more

Rabbit's OS3 agent runs on your laptop — and the R1 is done

Rabbit's OS3 agent runs on your laptop — and the R1 is done

Rabbit spent two years as the cautionary tale of AI hardware. On Tuesday it started selling the software instead: an agent that runs on the machines you already own. Rabbit has launched OS3, a standalone "agentic operating system" that lives in the cloud but executes work through a local agent node you install on Windows, Mac, or Linux — and founder Jesse Lyu told Wired the company has stopped manufacturing the R1, the pocket gadget that made it famous.

Fastino's 340M decision model beats its own 1B on routing

Fastino's 340M decision model beats its own 1B on routing

Most agent stacks answer a yes/no question by having a large model write a paragraph. Fastino's answer is to not ask a large model at all — and its new release is small enough to run on the CPU you already have. Fastino shipped GLiNER2.5-Decide, a 340M-parameter classifier that takes a bounded question — intent, routing, sentiment, severity, spam — and returns a label in a single forward pass, with no prompt template and no generated tokens.

Claude Code now reads AGENTS.md — and makes its harness rewritable

Claude Code now reads AGENTS.md — and makes its harness rewritable

Anthropic conceded the file-format fight this week, then quietly shipped something larger underneath it: a plugin layer that lets anyone — an organization, a vendor, a person — rewrite how Claude Code behaves at the hook level. Version 2.1.277 of Claude Code now loads AGENTS.md as project instructions in any project that carries no CLAUDE.md of its own, with a setting to flip it to "load both" or "ignore the shared file entirely." The same release publishes the source of the mechanism behind…

llama.cpp's CPU prompt processing gets 3–7x faster on k-quants

llama.cpp's CPU prompt processing gets 3–7x faster on k-quants

Three local-AI items for this hour: a kernel change that makes the wait before the first token 3 to 7 times shorter on a plain CPU, two small Chinese models that answer questions instead of writing text, and a 2017 laptop finishing real agent work. Local models just got dramatically faster to query on a processor most people already own — a kernel merged into llama.cpp today makes prompt processing 3 to 7 times quicker for the quantised formats the local community actually runs.

Workiva ships no-code agents for work that has to survive an audit

Workiva ships no-code agents for work that has to survive an audit

Enterprise AI's next fight is not capability, it is defensibility. Workiva spent its annual conference selling the version of agents that ends in a signature. Workiva put its agent layer in customers' hands at Amplify 2026, unveiling Agent Studio — a no-code capability for building, customizing and deploying AI agents inside its own reporting platform — alongside three agentic products for regulatory filings and a fourth for internal audit testing.