Glasshouse ships an agent-memory benchmark with no headline number
A memory-infrastructure vendor has published the test that grades its own industry — and written down, in advance, the ways it could cheat it.
Wontopos has put the files for glasshouse v0.1 into the open: a long-term memory benchmark for AI systems whose central design decision is that it will not produce a single score. The corpus is 2,847 questions over a conversation running to 1.97 million tokens, in 10 languages, with 50 uncaptioned photographs, and every axis is reported on its own because a system can be excellent on one and useless on another. The motive is a number that never settled: Zep reported 84% on LoCoMo, mem0 recomputed the same system and got 58.44%, and Zep returned with 75.14% — one system, one benchmark, three answers. LongMemEval and BEAM each ship a dataset and a script with nowhere to submit, so every vendor reports its own figure. The repository's own README still calls the version a draft.
The answer to that is receipts: a submission is a manifest plus the per-question record, and if the record does not reproduce the number, it does not go in. Seventeen axes cover ordinary retrieval, but also the failures memory vendors rarely publish — a fact that changed and can no longer be found, two stored facts that contradict with nothing in the history to settle them, a photograph stored with no caption, and 500 probes that assert something the conversation never said. One scoring rule runs through all of it: an incomplete answer whose gap is visible earns half credit, an incomplete answer that reads as current earns zero, because the second stops the user from asking again. The test ships as four nested tiers and the standard run is two of them — core at 1,882 turns with no filler, full at 103,572 turns. Only one variable separates the runs, so a score that falls between them has exactly one explanation: retrieval, not capability. The authors also measured their own instrument: asked with no memory at all the floor is 2.4%, handed the turns holding the answer the ceiling is 83.6% — a gap of 81.2 points, measured on 1,348 questions for $1.61 of model spend.
The two baselines are the part that will annoy vendors. On the medium tier at ten retrieved turns, BM25 scores 34.7% and an embedding model 35.8% — level, in one language. Asked in another language, both collapse to 1.8% and 4.6%, while the same 390 questions asked in English score 23.1% and 28.2%: the vector retriever answers in the language of the question rather than its meaning, returning turns in that language 74% of the time when such turns are 9% of the corpus. Neither baseline retrieves a single one of the 150 photograph answers, which is the control working as designed — no text retriever can find a caption that was never written.
Wontopos sells memory infrastructure, which means it competes in the thing it administers, and it says so on the front page rather than in a footnote. Its own submissions go through the same approval as everyone else's, the harness has to be one a customer could use rather than a private path, three runs is a result and five is certified, and the host publishes nothing on a new version for fourteen days so that hosting is not a head start. Everything is Apache 2.0, so if the arrangement fails the benchmark can be taken and run elsewhere. The submissions/ directory is empty — the rules went up before any results did — and three of the six Wontopos rows in its public claims register lack a published reader and judge, which by its own contribution guide would not be accepted as a submission.
That candour is also the limit. A run rigged from the start is not caught by any of this, and the README says so plainly; the only real defence is a third party running it on their own key. It is the same integrity problem we walked through in AI 101 — What is an AI benchmark?, and the self-audit instinct the site covered when IFM audited its own benchmark scores and found them 3.4 points too high.
What to watch: whether a rival memory vendor submits a result — the first non-Wontopos number is the one that decides whether this becomes a standard or stays a vendor's homework — and whether the judge-scored baselines, still unrun, ever publish.
A benchmark with no headline number is harder to market and harder to fake. Does that trade hold, or does the industry keep quoting LoCoMo? Tell us in the comments.
Sources: glasshouse (GitHub) · glasshouse v0.1 README · glasshouse submission rules · glasshouse CREDITS.md
