LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

Summary

A benchmark of 500 questions embedded in freely scalable chat histories, testing five long-term memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. Commercial assistants and long-context LLMs show a 30% accuracy drop; the authors break memory design into indexing, retrieval and reading, and propose optimizations for each.

CommentaryTwo of its abilities — knowledge updates and abstention, using the latest information and saying “I don’t know” when the answer is not there — are behaviours we consider essential for trust.