Closed-Book vs Open-Book

Controlled evaluation variants that test the same legal task under different information conditions to distinguish between parametric knowledge and document synthesis.

Category: Robust
CB High + OB High
Strong internal knowledge + effective document utilization.
Category: Research-Dependent
CB Low + OB High
Dependent on retrieval; limited innate legal understanding.
Category: Logic-Heavy
CB High + OB Low
Intrinsic understanding, but fails to extract context.
Category: Fundamental Gap
CB Low + OB Low
Insufficient capability for the assigned legal task.
The Mechanism

When a legal AI fails, the cause matters. Did it lack knowledge, or did it fail to find information? Most benchmarks conflate these. LegalChain isolates them by running the exact same task with and without source documents.

[ INPUT: CLOSED-BOOK ]
{
  "task": "Compare Citations",
  "anchor": "384 U.S. 436",
  "authority": "418 U.S. 683",
  "context": null
}
// Only metadata provided.
The Reasoning Bridge

The performance delta between modalities reveals the Reasoning Bridge. A model excelling closed-book demonstrates genuine legal understanding. Strong open-book performance reveals document analysis capability—extracting info from text it has never encountered.

[ INPUT: OPEN-BOOK ]
{
  "task": "Compare Citations",
  "anchor": "384 U.S. 436",
  "authority": "418 U.S. 683",
  "context": "RESEARCH_PACK_V2"
}
// Full opinion text injected.