framesLLM

framesLLM is a chat-formatted adaptation of the FRAMES benchmark for evaluating retrieval-augmented generation (RAG) systems. It pairs each FRAMES question and reference answer with a fixed retrieved context embedded in a two-message conversation.

The dataset is designed to test two related capabilities:

• answering multi-hop questions from the supplied documents; and
• recognizing when the supplied context is insufficient to support the reference answer.

The source benchmark contains 824 English questions covering numerical reasoning, tabular reasoning, multiple constraints, temporal reasoning, and post-processing.

Dataset origin and transformation

The questions, reference answers, and reasoning labels originate from google/frames-benchmark, introduced in Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation.

In this adaptation, every example was enriched with retrieved document excerpts and converted to a chat-oriented representation:

1. a system message contains the response instructions and retrieved context, with numbered sources; and
2. a user message contains the original FRAMES question.

Splits

Split	Rows	Answerable	Not answerable	Recommended use
complete	824	87	737	Full benchmark evaluation
balanced	174	87	87	Class-balanced answerability evaluation and faster experiments

balanced contains every answerable example from complete and an equally sized subset of non-answerable examples. It is an evaluation subset, not a training split.

Dataset fields

Field	Type	Description
id	int64	Identifier inherited from the complete adapted dataset.
question	string	Original English multi-hop question.
gold_answer	string	Reference answer from FRAMES.
is_answerable	bool	Whether the embedded retrieved context is sufficient to support the reference answer.
reasoning_types	string	One or more reasoning labels separated by `
messages	list of {role, content}	Two-message chat representation: one system message followed by one user message.

The possible reasoning families are numerical reasoning, tabular reasoning, multiple constraints, temporal reasoning, and post-processing. A row may combine several families.

Loading the dataset

from datasets import load_dataset

complete = load_dataset(
    "Mvanypersele/framesLLM",
    split="complete",
)

balanced = load_dataset(
    "Mvanypersele/framesLLM",
    split="balanced",
)

To send the prepared conversation to a chat model:

example = complete[0]
messages = example["messages"]

The system prompt requires the answer to be grounded exclusively in the supplied context and to end with a source line. Evaluation code should check the prompt requirements as well as answer correctness.

Licensing and attribution

The original FRAMES benchmark is released under the Apache License 2.0. The embedded contexts include third-party source material, notably Wikipedia-derived excerpts, which may be governed by separate licenses and attribution requirements. For that reason this repository uses the Hugging Face metadata value license: other rather than applying Apache-2.0 to every stored passage.

Downstream users are responsible for reviewing the licenses of the underlying source documents and preserving required notices and attribution. The original FRAMES authors and paper must also be cited.

Citation

@inproceedings{krishna-etal-2025-fact,
    title = "Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation",
    author = "Krishna, Satyapriya and Krishna, Kalpesh and Mohananey, Anhad and Schwarcz, Steven and Stambler, Adam and Upadhyay, Shyam and Faruqui, Manaal",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.naacl-long.243/",
    doi = "10.18653/v1/2025.naacl-long.243",
    pages = "4745--4759",
}

Maintainers

Maintained by OpenLLM France. For questions about the adaptation or its construction, open a discussion in this dataset repository.