Self-RAG: self-reflective retrieval-augmented generation

Large language models can answer from their parameters, but a fixed retrieve-then-read pipeline may fetch irrelevant passages or skip evidence when it would help. Self-RAG, introduced by Asai, Wu, Wang, Sil, and Hajishirzi, trains a model to decide when to retrieve and to emit reflection tokens that evaluate retrieved evidence and its own generation. The paper reports results for its evaluated tasks and models; those results do not establish that every Self-RAG implementation is more accurate or reliable.

How Self-RAG works

Self-RAG is trained with special control tokens for retrieval and critique. At inference time, the model can request passages, score their relevance and support, and decide whether the resulting answer is useful. Retrieval and reflection become conditional parts of generation rather than mandatory steps for every query. The reflection scores are model outputs that can support selection or inspection; they do not prove that a passage or answer is correct.

The paper’s model emits the special [Retrieve] token when it predicts that external passages would help. A regular RAG application can implement a similar control flow, but it does not become the trained Self-RAG model without the model and reflection-token training described in the paper.

The current LlamaIndex query-engine documentation describes query-engine composition. It is a useful integration reference for a retriever and response synthesizer.

The snippets below are pseudocode for the control flow. They omit the model client, prompt templates, response types, and reflection-token parsers needed for a runnable implementation.

1. Generate and decide whether to retrieve

The initial generation can contain [Retrieve]. The application parses that learned token and calls its retriever only when the token is present.

def custom_query(self, query_str: str) -> Response:
    response = self.llm(prompt=_format_prompt(query_str), **_GENERATE_KWARGS)
    answer = response["choices"][0]["text"]
    source_nodes = []

    if "[Retrieve]" in answer:
        documents = self.retriever.retrieve(query_str)
        paragraphs = [
            _format_prompt(query_str, document.node.text)
            for document in documents
        ]
        critic_output = self._run_critic(paragraphs)
        answer, source_nodes = self._select_answer(critic_output)

    return Response(response=_postprocess_answer(answer), source_nodes=source_nodes)

If [Retrieve] is absent, the model can continue from its parameters. The token parser, retrieval budget, and permission checks are application responsibilities.

2. Retrieve candidate passages

The retriever searches a corpus and returns up to K candidate passages. K, ranking, freshness, and access controls are application choices. Retrieval alone does not establish that a passage is relevant or credible.

documents = self.retriever.retrieve(query_str)
paragraphs = [
    _format_prompt(query_str, document.node.text)
    for document in documents
]

3. Score relevance, support, and usefulness

The critic evaluates each candidate with reflection signals. source_nodes is a list because one entry is appended for each paragraph.

def _run_critic(self, paragraphs: list[str]) -> CriticOutput:
    paragraphs_final_score = {}
    llm_response_text = {}
    source_nodes = []

    for p_idx, paragraph in enumerate(paragraphs):
        prediction = self.llm(paragraph, **self.generate_kwargs)
        text = prediction["choices"][0]["text"]
        logprobs = prediction["choices"][0]["logprobs"]
        top_logprobs = logprobs["top_logprobs"]
        relevance = _relevance_score(top_logprobs[0])
        support = _is_supported_score(logprobs["tokens"], top_logprobs)
        usefulness = _is_useful_score(logprobs["tokens"], top_logprobs)

        llm_response_text[p_idx] = text
        paragraphs_final_score[p_idx] = relevance + support + 0.5 * usefulness
        source_nodes.append(
            NodeWithScore(
                node=TextNode(text=paragraph, id_=str(p_idx)),
                score=relevance,
            )
        )

    return CriticOutput(
        llm_response_text,
        paragraphs_final_score,
        source_nodes,
    )

The reflection signals are useful for ranking and filtering, subject to the calibration of the model and the task. They are not a correctness certificate.

4. Select or synthesize an answer

This small example selects one candidate, so its returned provenance matches one selected node.

def _select_answer(self, critic_output: CriticOutput):
    best_id = max(
        critic_output.paragraphs_final_score,
        key=critic_output.paragraphs_final_score.get,
    )
    answer = critic_output.llm_response_text[best_id]
    selected_nodes = [critic_output.source_nodes[best_id]]
    return answer, selected_nodes

An application that wants multi-document synthesis needs a separate synthesis prompt that receives the selected passages. It should return all passages used by that synthesis step. The single-candidate example above does not claim to perform multi-document synthesis.

5. Iterate within a budget

The model may emit another retrieval request while generating, and the paper describes parallel candidate evaluation at inference time. An application that supports repeated retrieval needs a stopping condition, a maximum number of retrievals, and a token or time budget. The example above performs one retrieval pass.

What the framework establishes

Self-RAG makes retrieval and evidence critique part of the model’s generation process. Its value depends on the trained model, retriever, corpus, reflection calibration, and evaluation setting. The paper’s results support its reported benchmark comparisons; an application still needs its own tests for retrieval quality, citation coverage, answer correctness, latency, and cost.

Resources

• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• Self-RAG project page
• LlamaIndex query-engine documentation