Building and Optimizing Multi-Agent RAG Systems with DSPy and GEPA

Generated by ChatGPT

Recently, I have started to play around and learn more about DSPy and decided to write several blog posts as I learn more. And as the medical domain is one of my interests, I thought we could do a simple project there. In this post, we will build a multi-agent system with two subagents expert in two diseases, diabetes and COPD, each with a search tool in a vector DB of docs related to those diseases, then optimize each ReAct subagent’s prompts using GEPA, and then build a lead agent to use them, and will optimize that lead agent as well. You can find the code here:

GitHub - kargarisaac/dspy_gepa_optimization \ \ Contribute to kargarisaac/dspy_gepa_optimization development by creating an account on GitHub.\ \ github.com

Let’s get started.

Why Retrieval-Augmented Generation?

LLMs are powerful, but they can hallucinate. RAG mitigates this by retrieving relevant evidence from a trusted source — like your PDFs or a knowledge base — then asking the model to answer using the retrieved context. This grounding is critical for medical, legal, or scientific settings.

Import Packages and Set Up MLflow

Let’s first start with importing the required packages and setting up MLFlow for observability. We will use openrouter in this tutorial for LLM usage. Get your API key first, then put it in the .env file in your repo. Then run the MLflow server first:

mlflow server --backend-store-uri sqlite:///mydb.sqlite

For me, it fails, and I had to use the following and a file-based backend:

mlflow server --backend-store-uri file:///$(pwd)/mlruns --default-artifact-root ./mlartifacts --host 127.0.0.1 --port 5000

Then import the packages and set up mlflow and load env variables:

import dspy
import os
from dotenv import load_dotenv
# from dspy.adapters.baml_adapter import BAMLAdapter
import json
from dspy.evaluate import Evaluate
from dspy.teleprompt import GEPA
from dspy.teleprompt.gepa.gepa_utils import ScoreWithFeedback
from pathlib import Path
import random

# LangChain document loaders, text splitter, vectorstore and embeddings
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings   # wrapper for sentence-transformers
from langchain.vectorstores import FAISS
from langchain.schema import Document

import mlflow

mlflow.dspy.autolog(
    log_compiles=True,    # Track optimization process
    log_evals=True,       # Track evaluation results
    log_traces_from_compile=True  # Track program traces during optimization
)

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("med-ai-workshop")
mlflow.autolog()

load_dotenv()

Let’s do a simple test first. You can also use a local model using ollama if you want. Here is a simple code on how to use a local model. First download the model (here qwen3:4b):

ollama run qwen3:4b

Then use it in DSPy:

question = "What is a language model in one sentence?"
local_lm = dspy.LM(
        model='ollama_chat/qwen3:4b',
        api_base='http://localhost:11434',
        api_key=''
    )

print(local_lm(question))

Pretty simple. And this is the answer:

['\nOkay, the user is asking for a one-sentence definition of a language model. Hmm, this seems straightforward but I should be precise. They probably want something concise yet technically accurate—maybe they\'re a student, developer, or just curious about AI. \n\nFirst, I recall that language models are statistical models trained on text data. But "statistical" might be too vague for a one-liner. The key points are: probabilistic predictions, language patterns, and the purpose (generating/understanding text). \n\nWait, should I mention specific types like transformers? Nah, the user said "one sentence," so keeping it broad is better. They might not need the technical depth. \n\nI think the best approach is to emphasize *predicting next words* since that\'s the core function. Also, must clarify it\'s *learned from data*—not magic. \n\n*Brainstorming phrasing*: "A language model is a..." → too vague. "A machine learning model..." → better. "Trained on vast text data to predict the next word in a sequence..." → yes, that covers it without jargon. \n\nUser didn\'t specify context, but if they\'re new to this, avoiding terms like "probabilistic" or "transformers" is safer. Just stick to the basics: what it does (predicts text), how (learned from data), and why (for tasks like generation). \n\n*Double-checking*: Is "next word" too narrow? Well, for most practical purposes, it\'s accurate—advanced models handle longer contexts, but the one-sentence definition should stay simple. \n\nFinal phrasing: "A language model is a machine learning model trained on vast amounts of text data to predict the next word in a sequence, enabling tasks like text generation and understanding." \n\n...Wait, "enabling tasks" might be extra. User said *one sentence*, so I\'ll trim it to the essentials. *Simplifies*: "A language model is a machine learning system that learns patterns in text data to predict the next word, allowing it to generate or understand human language." \n\nYes, that\'s tight. Nailed it.\n\n\nA language model is a machine learning system trained on vast amounts of text data to predict the next word in a sequence, enabling tasks like text generation, translation, and understanding.']

Pretty long answer for such a simple question. As we want to use tool calling and agentic systems, we need to use a larger model in later steps.

Build a Vector Database from Your PDFs

We’ll use LangChain to load PDFs, split them into overlapping text chunks, embed them with Sentence Transformers, and store them in a FAISS vector database for fast similarity search.

Load and Chunk Documents

I have found two papers related to each disease and have them locally, then I load them and convert them to text and do the chunking. I put the files in the github repo. The following code does:

Loading two local PDF papers with LangChain
Splitting pages into overlapping chunks for dense retrieval
Creating embeddings with a Sentence Transformers model
Storing vectors in a local FAISS index for fast similarity search

"""
Build a FAISS vector DB from two local PDF papers using LangChain.
Outputs a directory "faiss_index" with the persisted vectorstore.
"""

# --- Config: change these paths to your downloaded PDFs ---
DIABETES_PDF_PATHS = ["docs/diabets1.pdf", "docs/diabets2.pdf"]   # <-- put your two PDF filenames here
COPD_PDF_PATHS = ["docs/copd1.pdf", "docs/copd2.pdf"]
OUTPUT_DIABETES_FAISS_DIR = "faiss_index/diabetes"
OUTPUT_COPD_FAISS_DIR = "faiss_index/copd"
EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
# chunk settings (tweak for your needs)
CHUNK_SIZE = 400
CHUNK_OVERLAP = 200

# - We use `PyPDFLoader` to read each PDF into page-level `Document` objects.
# - We add `source` and `page` metadata to every `Document` to keep track of provenance.
# - Keeping page granularity helps trace answers back to the original PDF.

def load_pdfs(paths):
    """Load PDFs into LangChain Document objects (keeps page-level granularity)."""
    all_docs = []
    for p in paths:
        p = Path(p)
        if not p.exists():
            raise FileNotFoundError(f"PDF not found: {p}")
        loader = PyPDFLoader(str(p))
        # load returns a list of Document objects (one per page typically)
        pages = loader.load()
        # add a source filename into metadata for traceability
        for i, doc in enumerate(pages):
            # ensure a copy of metadata dict (avoid mutating shared objects)
            meta = dict(doc.metadata or {})
            meta["source"] = str(p.name)
            meta["page"] = i
            doc.metadata = meta
        all_docs.extend(pages)
    return all_docs

Why these choices?

FAISS is a fast, proven vector database for approximate nearest neighbor search.
Overlapping chunks preserve context across boundaries for better retrieval. There are many better solutions as well, which you can use to improve the performance. Here I just use this simple way.
Sentence Transformers (MiniLM) provide lightweight, strong embeddings suitable for CPU.

def chunk_documents(documents, chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP):
    """Split documents into smaller chunks (keeps metadata)."""
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", " ", ""],
    )
    # split_documents returns list[Document] (with page_content and metadata)
    chunks = text_splitter.split_documents(documents)
    return chunks

Why chunking matters:

LLMs answer better when given concise, focused passages. We split long pages into

overlapping chunks so that:

Important sentences near boundaries are not lost (chunk_overlap)
Retrieval returns smaller, more relevant contexts for the model.

Build the Vector Stores

We can now build the vector stores:

def build_vectorstore(chunks, model_name=EMBEDDING_MODEL, save_dir=OUTPUT_DIABETES_FAISS_DIR):
    """Create embeddings and store them in a FAISS vectorstore, then persist to disk."""
    # Instantiate HuggingFaceEmbeddings wrapper (requires sentence-transformers installed)
    hf_emb = HuggingFaceEmbeddings(model_name=model_name,
                                   model_kwargs={"device": "cpu"})  # change to "cuda" if available

    # Build FAISS index from LangChain Document objects
    print("Creating FAISS vector store from", len(chunks), "chunks. This may take a while...")
    vectorstore = FAISS.from_documents(chunks, hf_emb)

    # Persist to disk
    vectorstore.save_local(save_dir)
    print(f"Saved FAISS vectorstore to: {save_dir}")
    return vectorstore, hf_emb

And then:

print("Loading Diabetes PDFs...")
docs = load_pdfs(DIABETES_PDF_PATHS)
print(f"Loaded {len(docs)} page-documents from {len(DIABETES_PDF_PATHS)} PDFs.")

print("Chunking Diabetes documents...")
chunks = chunk_documents(docs)
print(f"Produced {len(chunks)} chunks (chunk_size={CHUNK_SIZE}, overlap={CHUNK_OVERLAP}).")

diabetes_vectorstore, diabetes_embeddings = build_vectorstore(chunks, save_dir=OUTPUT_DIABETES_FAISS_DIR)

print("Loading COPD PDFs...")
docs = load_pdfs(COPD_PDF_PATHS)
print(f"Loaded {len(docs)} page-documents from {len(COPD_PDF_PATHS)} PDFs.")

print("Chunking COPD documents...")
chunks = chunk_documents(docs)
print(f"Produced {len(chunks)} chunks (chunk_size={CHUNK_SIZE}, overlap={CHUNK_OVERLAP}).")

copd_vectorstore, copd_embeddings = build_vectorstore(chunks, save_dir=OUTPUT_COPD_FAISS_DIR)

In order to use the retrieval as a tool, we need to define a function for it and here we build two tools for two subagents which we will build later:

def diabetes_vector_search_tool(query: str, k: int = 3) -> str:
    """
    A tool for the ReAct agent.
    Performs vector search and returns a formatted string of results.
    """
    results = diabetes_vectorstore.similarity_search_with_score(query, k=k)
    context = ""
    for i, (doc, score) in enumerate(results):
        doc_content = doc.page_content
        context += f"[PASSAGE {i+1}, score={score:.4f}]\n{doc_content}\\n\\n"
    return context

def copd_vector_search_tool(query: str, k: int = 3) -> str:
    """
    A tool for the ReAct agent.
    Performs vector search and returns a formatted string of results.
    """
    results = copd_vectorstore.similarity_search_with_score(query, k=k)
    context = ""
    for i, (doc, score) in enumerate(results):
        doc_content = doc.page_content
        context += f"[PASSAGE {i+1}, score={score:.4f}]\n{doc_content}\\n\\n"
    return context

These tools retrieve the top k similar chunks and put them together.

Build a Simple RAG System

In the next step, let’s build a simple RAG system. We use the tools we built in the previous step and retrieve related chunks to a query and then use an LLM to generate the answer for user query. Let’s define the LLM. We use the gpt-oss-20b model from OpenAI:

lm = dspy.LM(
    "openrouter/openai/gpt-oss-20b",  # LiteLLM route for OpenRouter models
    api_base="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
    model_type="chat",
    cache=False,
    temperature=0.3,
    max_tokens=64000
)
dspy.settings.configure(lm=lm)

We can then define the Signature for the RAG:

class RAGQA(dspy.Signature):
    """You are a helpful assistant. Answer a question using retrieved passages"""
    question: str = dspy.InputField()
    context: str = dspy.InputField()
    answer: str = dspy.OutputField()

In DSPy, we do not write the whole prompt. We use Signatures to define “what” we want the system to do, and then it converts it to a format and takes care of the “How” of doing the job. The Module we use is also important in building the prompt. Here we use the ChainOfThough module from DSPy. In the defined Signature above, we see that we define the task we want the system to do and then programmatically define the inputs and outputs of the system. We can then easily pass that to the module:

rag = dspy.ChainOfThought(RAGQA)

We can then retrieve the context and run the system:

question = "What is Gestational Diabetes Mellitus (GDM)?"
retrieved_context = diabetes_vector_search_tool(question, k=3)
rag(context=retrieved_context, question=question)

Note the inputs when we called the rag module and what we passed vs what we defined in the Signature. Let’s take a look at the prompt as well:

lm.inspect_history(n=1)

System message:

Your input fields are:
1. `question` (str):
2. `context` (str):
Your output fields are:
1. `reasoning` (str):
2. `answer` (str):
All interactions will be structured in the following way, with the appropriate values filled in.

[[ ## question ## ]]
{question}

[[ ## context ## ]]
{context}

[[ ## reasoning ## ]]
{reasoning}

[[ ## answer ## ]]
{answer}

[[ ## completed ## ]]
In adhering to this structure, your objective is:
        You are a helpful assistant. Answer a question using retrieved passages

You see the fields you have defined in the Signature, the task as docstring in that signature, and one extra reasoning filed which is added by DSPy because of the ChainOfThough module, and some extra structures. This is the prompt that goes to the LLM. And here is the user message and the LLM response:

User message:

[[ ## question ## ]]
What is Gestational Diabetes Mellitus (GDM)?

[[ ## context ## ]]
[PASSAGE 1, score=0.2890]
with pregnancy, gestational diabetes mellitus (GDM) results in hyperglycemia.
GDM is essential to detect and treat during pregnancy due to the harmful impact it has on both the mother
and the fetus, in both the short and long runs. Gestational diabetes can cause short-term pregnancy
complications such as high blood pressure, the necessity for a cesarean section (C-section), pre-eclampsia,

[PASSAGE 2, score=0.2915]
Research, Wardha, IND
Corresponding author:
Vaishnavi S. Nakshine,
isaac@nazmi.tech
Abstract
This review article conducts a comprehensive analysis of gestational diabetes mellitus (GDM) and its
ramifications for both maternal health and the well-being of their offspring. GDM is a significant pregnancy

[PASSAGE 3, score=0.2931]
2.
 Pharmacology and Therapeutics, Jawaharlal Nehru Medical College, Datta Meghe Institute of Higher Education and
Research, Wardha, IND
Corresponding author:
Vaishnavi S. Nakshine,
isaac@nazmi.tech
Abstract
This review article conducts a comprehensive analysis of gestational diabetes mellitus (GDM) and its

Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## answer ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`.

Response:

[[ ## reasoning ## ]]
Gestational Diabetes Mellitus (GDM) is a form of diabetes that develops during pregnancy. It is characterized by hyperglycemia that is first recognized while the woman is pregnant and typically resolves after delivery. GDM is important to detect and treat because it can have harmful effects on both the mother and the fetus. The condition can lead to short-term pregnancy complications such as high blood pressure, the need for a cesarean section, and pre-eclampsia, and it also poses risks for long-term health of both mother and child. The retrieved passages describe GDM as a pregnancy-associated hyperglycemia that has significant ramifications for maternal health and the well-being of offspring.

[[ ## answer ## ]]
Gestational Diabetes Mellitus (GDM) is a type of diabetes that appears during pregnancy, characterized by high blood sugar levels that are first detected while the woman is pregnant and usually resolve after delivery. It is important to identify and manage because it can lead to serious short-term complications for the mother—such as high blood pressure, pre-eclampsia, and the need for a cesarean section—as well as adverse outcomes for the fetus. GDM has significant implications for both maternal and child health, making early detection and treatment essential.

Pretty easy, fast, and amazing!

This was a RAG system without any agentic behavior. We retrieved the context, fed it into the model, and got the response. Next, let’s build an agent.

Build a Diabetes ReAct Agent and Evaluate

Let’s now build our first subagent. We will also use GEPA, which is a prompt optimizer or prompt reworder, to update the prompt. For that, we need a dataset of queries and the ground truth response first. To do that, I simply provided the files to ChatGPT and asked it to build me a dataset. Here are a few samples (you can find the rest in the repo):

  {
    "question": "Integrate the ADA/WHO glucose-based diagnostic criteria with the IEC HbA1c rule and explain how you would confirm diabetes in a non-pregnant adult with FPG 7.2 mmol/L, 2-h OGTT 10.8 mmol/L, and HbA1c 6.4%. Include why a single test is insufficient in some scenarios and what to repeat.",
    "answer": "ADA/WHO criteria diagnose diabetes when FPG ≥7.0 mmol/L (126 mg/dL) or 2-h OGTT ≥11.1 mmol/L (200 mg/dL), usually on two separate occasions unless classic symptoms are present. Here, FPG is diagnostic, while the 2-h OGTT is below the 11.1 mmol/L threshold, creating discordance that warrants repeating the abnormal test on a different day to confirm. The IEC permits diagnosis at HbA1c ≥6.5%; at 6.4% this remains "high-risk," not diagnostic, so HbA1c alone would not confirm. Therefore, repeat FPG (or perform a second glucose-based test) to confirm diabetes on specificity-oriented grounds outlined by the IEC."
  },
  {
    "question": "Using the GDM screening/management flow and ADA's first-prenatal risk assessment, outline a stepwise pathway (screen → confirm → treat) for a pregnant patient with risk factors and an abnormal screen. Clarify when to choose one-step versus two-step strategies and how that impacts case load.",
    "answer": "Risk assessment should occur at the first prenatal visit to identify women needing early screening. When screened and found abnormal, proceed into the GDM pathway: confirmatory testing, then diet/exercise, then pharmacotherapy if targets are not met within ~10–14 days. The one-step approach captures two to three times more GDM than the two-step approach, increasing case load and downstream treatment volume; programs should plan resources accordingly. The figures and narrative emphasize structured escalation after confirmation."
  },
  {
    "question": "Design a postpartum surveillance plan for a woman with prior GDM that minimizes future T2DM risk. Specify testing cadence, lifestyle pillars, and the rationale based on beta-cell compensation decline and obesity-related markers.",
    "answer": "Women with prior GDM face ~20–60% risk of progressing to diabetes within 5–10 years; surveillance should therefore be long-term. Plan regular examinations and testing at least every 1–3 years, with counseling focused on diet, sustained physical activity, and weight control as first-line defenses. The review links progression to declining beta-cell compensation, weight gain, higher CRP, and lower adiponectin, supporting intensive lifestyle measures to mitigate obesity's metabolic effects."
  }

We can then load the JSON file of question-answer pairs and convert it to DSPy dataset format:

# Load the dataset
with open("docs/qa_pairs_diabets.json", "r") as f:
    qa_diabetes_data = json.load(f)

# Convert to dspy.Example objects
dataset_diabetes = [dspy.Example(question=item["question"], answer=item["answer"]).with_inputs("question") for item in qa_diabetes_data]

# shuffle the dataset
random.shuffle(dataset_diabetes)

# Split the dataset as requested
train_size = 20
trainset_diabetes = dataset_diabetes[:train_size]
devset_diabetes = dataset_diabetes[train_size:]

print(f"Loaded {len(dataset_diabetes)} examples.")
print(f"Train set size: {len(trainset_diabetes)}")
print(f"Dev set size: {len(devset_diabetes)}")

Loaded 49 examples.
Train set size: 20
Dev set size: 29

Now let’s build the agent. Almost the same as what we did before, but using a ReAct module from DSPy. Here I write a class for this agent in a more standard way, which you can add more stuff to as well. You can also do the same as before, completely fine. Here again, we define the Signature, but without context, as the agent will use the tool internally to retrieve and utilize the context.

class ReActSignature(dspy.Signature):
    """You are a helpful assistant. Answer user's question."""
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

class DiabetesAgent(dspy.Module):
    def __init__(
        self
    ):
        # init LLM
        self.lm = dspy.LM(
            "openrouter/openai/gpt-oss-20b",
            api_key=os.getenv("OPENROUTER_API_KEY"),
            temperature=0.3,
            max_tokens=64000,
            cache=False
        )
        dspy.configure(lm=self.lm)
        self.diabetes_agent = dspy.ReAct(ReActSignature, tools=[diabetes_vector_search_tool])

    def forward(self, question: str):
        return self.diabetes_agent(question=question)

diabetes_agent = DiabetesAgent()

Now let’s test it:

diabetes_agent(question="What are the main treatments for Type 2 diabetes?")

And then let’s see what has happened:

diabetes_agent.lm.inspect_history(n=2)

Let’s now explore what happens under the hood when we call the ReAct agent. First, the system message:

System message:

Your input fields are:
1. `question` (str):
2. `trajectory` (str):
Your output fields are:
1. `next_thought` (str):
2. `next_tool_name` (Literal['diabetes_vector_search_tool', 'finish']):
3. `next_tool_args` (dict[str, Any]):
All interactions will be structured in the following way, with the appropriate values filled in.

[[ ## question ## ]]
{question}

[[ ## trajectory ## ]]
{trajectory}

[[ ## next_thought ## ]]
{next_thought}

[[ ## next_tool_name ## ]]
{next_tool_name}        # note: the value you produce must exactly match (no extra characters) one of: diabetes_vector_search_tool; finish

[[ ## next_tool_args ## ]]
{next_tool_args}        # note: the value you produce must adhere to the JSON schema: {"type": "object", "additionalProperties": true}

[[ ## completed ## ]]
In adhering to this structure, your objective is:
        You are a helpful assistant. Answer user's question.

        You are an Agent. In each episode, you will be given the fields `question` as input. And you can see your past trajectory so far.
        Your goal is to use one or more of the supplied tools to collect any necessary information for producing `answer`.

        To do this, you will interleave next_thought, next_tool_name, and next_tool_args in each turn, and also when finishing the task.
        After each tool call, you receive a resulting observation, which gets appended to your trajectory.

        When writing next_thought, you may reason about the current situation and plan for future steps.
        When selecting the next_tool_name and its next_tool_args, the tool must be one of:

        (1) diabetes_vector_search_tool, whose description is <desc>      A tool for the ReAct agent.      Performs vector search and returns a formatted string of results.      </desc>. It takes arguments {'query': {'type': 'string'}, 'k': {'type': 'integer', 'default': 3}}.
        (2) finish, whose description is <desc>Marks the task as complete. That is, signals that all information for producing the outputs, i.e. `answer`, are now available to be extracted.</desc>. It takes arguments {}.
        When providing `next_tool_args`, the value inside the field must be in JSON format

Note how the system message is created and what template it has, and how it is different from the ChainOfThough module we used before. It is created and contains the trajectory, thinking, and tool information. Note the tool we provided and the finish tool that the agent uses to end the episode.

Then the user message is created from the user query and some more instructions:

User message:

[[ ## question ## ]]
What are the main treatments for Type 2 diabetes?

[[ ## trajectory ## ]]

Respond with the corresponding output fields, starting with the field `[[ ## next_thought ## ]]`, then `[[ ## next_tool_name ## ]]` (must be formatted as a valid Python Literal['diabetes_vector_search_tool', 'finish']), then `[[ ## next_tool_args ## ]]` (must be formatted as a valid Python dict[str, Any]), and then ending with the marker for `[[ ## completed ## ]]`.

And then the LLM response, which thinks about the query and decides the tool to use and the args for that:

Response:

[[ ## next_thought ## ]]
I need to retrieve up-to-date information on the main treatments for Type 2 diabetes. I will use the diabetes_vector_search_tool to get the most relevant documents.

[[ ## next_tool_name ## ]]
diabetes_vector_search_tool

[[ ## next_tool_args ## ]]
{"query": "main treatments for type 2 diabetes", "k": 3}

[[ ## completed ## ]]

Then again, we have the same system message, which I do not repeat, and then a new user message which has the trajectory of the agent so far:

User message:

[[ ## question ## ]]
What are the main treatments for Type 2 diabetes?

[[ ## trajectory ## ]]
[[ ## thought_0 ## ]]
I need to retrieve up-to-date information on the main treatments for Type 2 diabetes. I will use the diabetes_vector_search_tool to get the most relevant documents.

[[ ## tool_name_0 ## ]]
diabetes_vector_search_tool

[[ ## tool_args_0 ## ]]
{"query": "main treatments for type 2 diabetes", "k": 3}

[[ ## observation_0 ## ]]
[PASSAGE 1, score=0.5143]
emerging epidemic. Novel drugs are being developed, yet no cure
is available in sight for the disease, despite new insight into the
pathophysiology of the disease. Management should be tailored to
improve the quality of life of individuals with type 2 DM.
References
1.  A hmed AM. History of diabetes mellitus. Saudi Med J 2002 Apr;23(4):373-
378.
2.  D

[PASSAGE 2, score=0.6916]
on World Health Organization (WHO) and American Diabetes
Association (ADA) criteria which include both clinical and
laboratory parameters. No cure has yet been found for the disease;
however, treatment modalities include lifestyle modifications,
treatment of obesity, oral hypoglycemic agents, and insulin
sensitizers like metformin, a biguanide that reduces insulin

[PASSAGE 3, score=0.7153]
however, treatment modalities include lifestyle modifications,
treatment of obesity, oral hypoglycemic agents, and insulin
sensitizers like metformin, a biguanide that reduces insulin
resistance, is still the recommended first line medication especially
for obese patients. Other effective medications include non-
sulfonylurea secretagogues, thiazolidinediones, alpha glucosidase

Respond with the corresponding output fields, starting with the field `[[ ## next_thought ## ]]`, then `[[ ## next_tool_name ## ]]` (must be formatted as a valid Python Literal['diabetes_vector_search_tool', 'finish']), then `[[ ## next_tool_args ## ]]` (must be formatted as a valid Python dict[str, Any]), and then ending with the marker for `[[ ## completed ## ]]`.

We can see the first phase of thinking, tool selection, execution, and the retrieved context in it. And then the LLM response is as below, which decides to use the finish tool and end the run:

Response:

[[ ## next_thought ## ]]
The vector search returned relevant passages that list the main treatments for Type 2 diabetes, including lifestyle modifications, weight management, oral hypoglycemic agents (metformin, sulfonylureas, non-sulfonylurea secretagogues, thiazolidinediones, alpha-glucosidase inhibitors), and insulin therapy. This information is sufficient to answer the question.

[[ ## next_tool_name ## ]]
finish

[[ ## next_tool_args ## ]]
{}

And then it uses a different system message and the trajectory, and the retrieved context, and generates the final answer:

System message:

Your input fields are:
1. `question` (str):
2. `trajectory` (str):
Your output fields are:
1. `reasoning` (str):
2. `answer` (str):
All interactions will be structured in the following way, with the appropriate values filled in.

[[ ## question ## ]]
{question}

[[ ## trajectory ## ]]
{trajectory}

[[ ## reasoning ## ]]
{reasoning}

[[ ## answer ## ]]
{answer}

[[ ## completed ## ]]
In adhering to this structure, your objective is:
        You are a helpful assistant. Answer user's question.

User message:

[[ ## question ## ]]
What are the main treatments for Type 2 diabetes?

[[ ## trajectory ## ]]
[[ ## thought_0 ## ]]
I need to retrieve up-to-date information on the main treatments for Type 2 diabetes. I will use the diabetes_vector_search_tool to get the most relevant documents.

[[ ## tool_name_0 ## ]]
diabetes_vector_search_tool

[[ ## tool_args_0 ## ]]
{"query": "main treatments for type 2 diabetes", "k": 3}

[[ ## observation_0 ## ]]
[PASSAGE 1, score=0.5143]
emerging epidemic. Novel drugs are being developed, yet no cure
is available in sight for the disease, despite new insight into the
pathophysiology of the disease. Management should be tailored to
improve the quality of life of individuals with type 2 DM.
References
1.  A hmed AM. History of diabetes mellitus. Saudi Med J 2002 Apr;23(4):373-
378.
2.  D

[PASSAGE 2, score=0.6916]
on World Health Organization (WHO) and American Diabetes
Association (ADA) criteria which include both clinical and
laboratory parameters. No cure has yet been found for the disease;
however, treatment modalities include lifestyle modifications,
treatment of obesity, oral hypoglycemic agents, and insulin
sensitizers like metformin, a biguanide that reduces insulin

[PASSAGE 3, score=0.7153]
however, treatment modalities include lifestyle modifications,
treatment of obesity, oral hypoglycemic agents, and insulin
sensitizers like metformin, a biguanide that reduces insulin
resistance, is still the recommended first line medication especially
for obese patients. Other effective medications include non-
sulfonylurea secretagogues, thiazolidinediones, alpha glucosidase

[[ ## thought_1 ## ]]
The vector search returned relevant passages that list the main treatments for Type 2 diabetes, including lifestyle modifications, weight management, oral hypoglycemic agents (metformin, sulfonylureas, non-sulfonylurea secretagogues, thiazolidinediones, alpha-glucosidase inhibitors), and insulin therapy. This information is sufficient to answer the question.

[[ ## tool_name_1 ## ]]
finish

[[ ## tool_args_1 ## ]]
{}

[[ ## observation_1 ## ]]
Completed.

Respond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## answer ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`.

Response:

[[ ## reasoning ## ]]
The vector search returned passages that list the main treatment modalities for type 2 diabetes. These include lifestyle and weight-management strategies, first-line oral hypoglycemic agents (metformin), other oral agents (sulfonylureas, meglitinides, thiazolidinediones, alpha-glucosidase inhibitors, GLP-1 receptor agonists, SGLT2 inhibitors), and insulin therapy when required. This information is sufficient to answer the question.

[[ ## answer ## ]]
The main treatments for type 2 diabetes are:

1. **Lifestyle and weight management**
   * Healthy diet (low-glycaemic-index foods, portion control)
   * Regular physical activity (≥150 min/week of moderate exercise)
   * Weight loss for overweight/obese patients

2. **Oral hypoglycaemic agents** – usually started with **metformin** (first-line).
   Other options include:
   * Sulfonylureas or meglitinides (stimulate insulin secretion)
   * Thiazolidinediones (improve insulin sensitivity)
   * Alpha-glucosidase inhibitors (delay carbohydrate absorption)
   * GLP-1 receptor agonists (increase insulin, suppress glucagon)
   * SGLT2 inhibitors (promote urinary glucose excretion)

3. **Insulin therapy** – added when glycaemic targets cannot be met with oral agents or in cases of severe hyperglycaemia.

4. **Adjunctive measures** – blood-pressure control, lipid management, and regular monitoring of HbA1c, fasting glucose, and complications.

These strategies are tailored to each patient's clinical profile and treatment goals.

[[ ## completed ## ]]

Amazing. We built an agent and explored how it works under the hood and how it thinks, selects the tool, executes it, and continues until it is satisfied with the context, and then generates the answer and ends the run. Now, let’s evaluate this agent on the dataset we have created.

GEPA Optimization

In GEPA evaluation and optimization, we rely on a single metric function that produces both a numeric score and explanatory feedback. This function operates in two modes:

Program-level (overall answer): No pred_name or pred_trace is provided. The metric judges the model’s final answer against the gold answer using an LLM judge.
Predictor-level (module reflection): pred_name and pred_trace are provided. The metric delivers targeted feedback for the specific predictor, such as the ReAct loop.

The metric function is central to GEPA. Its design starts with the question: What decisions must it support? Because GEPA uses one metric for both overall scoring and module-level feedback, the function must adapt to the context. At the program level, it evaluates the final answer against the gold. At the module level, it analyzes the predictor’s behavior and provides concrete, actionable guidance.

For program-level judging, we use an LLM judge since outputs are open-ended text. String matching is too limited to capture qualities like correctness, completeness, and clarity. The judge returns a strict score in 0,10,1 along with a short justification. This makes scores easy to compare and provides clear reasons for changes.

For predictor-level judging, such as the ReAct loop, we combine heuristics with an LLM check. Heuristics enforce hard rules — valid JSON arguments, a non-empty query, a reasonable k, and avoiding premature termination. The LLM judge then evaluates whether the chosen tool and query are appropriate for the question. The metric outputs both a score and actionable feedback so GEPA can improve the loop’s instructions, not just rate them.

Our implementation, llm_metric_prediction, handles both cases:

Program-level (default): Uses JudgeConsistency to compare the final answer with the gold. Produces a [0, 1] score and short justification.
Predictor-level (e.g., ReAct loop): When pred_name matches the loop, the metric inspects pred_trace to check next_tool_name and next_tool_args. The step is scored by:
Heuristics: JSON validity, non-empty query, reasonable k, and no early finish.
LLM judge: JudgeReactStep assesses whether the tool and arguments fit the question.
Combined score: An average of heuristic and LLM scores, with targeted feedback for GEPA’s optimizer.

By branching internally on pred_name, this single metric function supports both whole-program evaluation and module-specific optimization, keeping the design unified while allowing tailored feedback.

Here is our implementation:

class JudgeConsistency(dspy.Signature):
    """Judge whether the predicted answer matches the gold answer.

    # Instructions:
    - The score should be between 0.0 and 1.0 and based on the similarity of the predicted answer and the gold answer.
    - The justification should be a brief explanation of the score.
    - If the answer doesn't address the question properly, the score should be less than 0.5.
    - If the answer is completely correct, the score should be 1.0. Otherwise, the score should be less than 1.0.
    - Be very strict in your judgement as this is a medical question.
    """
    question: str = dspy.InputField()
    gold_answer: str = dspy.InputField()
    predicted_answer: str = dspy.InputField()
    score: float = dspy.OutputField(desc="a float score between 0 and 1")
    justification: str = dspy.OutputField()

class JudgeReactStep(dspy.Signature):
    """Judge whether the next tool call (name + args) is appropriate, well-formed, and relevant.

    - Output a strict score in [0, 1].
    - Provide a brief justification and a yes/no style verdict in justification text.
    """
    question: str = dspy.InputField()
    tool_name: str = dspy.InputField()
    tool_args_json: str = dspy.InputField()
    score: float = dspy.OutputField(desc="a float score between 0 and 1")
    verdict: str = dspy.OutputField()
    justification: str = dspy.OutputField()

def llm_metric_prediction(*args, **kwargs):
    """Metric returning ScoreWithFeedback for GEPA and Evaluate.

    Accepts flexible arguments because GEPA may pass additional positional
    parameters (e.g., program, trace, batch metadata). We only need `example`
    and `pred` here; the rest are ignored.
    """
    # GEPA may pass predictor context for per-predictor feedback
    pred_name = kwargs.get("pred_name")
    pred_trace = kwargs.get("pred_trace")
    if pred_name is not None:
        logger.info(f"metric called for predictor={pred_name}")

    # Extract example and prediction from positional/keyword args
    example = kwargs.get("example") or kwargs.get("gold")
    pred = kwargs.get("pred") or kwargs.get("prediction")
    if example is None and len(args) > 0:
        example = args[0]
    if pred is None and len(args) > 1:
        pred = args[1]

    # Special handling: when optimizing the ReAct loop predictor
    if pred_name and (pred_name == "react" or pred_name.endswith(".react")) and pred_trace:
        try:
            _, step_inputs, step_outputs = pred_trace[0]
        except Exception:
            step_inputs, step_outputs = {}, {}

        question_text = getattr(example, "question", None) or step_inputs.get("question", "") or ""

        # Read tool name/args from the predictor's outputs (dict or Prediction)
        def _get(o, key, default=""):
            if isinstance(o, dict):
                return o.get(key, default)
            return getattr(o, key, default)

        tool_name = _get(step_outputs, "next_tool_name", "")
        tool_args = _get(step_outputs, "next_tool_args", {})

        # Heuristics: well-formed JSON args and sensible fields
        args_is_dict = isinstance(tool_args, dict)
        has_query = args_is_dict and isinstance(tool_args.get("query"), str) and tool_args.get("query", "").strip() != ""
        k_val = tool_args.get("k") if args_is_dict else None
        k_ok = isinstance(k_val, int) and 1 <= k_val <= 10 or k_val is None
        used_tool = tool_name not in ("", "finish")
        early_finish = tool_name == "finish"

        logger.debug(
            "react-step details | used_tool=%s tool=%s args_keys=%s has_query=%s k=%s early_finish=%s pred_trace_len=%s",
            used_tool,
            str(tool_name),
            list(tool_args.keys()) if isinstance(tool_args, dict) else type(tool_args).__name__,
            has_query,
            k_val,
            early_finish,
            len(pred_trace) if pred_trace else 0,
        )

        heuristics_score = 0.0
        if used_tool:
            heuristics_score += 0.4
        if has_query:
            heuristics_score += 0.4
        if k_ok:
            heuristics_score += 0.1
        if not early_finish:
            heuristics_score += 0.1
        heuristics_score = max(0.0, min(1.0, heuristics_score))

        # LLM judge for the loop step (tool choice + query relevance)
        tool_args_json = json.dumps(tool_args) if isinstance(tool_args, dict) else str(tool_args)
        with dspy.settings.context(lm=lm):
            react_judge = dspy.Predict(JudgeReactStep)
            judged = react_judge(
                question=question_text,
                tool_name=str(tool_name),
                tool_args_json=tool_args_json,
            )

        llm_score = getattr(judged, "score", 0.0) or 0.0
        llm_score = max(0.0, min(1.0, llm_score))
        llm_just = getattr(judged, "justification", "") or ""

        total = 0.5 * heuristics_score + 0.5 * llm_score

        logger.info(
            "react-step scores | heuristics=%.3f llm=%.3f total=%.3f",
            heuristics_score,
            llm_score,
            total,
        )

        # Actionable feedback
        suggestions = []
        if not used_tool:
            suggestions.append("Select a retrieval tool before finishing.")
        if early_finish:
            suggestions.append("Avoid selecting 'finish' until you have evidence from the retrieval tool.")
        if not args_is_dict:
            suggestions.append("Emit next_tool_args as a valid JSON object.")
        else:
            if not has_query:
                suggestions.append("Include a non-empty 'query' string in next_tool_args.")
            if k_val is not None and (not isinstance(k_val, int) or k_val < 1 or k_val > 10):
                suggestions.append("Choose a reasonable k (e.g., 3-5).")
        if not suggestions:
            suggestions.append("Good step. Keep queries concise and set k=5 by default.")

        feedback_text = (
            f"ReAct step — LLM score: {llm_score:.2f}, heuristics: {heuristics_score:.2f}. "
            + " ".join(suggestions)
            + (f" LLM justification: {llm_just}" if llm_just else "")
        ).strip()

        return ScoreWithFeedback(score=total, feedback=feedback_text)

    # Program-level or non-react predictor: judge final answer quality
    # Defensive checks
    if example is None or pred is None:
        return ScoreWithFeedback(score=0.0, feedback="Missing example or pred")

    predicted_answer = getattr(pred, "answer", None) or ""
    if not predicted_answer.strip():
        return ScoreWithFeedback(score=0.0, feedback="Empty prediction")

    with dspy.settings.context(lm=lm):
        judge = dspy.Predict(JudgeConsistency)
        judged = judge(
            question=example.question,
            gold_answer=example.answer,
            predicted_answer=predicted_answer,
        )

    score = getattr(judged, "score", None) or 0.0
    score = max(0.0, min(1.0, score))
    justification = getattr(judged, "justification", "") or ""
    logger.info("answer-level score=%.3f for question='%s'", score, (example.question[:80] + "...") if len(example.question) > 80 else example.question)
    feedback_text = f"Score: {score}. {justification}".strip()
    return ScoreWithFeedback(score=score, feedback=feedback_text)

Then we can define the evaluator and do the evaluation on the dataset we created. We pass the dataset, and then the agent and the metric. We can run it several times and consider the mean performance as well.

evaluator_diabetes = Evaluate(devset=devset_diabetes, num_threads=32, display_progress=True, display_table=5, provide_traceback=True)
print("Evaluating the baseline ReAct agent...")
diabetes_baseline_eval = evaluator_diabetes(diabetes_agent, metric=llm_metric_prediction)
diabetes_baseline_eval

EvaluationResult(score=90.72, results=<list of 29 results>)

The performance is quite good, even with the simple prompt we provided and with the help of DSPy templates and what it adds under the hood!

Optimize the Agent using GEPA

In order to use GEPA for prompt optimization, we need a reflection LLM as well.

# Teacher LM for reflection (GEPA)
teacher_lm = dspy.LM(
    "openrouter/openai/gpt-oss-120b",
    api_base="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
    model_type="chat",
    cache=False,
    temperature=0.3,
    max_tokens=64000
)

We use a larger model like gpt-oss-120b as the reflection LLM, which gets the feedback texts and the agent’s traces, reflects, and proposes a better prompt.

Let’s first see how GEPA works. GEPA is like a smart system that helps you improve your prompts or programs automatically. You start with a rough version (your “seed”), and GEPA makes it better step by step — by testing, critiquing, and evolving it. GEPA uses three different LLMs, each with its own job:

Student (your program)

This is the prompt or agent you want to improve. It tries to answer questions or perform tasks. GEPA treats this as the thing it’s trying to evolve.
Judge (metric)

This LLM looks at the Student’s answers and scores them. It checks how good they are, maybe comparing them to ground truth answers. It doesn’t just give a score — it can explain why an answer is good or bad.
Teacher (reflector)

This is the smartest LLM. It reads the Judge’s feedback and the Student’s mistakes. Then it suggests edits: how to rewrite the prompt, change instructions, or improve examples to make the Student better next time.

In simple terms, GEPA works as follows:

Start with your first version

You give GEPA your initial prompt or DSPy program. This is your “Student.”

Make some quick edits (mutations)

GEPA creates a few slightly changed versions of your prompt — like rewording instructions or changing the order of examples — just to explore other options.

Test each version

It runs all versions on some examples (train/dev data) and asks the Judge to score them. GEPA tracks how well each one performs.

Analyze mistakes

For any version that does poorly, GEPA gives the failed examples to the Teacher. The Teacher explains what went wrong and suggests exact changes to fix the problem.

Create improved versions

GEPA takes the Teacher’s suggestions and makes new versions of the prompt with those edits.

Pick the best ones

After each round, GEPA keeps only the top-performing versions — those that are most accurate, efficient, and compact. It throws away the weaker ones.

Repeat until the budget runs out

GEPA keeps repeating the process — test, critique, edit, select — until it runs out of budget (how many times it’s allowed to run the LLMs), or the improvements stop.

Return the best version

In the end, GEPA gives you the final, optimized version of your program. This is usually shorter, clearer, and more accurate than what you started with.

Let’s first see how the non-optimized agent is:

diabetes_agent.react = Predict(StringSignature(question, trajectory -> next_thought, next_tool_name, next_tool_args
    instructions="""
You are a helpful assistant. Answer user's question.

You are an Agent. In each episode, you will be given the fields `question` as input.
And you can see your past trajectory so far.
Your goal is to use one or more of the supplied tools to collect any necessary information for producing `answer`.
To do this, you will interleave next_thought, next_tool_name, and next_tool_args in each turn, and also when finishing the task.
After each tool call, you receive a resulting observation, which gets appended to your trajectory.

When writing next_thought, you may reason about the current situation and plan for future steps.
When selecting the next_tool_name and its next_tool_args, the tool must be one of:
(1) diabetes_vector_search_tool, whose description is <desc>      A tool for the ReAct agent.      Performs vector search and returns a formatted string of results.      </desc>. It takes arguments {'query': {'type': 'string'}, 'k': {'type': 'integer', 'default': 3}}.
(2) finish, whose description is <desc>Marks the task as complete. That is, signals that all information for producing the outputs, i.e. `answer`, are now available to be extracted.</desc>. It takes arguments {}.
When providing `next_tool_args`, the value inside the field must be in JSON format
"""
    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})
    trajectory = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Trajectory:', 'desc': '${trajectory}'})
    next_thought = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Thought:', 'desc': '${next_thought}'})
    next_tool_name = Field(annotation=Literal['diabetes_vector_search_tool', 'finish'] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Name:', 'desc': '${next_tool_name}'})
    next_tool_args = Field(annotation=dict[str, Any] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Args:', 'desc': '${next_tool_args}'})
))
diabetes_agent.extract.predict = Predict(StringSignature(question, trajectory -> reasoning, answer
    instructions="You are a helpful assistant. Answer user's question."
    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})
    trajectory = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Trajectory:', 'desc': '${trajectory}'})
    reasoning = Field(annotation=str required=True json_schema_extra={'prefix': "Reasoning: Let's think step by step in order to", 'desc': '${reasoning}', '__dspy_field_type': 'output'})
    answer = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Answer:', 'desc': '${answer}'})
))

If I understood correctly, and based on the above output and what we saw in the ReAct loop, there are two modules in the ReAct module in DSPy. One for the ReAct loop (question, trajectory -> next_thought, next_tool_name, next_tool_args) and then when it is satisfied, it finishes the loop and goes to generating the answer (question, trajectory -> reasoning, answer) with two different schema and task definitions.

To focus optimization on the ReAct loop, we can freeze the answerer (e.g., agent.extract._compiled = True). We also need to set add_format_failure_as_feedback to True. Now, let’s define the optimizer and run it:

diabetes_agent.agent.extract._compiled = True
diabetes_agent.agent.react._compiled = False

teleprompter = GEPA(
    metric=llm_metric_prediction,
    max_full_evals=2,
    num_threads=32,
    track_stats=True,
    track_best_outputs=True,
    add_format_failure_as_feedback=True,
    reflection_lm=teacher_lm,
)

optimized_diabetes_agent = teleprompter.compile(student=diabetes_agent, trainset=trainset_diabetes, valset=devset_diabetes)

And now let’s see the optimized agent’s prompt (note that as we froze the answerer part, it doesn’t show it here):

results = optimized_diabetes_agent.detailed_results

# Get all candidates and their validation scores
candidates = results.candidates
val_scores = results.val_aggregate_scores

# Find the best candidate by validation score
best_idx = results.best_idx  # This is automatically calculated
best_score = val_scores[best_idx]
best_candidate = results.best_candidate

print(f"Best candidate index: {best_idx}")
print(f"Best validation score: {best_score}")
print(f"Best candidate components: {best_candidate}")

Best candidate index: 1
Best validation score: 0.9470000000000001
Best candidate components: agent.react = Predict(StringSignature(question, trajectory -> next_thought, next_tool_name, next_tool_args
    instructions="""
## Task Overview
You are a **ReAct** agent that must answer a medical question about diabetes (gestational, type 2, or related therapies).
The agent can only gather evidence by calling the **`diabetes_vector_search_tool`**.
Once all necessary information is collected, the agent must signal completion with the **`finish`** tool and then produce the final answer
### Required Output Format

For every turn you must output **exactly** the following fields (in this order):
1. `next_thought` – a short paragraph explaining your current reasoning, what you plan to do next, and why.
2. `next_tool_name` – the name of the tool you will call next.
  *If you are finished, this must be `finish`.*
  *Never output `None` as a tool name.*
3. `next_tool_args` – a JSON object containing the arguments for the chosen tool.
  *If the tool is `finish`, this must be `{}`.*

After the `finish` call, you must output a single field:
4. `answer` – the final, fully-formed answer to the user's question.
**No other text or fields are allowed.**
If you need to call the vector search tool again, repeat the three fields above with a new `next_tool_name` of `diabetes_vector_search_tool` and appropriate arguments.

### Tool Specifications
| Tool | Arguments | Description |
|------|-----------|-------------|
| `diabetes_vector_search_tool` | `{"query": string, "k": integer (default 3)}` | Performs a vector search over a curated diabetes literature corpus and returns the top-k passages as a single string. |
| `finish` | `{}` | Signals that you have all the information needed to produce the final answer. |

### Interaction Flow

1. **Start**: You receive the user's `question` and an empty `trajectory`.
2. **Plan**: In `next_thought`, decide what evidence you need.
3. **Search**: Call `diabetes_vector_search_tool` with a focused query.
4. **Observe**: Read the returned passages.
5. **Iterate**: If more evidence is needed, repeat steps 2-4.
6. **Finish**: When you believe you have enough information, call `finish`.
7. **Answer**: Output the final answer in the `answer` field.

### Domain-Specific Guidance

- **Gestational Diabetes Mellitus (GDM)**:
  - Key mechanisms: placental hormones (human placental lactogen, progesterone, estrogen) → insulin resistance; maternal adiposity → inflammatory adipokines (leptin, TNF-α, IL-6).
  - Long-term cardiovascular risk: endothelial dysfunction, dyslipidemia, pro-thrombotic state; persists even after postpartum weight adjustment.
  - Transition to type 2 diabetes: chronic inflammation (↑CRP, ↓adiponectin), β-cell dysfunction, insulin resistance.

- **Inflammation Markers**:
  - **CRP**: high-sensitivity C-reactive protein, marker of systemic inflammation.
  - **Adiponectin**: adipokine with anti-inflammatory and insulin-sensitizing effects; low levels are associated with insulin resistance.

- **Incretin-Based Therapies**:
  - **GLP-1 Receptor Agonists**: stimulate glucose-dependent insulin secretion, suppress glucagon, slow gastric emptying, promote weight loss, reduce appetite, have anti-inflammatory, cardiovascular, and renal benefits.
  - **DPP-4 Inhibitors**: prolong endogenous GLP-1 activity, modest glycemic effect, generally weight-neutral, may improve endothelial function.
  - **Adipokines**: influence insulin sensitivity and inflammation; incretin therapies can modulate adipokine profiles.

- **Interventions Targeting Inflammation**:
  - Lifestyle: weight loss, aerobic exercise, Mediterranean diet.
  - Pharmacologic: metformin, GLP-1 agonists, anti-inflammatory agents (e.g., IL-1β inhibitors).
  - Monitoring: serial CRP and adiponectin levels can guide therapy.

### Common Pitfalls to Avoid

- **Calling `None` as a tool name** – this causes a `KeyError`.
- **Skipping the `finish` tool** – the system will never know you are done.
- **Providing non-JSON arguments** – must be valid JSON.
- **Including extraneous text** – only the four required fields should appear.
- **Over-reliance on a single search** – if the first search returns low-quality passages, refine the query and search again

### Example Interaction Skeleton

next_thought: I need evidence on how placental hormones cause insulin resistance in GDM.
next_tool_name: diabetes_vector_search_tool
next_tool_args: {"query": "placental lactogen insulin resistance GDM", "k": 3}

# Observation returned

next_thought: The passages mention hPL and maternal adiposity; I need more on long-term cardiovascular risk.
next_tool_name: diabetes_vector_search_tool
next_tool_args: {"query": "GDM long-term cardiovascular risk postpartum weight", "k": 3}

# Observation returned

next_thought: I have sufficient evidence; I will finish.
next_tool_name: finish

next_tool_args: {}

answer: [final answer text]
"""
    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})
    trajectory = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Trajectory:', 'desc': '${trajectory}'})
    next_thought = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Thought:', 'desc': '${next_thought}'})
    next_tool_name = Field(annotation=Literal['diabetes_vector_search_tool', 'finish'] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Name:', 'desc': '${next_tool_name}'})
    next_tool_args = Field(annotation=dict[str, Any] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Args:', 'desc': '${next_tool_args}'})
))

Note that each time you run the agent with the same prompt, it might perform differently. We gained ~8% improvement within 10 minutes of optimization!!

One interesting point that I have seen in the optimization process, and as you see in the optimized prompt above, is that GEPA adds a lot of details about the dataset in the prompt. But based on a comment from the author of the GEPA paper in a Discord channel, GEPA might overfit and put a lot of details from the data into the prompt in the beginning, but as it goes forward, the prompt gets shorter and can generalize better.

Build the Multi-Agent System

Now, let’s define the other agent, optimize it, and then build the multi-agent system with the lead agent as well.

First, we need the other subagent. Let’s first load its dataset:

# Load the dataset
with open("docs/qa_pairs_copd.json", "r") as f:
    qa_copd_data = json.load(f)

# Convert to dspy.Example objects
dataset_copd = [dspy.Example(question=item["question"], answer=item["answer"]).with_inputs("question") for item in qa_copd_data]

# shuffle the dataset
random.shuffle(dataset_copd)

# Split the dataset as requested
trainset_copd = dataset_copd[:10]
devset_copd = dataset_copd[10:]

print(f"Loaded {len(dataset_copd)} examples.")
print(f"Train set size: {len(trainset_copd)}")
print(f"Dev set size: {len(devset_copd)}")

Then we define the agent itself:

class COPDAgent(dspy.Module):
    def __init__(self):
        self.lm = dspy.LM(
            "openrouter/openai/gpt-oss-20b",
            api_key=os.getenv("OPENROUTER_API_KEY"),
            temperature=0.3,
            max_tokens=64000,
            cache=False
        )
        dspy.configure(lm=self.lm)
        self.copd_agent = dspy.ReAct(ReActSignature, tools=[copd_vector_search_tool])

    def forward(self, question: str):
        return self.copd_agent(question=question)

copd_agent = COPDAgent()

And then we evaluate the agent:

evaluator_copd = Evaluate(devset=devset_copd, num_threads=32, display_progress=True, display_table=5, provide_traceback=True)

# Evaluate the baseline agent (the existing `react`)
print("Evaluating the baseline ReAct agent...")
copd_baseline_eval = evaluator_copd(copd_agent, metric=llm_metric_prediction)
copd_baseline_eval

EvaluationResult(score=89.44, results=<list of 9 results>)

We can again optimize this agent as well as the diabetes agent:

copd_agent.copd_agent.extract._compiled = True
copd_agent.copd_agent.react._compiled = False
teleprompter = GEPA(
    metric=llm_metric_prediction,
    max_full_evals=2,
    num_threads=32,
    track_stats=True,
    track_best_outputs=True,
    add_format_failure_as_feedback=True,
    reflection_lm=teacher_lm,
)

optimized_copd_agent = teleprompter.compile(student=copd_agent, trainset=trainset_copd, valset=devset_copd)

# Access the detailed results from your optimized agent
results = optimized_copd_agent.detailed_results

# Get all candidates and their validation scores
candidates = results.candidates

val_scores = results.val_aggregate_scores

# Find the best candidate by validation score
best_idx = results.best_idx  # This is automatically calculated
best_score = val_scores[best_idx]
best_candidate = results.best_candidate

print(f"Best candidate index: {best_idx}")
print(f"Best validation score: {best_score}")
print(f"Best candidate components: {best_candidate}")

Best candidate index: 2
Best validation score: 0.9422222222222223
Best candidate components: copd_agent.react = Predict(StringSignature(question, trajectory -> next_thought, next_tool_name, next_tool_args
    instructions='
json
  {
    "instruction":
      """
        You are a medical assistant tasked with answering a clinical question about a COPD patient. The user will provide a JSON object with a single key, \"question\", containing a detailed scenario. Your response must be structured in the following format:
1. **next_thought** – A comprehensive, evidence-based answer that:
   - Integrates micro-CT imaging, Parametric Response Mapping (PRM) data (specifically PRMfSAD and PRMemph), and the latest GOLD guidelines.
   - Explains the patient's prognosis given borderline spirometry but high PRMfSAD and minimal PRMemph.
   - Justifies a monitoring schedule (spirometry, imaging, symptom assessment, exacerbation history, biomarkers) and a treatment plan (pharmacologic therapy, lifestyle interventions, vaccinations, pulmonary rehabilitation, smoking cessation, patient education)
   - Uses clear, concise language suitable for a clinician, referencing key evidence and risk-stratification concepts (e.g., pre-COPD, early COPD, GOLD 0/1, imaging-based risk).
   - Provides actionable recommendations (e.g., frequency of follow-up, imaging intervals, medication choices, when to consider escalation or specialist referral).
2. **next_tool_name** – Always set to \"finish\".
3. **next_tool_args** – Always an empty JSON object `{}`.

Do not mention any other tool names or invoke external APIs. Do not include any explanatory text outside the three fields. The entire output must be a single JSON object with the three keys above.

Example output format:
```json
{
    "next_thought": "...",
    "next_tool_name": "finish",
    "next_tool_args": {}
}```
Ensure the answer is accurate, up-to-date (using GOLD 2023/2024 guidance), and reflects the integration of micro-CT, PRM, and GOLD data for prognosis, monitoring, and treatment planning."
}'
    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})
    trajectory = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Trajectory:', 'desc': '${trajectory}'})
    next_thought = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Thought:', 'desc': '${next_thought}'})
    next_tool_name = Field(annotation=Literal['copd_vector_search_tool', 'finish'] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Name:', 'desc': '${next_tool_name}'})
    next_tool_args = Field(annotation=dict[str, Any] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Args:', 'desc': '${next_tool_args}'})
))

And then save both agents:

optimized_diabetes_agent.save("dspy_program/optimized_react_diabets.json", save_program=False)
optimized_copd_agent.save("dspy_program/optimized_react_copd.json", save_program=False)

Now that we have two subagents and have optimized each, we can wrap them in two functions and give them as tools to another ReAct agent as our lead agent:

def ask_diabetes(question: str) -> str:
    """Call the Diabetes expert agent and return its answer text."""
    pred = optimized_diabetes_agent(question=question)
    return pred.answer

def ask_copd(question: str) -> str:
    """Call the COPD expert agent and return its answer text."""
    pred = optimized_copd_agent(question=question)
    return pred.answer

And now, let’s define our lead agent and provide the two subagents as tool:

# Lead ReAct agent that can call sub-agents as tools
class LeadReAct(dspy.Module):
    def __init__(self):
        self.lm = dspy.LM(
            "openrouter/openai/gpt-oss-20b",
            api_key=os.getenv("OPENROUTER_API_KEY"),
            temperature=0.3,
            max_tokens=64000,
            cache=False
        )
        dspy.configure(lm=self.lm)
        self.lead_react = dspy.ReAct(ReActSignature, tools=[ask_diabetes, ask_copd])

    def forward(self, question: str):
        return self.lead_react(question=question)

lead_react = LeadReAct()

lead_react.react = Predict(StringSignature(question, trajectory -> next_thought, next_tool_name, next_tool_args
    instructions="You are a helpful assistant. Answer user's question.\n\nYou are an Agent. In each episode, you will be given the fields `question` as input. And you can see your past trajectory so far.\nYour goal is to use one or more of the supplied tools to collect any necessary information for producing `answer`.\n\nTo do this, you will interleave next_thought, next_tool_name, and next_tool_args in each turn, and also when finishing the task.\nAfter each tool call, you receive a resulting observation, which gets appended to your trajectory.\n\nWhen writing next_thought, you may reason about the current situation and plan for future steps.\nWhen selecting the next_tool_name and its next_tool_args, the tool must be one of:\n\n(1) ask_diabetes, whose description is <desc>Call the Diabetes expert agent and return its answer text.</desc>. It takes arguments {'question': {'type': 'string'}}.\n(2) ask_copd, whose description is <desc>Call the COPD expert agent and return its answer text.</desc>. It takes arguments {'question': {'type': 'string'}}.\n(3) finish, whose description is <desc>Marks the task as complete. That is, signals that all information for producing the outputs, i.e. `answer`, are now available to be extracted.</desc>. It takes arguments {}.\nWhen providing `next_tool_args`, the value inside the field must be in JSON format"
    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})
    trajectory = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Trajectory:', 'desc': '${trajectory}'})
    next_thought = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Thought:', 'desc': '${next_thought}'})
    next_tool_name = Field(annotation=Literal['ask_diabetes', 'ask_copd', 'finish'] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Name:', 'desc': '${next_tool_name}'})
    next_tool_args = Field(annotation=dict[str, Any] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Args:', 'desc': '${next_tool_args}'})
))
lead_react.extract.predict = Predict(StringSignature(question, trajectory -> reasoning, answer
    instructions="You are a helpful assistant. Answer user's question."
    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})
    trajectory = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Trajectory:', 'desc': '${trajectory}'})
    reasoning = Field(annotation=str required=True json_schema_extra={'prefix': "Reasoning: Let's think step by step in order to", 'desc': '${reasoning}', '__dspy_field_type': 'output'})
    answer = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Answer:', 'desc': '${answer}'})
))

We see that the lead agent has again two modules like the previous agents, and the subagents are used as tools.

To make it a bit more complex, I’ve created a new dataset that combines two previous datasets, incorporating mixed and changed questions that require the lead agent to utilize both subagents. Let’s load this dataset as well:

# Load the dataset
with open("docs/qa_pairs_joint.json", "r") as f:
    qa_joint_data = json.load(f)

# Convert to dspy.Example objects
joint_dataset = [dspy.Example(question=item["question"], answer=item["answer"]).with_inputs("question") for item in qa_joint_data]

# shuffle the dataset
random.shuffle(joint_dataset)

# Split the dataset as requested
trainset_joint = joint_dataset[:train_size]
devset_joint = joint_dataset[train_size:]

print(f"Loaded {len(joint_dataset)} examples.")
print(f"Train set size: {len(trainset_joint)}")
print(f"Dev set size: {len(devset_joint)}")

Note that this optimization considers two subagents as black boxes and doesn’t optimize their prompts. To optimize all prompts, you need to build your module in a way that calls those subagents directly within the module. I think!

Let’s now evaluate our non-optimized lead agent:

# Baseline evaluation of the lead agent on the joint dev set
evaluator_joint = Evaluate(devset=devset_joint, num_threads=32, display_progress=True, display_table=5, provide_traceback=True)
print("Evaluating baseline Lead ReAct (agents-as-tools) on joint dev set...")
baseline_lead_eval = evaluator_joint(lead_react, metric=llm_metric_prediction)
baseline_lead_eval

EvaluationResult(score=88.79, results=<list of 33 results>)

And we can use the same optimizer and optimize the lead agent (I do three full evals here — you can increase it and do more optimizations):

lead_react.lead_react.extract._compiled = True
lead_react.lead_react.react._compiled = False

teleprompter_joint = GEPA(
    metric=llm_metric_prediction,
    max_full_evals=3,
    num_threads=32,
    track_stats=True,
    track_best_outputs=True,
    add_format_failure_as_feedback=True,
    reflection_lm=teacher_lm,
)

optimized_lead_react = teleprompter_joint.compile(student=lead_react, trainset=trainset_joint, valset=devset_joint)

# Access the detailed results from your optimized agent
results = optimized_lead_react.detailed_results

# Get all candidates and their validation scores
candidates = results.candidates

val_scores = results.val_aggregate_scores

# Find the best candidate by validation score
best_idx = results.best_idx  # This is automatically calculated
best_score = val_scores[best_idx]
best_candidate = results.best_candidate

print(f"Best candidate index: {best_idx}")
print(f"Best validation score: {best_score}")
print(f"Best candidate components: {best_candidate}")

Best candidate index: 1
Best validation score: 0.9242424242424242
Best candidate components: lead_react.react = Predict(StringSignature(question, trajectory -> next_thought, next_tool_name, next_tool_args
    instructions='markdown\n## Instruction for the Assistant\n\nYou are a **helpful agent** that answers medical questions about COPD and diabetes.  \nYour job is to produce a *structured response* that interleaves the following\nelements in the exact order:\n\n1. `next_thought` – a short paragraph describing your current reasoning and\n   the next step you plan to take.\n2. `next_tool_name` – the name of the tool you will call next.  \n   Allowed tool names are:\n   * `ask_diabetes`\n   * `ask_copd`\n   * `finish`\n3. `next_tool_args` – a JSON object containing the arguments for the chosen\n   tool.  The JSON must be syntactically valid.  \n   * For `ask_diabetes` and `ask_copd` the argument is a single field\n     `question` of type string.\n   * For `finish` the argument object is empty `{}`.\n\nAfter you call a tool, you will receive an `observation` that is appended to\nyour trajectory.  Use that observation to update your next thought and decide\nwhether another tool call is needed or if you should finish.\n\n**Important constraints**\n\n- **Do not output anything else** besides the three fields above.  \n  No additional text, comments, or tool calls are allowed.\n- **Do not use any other tools** (e.g., `browser`, `python`).  \n  Only the three tools listed above are permitted.\n- The `next_tool_args` must be valid JSON; do not embed it in quotes or\n  add stray characters.\n- When you are ready to produce the final answer, call the `finish` tool\n  with an empty argument object `{}`.\n\n**Domain knowledge you should incorporate**\n\n1. **COPD inpatient protocol for acute exacerbations**  \n   * Steroid course: 5 days of systemic corticosteroids (oral prednisone 40-60 mg/d or IV methylprednisolone 40-80 mg/d).  \n   * Oxygen target: SpO2 88-92 % (or 90-94 % for chronic hypercapnia).  \n   * Bronchodilators: Nebulized SABA + ipratropium; add LAMA/LABA if not on maintenance.  \n   * Antibiotics: Empiric if sputum purulence or fever; common regimens include amoxicillin-clavulanate or doxycycline.  \n   * Monitoring: Vitals & SpO2 q4 h, daily ABG if PaCO2 > 45 mmHg, labs (CBC, electrolytes, CRP).  \n   * Discharge planning: Stable vitals, SpO2 >= 88 % on <= 4 L/min O2, no IV antibiotics, provide written action plan, inhaler technique review, smoking cessation counseling, referral to pulmonary rehab.\n\n2. **Diabetes management in steroid-treated patients**  \n   * Immediate hyperglycemia management: Basal-bolus insulin regimen (e.g., glargine + rapid-acting insulin at meals).  \n   * Monitoring: SMBG 4-6x daily, including a dedicated post-steroid check 2-3 h after dose.  \n   * Adjustment: Increase basal dose or add a short-acting insulin before steroid dose if post-steroid glucose >200 mg/dL.  \n   * Discharge follow-up: Medication reconciliation, 2-4 week outpatient visit, HbA1c reassessment, education on steroid-induced hyperglycemia.\n\n3. **Integration of COPD and diabetes care**  \n   * Coordinate inhaled therapy with glucose control (avoid hypoglycemia when using beta-agonists).  \n   * Use multidisciplinary teams (pulmonology, endocrinology, pharmacy) for medication reconciliation.  \n   * Emphasize that GLP-1 analogues or DPP-4 inhibitors have no proven direct benefit for COPD symptoms or exacerbation reduction; they should be used for metabolic indications only.\n\n**Example workflow**\n\n```\nnext_thought\nI need the COPD protocol first, so I will ask the COPD expert.\n\nnext_tool_name\nask_copd\n\nnext_tool_args\n{"question":"Provide an inpatient protocol for acute COPD exacerbations, including steroid duration."}\n\n(observation received)\n\nnext_thought\nNow I have the COPD protocol. I still need diabetes guidance on steroid-induced hyperglycemia and discharge follow-up.\n\nnext_tool_name\nask_diabetes\n\nnext_tool_args\n{"question":"What is the recommended immediate management of hyperglycemia in an inpatient with type 2 diabetes who has been on oral steroids, and what discharge diabetes follow-up should be planned?"}\n\n(observation received)\n\nnext_thought\nAll information is collected. I will finish.\n\nnext_tool_name\nfinish\n\nnext_tool_args\n{}'
    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})
    trajectory = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Trajectory:', 'desc': '${trajectory}'})
    next_thought = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Thought:', 'desc': '${next_thought}'})
    next_tool_name = Field(annotation=Literal['ask_diabetes', 'ask_copd', 'finish'] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Name:', 'desc': '${next_tool_name}'})
    next_tool_args = Field(annotation=dict[str, Any] required=True json_schema_extra={'__dspy_field_type': 'output', 'prefix': 'Next Tool Args:', 'desc': '${next_tool_args}'})
))

We see that we got 4% improvement again. That’s amazing!

Finally, let’s also save the lead agent:

optimized_lead_react.save("dspy_program/optimized_lead_react.json", save_program=False)

That’s it for this blog post. Hope it’s been helpful.

Conclusion

In this post, we explored how to build, optimize, and orchestrate multi-agent RAG systems using DSPy and GEPA. Starting from the fundamentals of RAG and vector databases, we walked through constructing specialized subagents for diabetes and COPD, each empowered with their own retrieval tools and datasets. We then delved into prompt optimization with GEPA, observing firsthand how iterative feedback and reflection can substantially improve agent performance. Finally, we brought everything together by creating a lead agent capable of coordinating these subagents and also optimizing the lead agent’s prompt using GEPA, demonstrating the flexibility and power of modular, agentic architectures in complex domains like medicine.