Mixture of Memory Experts: Lamini Memory Tuning

Lamini’s Memory Tuning description introduces a Mixture of Memory Experts (MoME) for storing domain-specific facts. In this article, a hallucination means an unsupported or incorrect generated claim. Lamini reports reducing hallucinations from 50% to 5% in its own factual-recall evaluation. That figure describes the cited evaluation and should not be treated as a general rate for every model or dataset.

The architecture

The design keeps a Transformer backbone and adds a large collection of auxiliary memory experts. The backbone remains frozen while the memory experts are trained on the facts to be recalled. The memory path is intended to retrieve precise facts without updating every backbone parameter. Saying that the retrieval path does not rely on a conventional Transformer does not mean that the complete model has no Transformer backbone.

The reported 50% to 5% comparison is a claim about the fraction of unsupported or incorrect answers in Lamini’s evaluation. To compare it with another system, the dataset, answer judge, sampling procedure, and baseline would need to match. Lamini’s current documentation also describes factual Q&A and text-to-SQL evaluations, which are distinct tasks with their own measures.

Training memory experts

The training procedure described by Lamini is:

1. Select a small subset of experts for a question, such as 32 experts from a much larger pool.
2. Freeze the backbone and the routing cross-attention.
3. Update the selected memory experts until the fact has sufficiently low training loss.

This is still training. The phrase “without retraining” is accurate only for the frozen backbone; the auxiliary experts are trained on the new data.

The Lamini documentation on Memory Tuning describes the training interface.

The motivation is that memorising a small fact set through all of a large model can be expensive. Lamini gives an illustrative comparison in which one epoch of a large model is expensive and reaching near-zero loss on selected facts would require many epochs. A 100-epoch calculation is a cost illustration, not a measurement of every memory-tuning run.

The useful question for a deployment is whether the memory-tuned model answers the target fact set correctly while retaining the backbone’s broader behavior. That requires a held-out evaluation with an explicit definition of factual correctness.