Expert Gate: Lifelong Learning with a Network of Experts

Expert Gate: Lifelong Learning with a Network of Experts, by Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars, adds a specialist expert as each new task arrives. The system does not require all earlier training data to remain available. At test time it loads the expert selected for the input instead of keeping every expert in memory at once.

A gate made from autoencoders

Each expert has a one-layer undercomplete autoencoder trained on that task’s data. The autoencoder learns a compact representation and reconstructs an input. At test time, Expert Gate computes the reconstruction error for each task autoencoder and selects the expert with the lowest error. This is the paper’s selection heuristic. A low reconstruction error is evidence that the input resembles the autoencoder’s training distribution; it is not proof that the selected expert understands the input or will classify it correctly.

The same autoencoders estimate task relatedness during training. The authors use that estimate to choose a prior model and then either fine-tune it or use Learning without Forgetting (LwF). If tasks are unrelated, transferring old outputs can hurt, so the paper uses a relatedness threshold to select between the two procedures.

Why specialist experts help

Training one model on every task can introduce negative transfer and requires access to the earlier datasets when a new task arrives. A network of experts keeps task-specific models and learns each new expert sequentially. The gate addresses the memory cost of that design by loading the selected expert for a prediction. These benefits are scoped to the paper’s sequential-learning setup and do not remove the need to store the expert parameters or train the gate.

Reported evaluation

On the paper’s sequential three-task image-classification experiment using Scenes, Birds, and Flowers, Expert Gate reports 63.5%, 57.6%, and 84.8% accuracy, with a 68.6% average. The comparison is against a jointly trained model and multiple fine-tuned or LwF experts; some comparison rows assume access to all previous data or an oracle gate.

In a six-task experiment, the reported Expert Gate accuracies are 60.4% on Scenes, 57.0% on Birds, 84.4% on Flowers, 80.3% on Cars, 72.2% on Aircrafts, and 49.5% on Actions, for a 67.3% average. The paper also evaluates video prediction for autonomous-driving data. Its average pixel L1 distance is 23.4 for Expert Gate and 23.8 for the jointly trained comparison, where lower is better.

The results include failure cases. Images that combine visual cues from different tasks can confuse the gate, and transfer can degrade when the relatedness estimate is wrong. The method’s claim is a way to manage sequential specialists and route inputs in the evaluated settings, not a guarantee that reconstruction error identifies the correct expert for every domain.