Bilevel Optimisation: Why Meta-Learning Is an Optimisation Problem Inside an Optimisation Problem

Most optimisation problems have one level: pick parameters, minimise a loss, stop. Bilevel optimisation has two. An outer problem chooses some variables, and an inner problem then optimises the rest, conditional on that choice. The outer objective can only be evaluated after the inner problem has been solved, at least approximately. That nesting shows up far more often than the name suggests. Hyperparameter tuning is bilevel: the outer variables are hyperparameters, the inner variables are model weights trained with those hyperparameters. Meta-learning is bilevel: the outer variables are a shared initialisation or learning rule, and the inner problem is fast adaptation to a new task. Data reweighting, architecture search, and some approaches to cleaning noisy labels are bilevel too. In each case, treating the two levels as if they were independent is not just sloppy, it gives the wrong gradient. This article explains the bilevel formulation, why its gradient is harder than it looks, the two main ways to compute it, and where the whole approach breaks down in practice. The formulation A bilevel problem has an outer objective and an inner objective. Let the outer variables be denoted by a vector, and the inner variables by another vector. The inner problem finds the best inner variables for a fixed outer choice, and the outer problem scores that result. λmin​F(λ,w(λ))subject tow(λ)=argwmin​f(λ,w) Here F is the outer objective, f is the inner objective, λ collects the outer variables, and w(λ) is the inner solution as a function of λ . The notation already contains the difficulty: w(λ) is not a free variable. It moves when λ moves, and the outer gradient must account for that movement. A concrete example helps. In hyperparameter optimisation, λ might be a regularisation strength or a set of per-example weights, f is the training loss, and F is the validation loss evaluated at the trained weights. In model-agnostic meta-learning, λ is a shared initialisation, the inner problem is a few gradient steps on a new task, and F is the loss after adaptation, averaged over tasks. The distinction between the two objectives matters. If the outer and inner objectives were the same, and both used the same data, the bilevel structure would collapse: the outer problem would just be ordinary training. Bilevel optimisation is interesting exactly when they differ, in their data, their loss, or both. Validation versus training is the classic split, and task-generalisation versus task-adaptation is the meta-learning split. Why the gradient is not obvious Suppose we want to improve λ by gradient descent on the outer objective. The total derivative of F with respect to λ has two parts: the direct effect of λ on F , and the indirect effect through the inner solution w∗(λ) . dλdF​=∂λ∂F​+∂w∂F​dλdw​ The first term is easy. The second term is the problem. It requires the Jacobian dw/dλ , which describes how the inner optimum shifts when the outer variables change. Ignoring it, which is equivalent to treating w(λ) as a constant, is a common shortcut, and it is sometimes defensible as an approximation. But it is not the gradient of the bilevel objective, and in meta-learning it throws away exactly the signal that makes the initialisation good for adaptation rather than good on its own. There are two principled ways to get dw/dλ . They differ in whether they use the optimality condition of the inner problem, or the actual optimisation trajectory that produced w(λ) . Method 1: implicit differentiation If the inner problem is solved to a stationary point, its gradient with respect to w is zero at the solution: ∇w​f(λ,w∗(λ))=0 Differentiating this identity with respect to λ and applying the chain rule gives a linear system for the Jacobian: ∇wλ2​f+∇ww2​fdλdw∗​=0 Rearranging, and assuming the inner Hessian is invertible: dλdw∗​=−(∇ww2​f)−1∇wλ2​f This is the implicit function theorem applied to optimisation. Its appeal is that it does not care how the inner problem was solved. Any solver that reaches a stationary point will do, and the gradient depends only on second derivatives at the solution, not on the path taken to get there. Its cost is also clear from the formula: it needs an inverse Hessian, or more precisely, the solution of a linear system involving the Hessian. For a large model, forming that Hessian is impossible, but solving the system approximately is often feasible. Hessian-vector products can be computed without the full Hessian, and iterative solvers such as conjugate gradient, or a truncated Neumann series, give a usable approximation. The quality of the outer gradient then depends on two approximations at once: how well the inner problem was solved, and how well the linear system was solved. Implicit differentiation also has a quiet assumption worth stating: it assumes the inner solution is a smooth function of λ . If the inner problem has multiple minima and the solver jumps between them as λ changes, w∗(λ) is not smooth, and the formula describes only the local branch the solver is on. In non-convex deep learning, that is the normal situation, not an edge case. Method 2: unrolled differentiation The second approach does not assume the inner problem is solved exactly. Instead, it treats the inner optimiser itself as a differentiable computation. If the inner solution is produced by a fixed number of gradient steps, each step is a differentiable function of λ , and backpropagation through those steps gives the outer gradient. For one inner gradient step with step size α , starting from w0​ : w1​=w0​−α∇w​f(λ,w0​) Differentiating with respect to λ already introduces a second derivative, because the gradient ∇w​f itself depends on λ . With several steps, the chain grows, and the computation graph holds every intermediate iterate. This is exactly what model-agnostic meta-learning does when it backpropagates through inner adaptation steps, and it is why second-order meta-learning is more expensive than its first-order approximation, which drops the second-derivative terms. Unrolled differentiation has a different set of trade-offs from the implicit approach. It is faithful to what the inner optimiser actually did, including early stopping and a finite step count, which is often what you care about in meta-learning: the outer objective is performance after a few adaptation steps, not performance at the exact inner optimum. But memory grows with the number of unrolled steps, the gradient can vanish or explode through a long chain of updates, and the result depends on optimiser details such as momentum state and learning rate schedule, not just on the inner objective. A minimal sketch makes the mechanics concrete. The code below unrolls two inner steps on a toy quadratic inner problem and differentiates the outer loss with respect to an outer scalar, using automatic differentiation. It is a toy, but the structure, an inner loop inside an outer loss, is the same structure that appears in real meta-learning code. import torch # Outer variable: a scalar that shifts the inner optimum. lam = torch.tensor(0.0, requires_grad=True) # Inner variable, starting from zero. w = torch.zeros(1) # Inner objective: a quadratic whose minimum moves with lam. def inner_loss(w, lam): return (w - lam) ** 2 + 0.1 * w ** 2 # Unrolled inner optimisation: two gradient steps. for _ in range(2): g = torch.autograd.grad(inner_loss(w, lam), w, create_graph=True)[0] w = w - 0.5 * g # Outer objective: penalise a large inner solution. outer_loss = (w - 1.0) ** 2 outer_loss.backward() print("outer loss:", outer_loss.item(), "d outer / d lam:", lam.grad.item()) Note the create_graph=True flag. Without it, the inner gradient is treated as a constant, and the outer gradient loses the indirect term entirely. That single flag is the difference between unrolled differentiation and the first-order shortcut, and forgetting it is one of the most common silent bugs in bilevel code. Which method to choose The choice is rarely about mathematical elegance. It is about which approximation you can afford and which quantity you actually want to optimise. Use implicit differentiation when the inner problem is solved close to convergence, when the number of outer variables is modest, and when you do not want the outer gradient to depend on the inner optimiser's trajectory. Hyperparameter optimisation with a well-trained inner model is a natural fit, provided you can solve the Hessian system accurately enough. Use unrolled differentiation when the inner problem is deliberately truncated, as in few-step meta-learning, or when the inner solver is part of the model you are learning, for example a learned optimiser. Then the trajectory is the point, and differentiating through it is correct even though it is expensive. In both cases, the first-order shortcut, dropping the indirect term, is worth trying as a baseline. It is cheap, it is often surprisingly competitive, and it tells you whether the bilevel machinery is earning its cost. If the first-order version performs as well as the full version, the indirect term was not carrying much signal for your problem, and you have saved yourself a Hessian system or an unrolled graph. If it performs much worse, that gap is evidence that the outer variables genuinely need to anticipate how the inner solution will move. Limitations and common pitfalls Bilevel optimisation looks clean on paper and is fragile in practice. The main failure modes are worth naming explicitly. First, the inner problem is almost never solved exactly. Both methods degrade when it is not, but they degrade differently. Implicit differentiation assumes a stationary point that does not exist yet, so its gradient is biased by the residual inner gradient. Unrolled differentiation is exact for the truncated procedure, but the truncated procedure may be a poor proxy for the inner optimum you actually care about. Either way, "solved the inner problem" is a modelling choice, not a fact, and it should be reported as one. Second, second derivatives are expensive and noisy. Hessian-vector products, truncated Neumann series, and short unrolls are all approximations, and their errors compound with the usual stochastic gradient noise. A bilevel gradient that is noisy and biased can still be useful, but it needs smaller outer step sizes and more careful tuning than a first-order gradient, and it will punish sloppy inner learning rates more harshly. Third, the outer objective can overfit its own split. In hyperparameter optimisation, tuning many outer variables against a single validation set is just overfitting with extra steps. The bilevel formulation does not protect against that; it formalises it. The same warning applies to meta-learning: an initialisation tuned on a narrow family of training tasks may adapt beautifully to those tasks and fail on genuinely new ones. The outer split, validation data or held-out tasks, must be treated as a scarce resource, not as a free lunch. Fourth, non-convexity breaks the clean theory. Multiple inner minima, flat regions, and solver-dependent solutions mean the map λ↦w∗(λ) may not be a function in any useful sense. Practical bilevel methods work anyway, because they optimise a specific branch or a specific trajectory, but the theory that justifies the gradient assumes more regularity than deep learning provides. Treat bilevel gradients as useful directions, not as exact derivatives of a well-defined function, and validate them empirically. Finally, there is a simpler pitfall: accidentally optimising the wrong level. If the outer variables are allowed to influence the inner loss in a way that lets the inner problem cheat, for example by scaling the loss down rather than improving the solution, the outer objective will exploit it. Constraining the outer variables, regularising them, or parametrising them so that cheating is impossible is part of the problem design, not an afterthought. Takeaway Bilevel optimisation is the right language whenever one optimisation problem sits inside another: hyperparameters inside training, initialisations inside adaptation, weights inside architecture search. The core insight is a single term, the Jacobian of the inner solution with respect to the outer variables, and the two main methods are just two ways of estimating it: implicitly, from the optimality condition, or explicitly, by differentiating through the inner optimiser. Use it when the nesting is real and the indirect effect matters. Test the first-order shortcut first, keep the inner solve honest, and remember that a bilevel gradient is only as trustworthy as the inner solution and the outer split behind it. Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial