← Publications

Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen, Serge Hoogendoorn, Elvin Isufi

NeurIPS 2026 · September 20, 2026

📄 Paper 💻 Code

Why expressive policies, and why they are hard to train online

In continuous control, the standard Gaussian policy has a single peak. Many tasks, however, have several equally good ways to act in the same state: open the cabinet first or turn on the burner first; go left around the obstacle or go right. A single-peaked policy facing such a choice either commits to one option arbitrarily or averages between them and ends up doing neither well.

SAC Gaussian

DPMD Diffusion

FMER Flow matching

Generative policies fix this by turning random noise into actions, conditioned on the state. Diffusion policies first became popular for learning from fixed datasets. Using them in online RL, where the policy has to improve itself from rewards, runs into two problems:

1. You can’t measure their randomness. Methods like SAC explore by keeping the policy’s entropy, its randomness, high. A diffusion model doesn’t tell you its entropy, so existing methods fall back on rough approximations that bias exploration.
2. Training is slow and unstable. Improving the policy usually means backpropagating through every denoising step, which is expensive and noisy.

Flow matching generates actions by following a smooth, deterministic path from noise to action. A learned velocity field \(v_\theta\) moves a noise sample \(x_0\) step by step until it becomes an action:

Because this path is deterministic, the probability of each action can be computed. Existing flow-based RL methods still backpropagate through that path, though, and approximate the entropy to keep the cost down. FMER avoids both: training never simulates the path, and the entropy is exact.

Ingredient 1: policy mirror descent as weighted flow matching

Policy mirror descent improves a policy in small steps: each new policy should earn more reward than the current one while staying close to it. The ideal next policy has a simple form:

In words: take the current policy \(\pi_k\) and boost the actions that are better than average (high advantage \(A\)). The temperature \(\tau\) controls how big the step is, and \(Z(s)\) just normalizes the result.

Fitting a flow policy to this target directly would mean differentiating through the whole generation path. Our main theoretical result (Theorem 1) shows that this isn’t necessary. Sample some actions from the current policy, give each one the weight \(w(s,a) = \exp(A(s,a)/\tau)/Z(s)\), and train with the ordinary flow matching loss, weighted by those numbers:

The velocity field is asked to point along the straight line from the noise \(x_0\) to the action \(x_1\), and high-weight actions count more. The best fit of this weighted loss is exactly the mirror descent target \(\pi^*\). Because the loss is a plain regression, there is no path to simulate and nothing to backpropagate through.

Weigh every candidate, not just the best one. In practice, FMER samples several candidate actions per state and lets the critic score them. Learning from all of them, in proportion to their weights, works better than copying only the top-scoring one: the top candidate may sit on a local peak, or the critic may simply overrate it.

Step size, tuned automatically. If \(\tau\) is too large, the policy barely moves; if it is too small, a single candidate takes over. FMER measures how evenly the weights are spread with the effective sample size, \(\text{ESS} = 1/\sum_l \hat{w}_l^2\), which runs from 1 (one candidate dominates) to \(M\) (all \(M\) candidates count equally). It adjusts \(\tau\) to keep ESS at a target value, so it needs no per-environment tuning.

Ingredient 2: an entropy you can compute exactly

Because the flow is deterministic, its entropy can be computed exactly by tracking how the flow stretches or squeezes space as it carries noise to actions. That stretching is the divergence of the velocity field, integrated along the path:

There is a catch that is easy to miss. Robots need bounded actions, so FMER squashes its output into the valid range with \(a = \tanh(x_1)\). That squashing distorts the probabilities, and what we care about is the randomness of the actual action, not of the value before squashing. Proposition 1 gives the corrected divergence in action space:

The second term pushes back when the policy piles probability onto the action limits. Like the main loss, the corrected entropy is evaluated along the same straight paths, so the flow never has to be simulated.

FMER then keeps the entropy above a target level, with a weight \(\alpha\) that is tuned automatically, much like SAC. The full policy objective is

The training loop

Each training step does four things:

• Act. Generate a handful of candidate actions, execute the one the critic likes best, and store the result.
• Critic. Update the estimates of how good each action is.
• Policy. Weight the candidates by the critic’s scores and update the policy with the weighted flow matching loss plus the entropy term.
• Tune. Adjust \(\alpha\) to track the target entropy and \(\tau\) to track the target ESS.

Results

We compared FMER with eight baselines that use Gaussian, diffusion or flow policies. Every method was trained for one million steps, over 5 random seeds.

FrankaKitchen (sparse reward). A robot arm in a simulated kitchen has 280 steps and earns a reward only when it finishes a sub-task, such as moving the kettle. With rewards this rare, exploration is the main challenge. We test sets of 1, 2, 4 and 7 tasks. FMER is best on 1, 4 and 7 tasks and level with the best baseline on 2 (48% vs. 50%). The gap widens as tasks are added: with 7 tasks, FMER completes 40% vs. 23% for the next best method.

The baselines fail in telling ways. SAC falls apart once there are several task orders to choose from. DPMD locks onto its early successes and can’t move on. And the methods with approximate entropy (QVPO, SAC-Flow) explore poorly exactly where exploration matters most.

MuJoCo (dense reward). These standard locomotion tasks check that FMER’s extra machinery doesn’t hurt on easier problems. FMER scores highest on HalfCheetah and Walker2d and second on Humanoid. It falls behind on Hopper, a simple task where its extra exploration goes on longer than needed.

What matters. Removing FMER’s parts one at a time shows that each one earns its place:

• Entropy constraint. Without it, the policy turns almost deterministic early in training and completes 0% of the 7-task kitchen.
• Weights. With equal weights, the policy just imitates its own samples, and performance drops.
• Squashing correction. Without it, every MuJoCo score goes down, HalfCheetah from 12.3k to 7.7k.

FMER also trains faster than the diffusion baselines, though slower than the lightest flow-based method.

Limitations

• Speed. FMER is slower than flow methods that generate actions in one or a few steps, because computing the exact entropy takes extra work.
• Simulation only. All experiments are in simulation. Whether the gains carry over to real robots is still an open question.

Takeaway

Choosing between diffusion and flow matching is not just about how fast you can sample. For online RL, flow matching’s deterministic path is what makes a principled algorithm possible: the policy update becomes a weighted regression, and the policy’s randomness becomes something you can measure and control. FMER puts both pieces together, and the gains show up exactly where exploration is hardest.

Citation

@inproceedings{gao2026fmer,
  title     = {Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints},
  author    = {Gao, Ting and Orfanoudakis, Stavros and Lin, Nan and Daamen, Winnie and Hoogendoorn, Serge and Isufi, Elvin},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}