FUMA

FUMA

FUture MAtching: Improving Your Vision-Language-Action Models
via Differentiable Latent World Model

Dohyun Lee Kinam Kim Jaegul Choo Minho Park

KAIST AI

TL;DR FUMA adds one auxiliary loss to VLA fine-tuning. A frozen latent world model predicts the future under the policy's action and under the demonstration action, and the policy is trained to make the two predictions match. Errors that change the future more are penalized more; the policy architecture and inference stay unchanged.

Push-T example: a red and a blue action lie the same distance from the demonstration action and get the same flow-matching loss (2.00 each), but the red action produces a 28.1 times larger next-state error (0.116 vs 0.0041). Right: FUMA raises the average success of four VLAs from 57.6 to 63.2 on RoboCasa and from 69.7 to 75.0 on LIBERO-Plus.
Equal action errors, unequal consequences. The red and blue actions lie 20 px from the demonstration and share one flow-matching loss (b), yet only the red one changes the T's outcome: a 28.1× larger next-state error (c). (d) FUMA raises the average success of four VLAs on both benchmarks.

Abstract

Recent vision-language-action (VLA) models commonly use flow matching for continuous action generation, but its squared-error objective does not distinguish action errors with different physical consequences. We introduce Future Matching (FUMA), an auxiliary objective that supervises actions through predicted future outcomes. Given the same observation, a separately trained, frozen latent world model predicts future features under the policy's one-step clean-action estimate and the demonstration action. FUMA matches these predictions, propagating gradients through the world model's action input to the policy. Locally, it penalizes deviations more strongly when they cause larger changes in predicted future features. FUMA complements each backbone's original objective without changing policy architecture or inference. Across four pretrained VLA backbones, FUMA improves average success rates on RoboCasa and LIBERO-Plus by 5.6 and 5.3 percentage points, respectively, and outperforms compared auxiliary objectives under the same fine-tuning protocol. Directional analyses show reduced errors along future-sensitive directions, which perturbation experiments associate with greater effects on grasp success.

The same action error can matter very differently

Flow-matching VLAs such as π0.5 and GR00T learn a velocity field by minimizing a squared error to demonstration-derived targets. At a fixed flow time, that loss is an isotropic penalty on the error of the one-step action estimate: every direction of error costs the same. In manipulation it does not. Whether a deviation matters depends on the current state and on which way the action is off. In the Push-T example above, two actions with identical action-space errors lead to next-state errors that differ by 28.1×.

Methods such as FLARE and VLA-JEPA bring future prediction into the policy's representations. FUMA instead evaluates the control actions the policy predicts, through the futures they induce. Comparing outcomes in the environment would need repeated rollouts and gives no gradient, so FUMA uses a differentiable world model that stays frozen while the policy trains.

Method

FUMA keeps each backbone's native fine-tuning loss and adds one term that compares predicted futures. The world model is an adapted V-JEPA 2-AC: an observation encoder EE with z=E(o)z=E(o) and an action-conditioned predictor Pϕ(z,s,a)P_\phi(z,s,a), fine-tuned on each benchmark's demonstrations and then frozen.

Four training diagrams. (a) Flow matching compares the predicted velocity with a minus epsilon. (b) FLARE and VLA-JEPA align policy-predicted features with encoded future observations. (c) FUMA with GT Latent compares the world model's prediction under the policy action with the encoded future observation. (d) FUMA with GT Action, ours, compares the world model's predictions under the policy action and under the demonstration action, with stop-gradient on the latter.
(a) Flow matching compares the predicted velocity with a−ϵa-\epsilon. (b) Future-representation supervision compares policy-predicted features with encoded future observations. (c) The GT Latent ablation compares z′πz'^{\pi} with the encoded future z′=E(o′)z'=E(o'). (d) FUMA (GT Action, ours) compares z′πz'^{\pi} with the prediction z′refz'^{\mathrm{ref}} under the demonstration action. sg denotes stop-gradient.
  1. One-step action estimate

    At the sampled flow time, turn the policy's predicted velocity into a clean action, without unrolling the sampler. Policies that predict clean actions directly use their own estimate.

    a^θ=at+(1−t) vθ(at,t; o,s,ℓ)\hat a_\theta = a_t + (1-t)\,v_\theta(a_t, t;\, o, s, \ell)
  2. Two futures from the same world model

    Run the frozen predictor on the same current representation and state, once with the policy's action and once with the demonstration action:

    z′π=Pϕ(z,s,a^θ)z′ref=Pϕ(z,s,a)\begin{aligned} z'^{\pi} &= P_\phi(z, s, \hat a_\theta) \\ z'^{\mathrm{ref}} &= P_\phi(z, s, a) \end{aligned}
  3. Match them

    Penalize the gap, with no gradient through the reference. Because both sides come from the same model and context, the loss and its gradient vanish when a^θ=a\hat a_\theta = a, even if the world model is imperfect.

    LFUMA=1M∥z′π−sg⁡ ⁣[z′ref]∥22\mathcal{L}_{\mathrm{FUMA}} = \frac{1}{M}\left\| z'^{\pi} - \operatorname{sg}\!\left[z'^{\mathrm{ref}}\right] \right\|_2^2
    L=LVanilla+λ LFUMA\mathcal{L} = \mathcal{L}_{\mathrm{Vanilla}} + \lambda\,\mathcal{L}_{\mathrm{FUMA}}

    Gradients reach the policy through the world model's action input. The world model is used only during training.

Why it helps: a context-dependent geometry over action errors. With δ=a^θ−a\delta = \hat a_\theta - a, flow matching is LFM=∥δ∥22/(1−t)2\mathcal{L}_{\mathrm{FM}} = \|\delta\|_2^2/(1-t)^2, the same for every direction. A first-order expansion of the world model, with Jacobian Jo,s,a=∂fϕ/∂aJ_{o,s,a} = \partial f_\phi / \partial a, gives
LFUMA≈δ⊤Go,s,a δ=∑kγk(qk⊤δ)2\mathcal{L}_{\mathrm{FUMA}} \approx \delta^\top G_{o,s,a}\,\delta = \sum_{k} \gamma_k \left(q_k^\top \delta\right)^2
Go,s,a=1MJo,s,a⊤Jo,s,aG_{o,s,a} = \tfrac{1}{M} J_{o,s,a}^\top J_{o,s,a}
Errors along eigenvectors qkq_k with large eigenvalues γk\gamma_k, the directions the predicted future is most sensitive to, are penalized most, and those directions change with the observation, state, and demonstrated action. The loss produces this weighting on its own; the Jacobian is never computed during training.

Four VLAs on RoboCasa and LIBERO-Plus

RoboCasa: 24 atomic kitchen tasks, 50 rollouts each (1,200 trials). LIBERO-Plus: trained on the 40 original LIBERO tasks and evaluated on 10,030 perturbed instances. Every backbone starts from the same 1k-step warm-up checkpoint and trains for 10k (RoboCasa) or 20k (LIBERO-Plus) steps. FUMA gives the best success rate for every backbone on both benchmarks; the compared auxiliary objectives improve the average by at most 1.2 points on RoboCasa and do not improve it on LIBERO-Plus.

MethodX-VLAMolmoAct2π0.5GR00T N1.6Average
RoboCasa
Vanilla40.8±1.761.0±1.460.0±1.768.7±0.857.6±1.4
FLARE39.0±1.1−1.858.6±1.1−2.461.7±1.4+1.770.2±1.2+1.557.4±1.2−0.2
VLA-JEPA43.3±0.6+2.557.1±0.7−3.965.0±1.3+5.069.8±0.7+1.158.8±0.8+1.2
AttenA+44.5±1.8+3.758.6±1.3−2.462.0±0.7+2.068.6±2.1−0.158.4±1.5+0.8
FUMA (ours)47.7±0.7+6.964.5±0.6+3.568.4±1.4+8.472.1±1.5+3.463.2±1.1+5.6
LIBERO-Plus
Vanilla52.1±0.379.8±0.675.7±0.271.4±0.569.7±0.4
FLARE49.2±0.8−2.980.6±0.2+0.875.1±0.6−0.670.7±0.1−0.768.9±0.4−0.8
VLA-JEPA51.3±0.9−0.880.7±0.5+0.975.6±0.5−0.171.4±0.70.069.7±0.70.0
AttenA+52.5±0.8+0.480.6±0.3+0.874.1±0.7−1.670.1±0.8−1.369.3±0.7−0.4
FUMA (ours)62.7±0.7+10.682.9±0.5+3.180.6±0.6+4.973.9±0.7+2.575.0±0.6+5.3

Success rate (%), mean ± std over three seeds. Vanilla trains with each backbone's original objective only; the small numbers are the change in mean success relative to Vanilla.

Real-world manipulation

Four tasks, 20 teleoperated demonstrations each, 10 rollouts per policy and task. Training follows the RoboCasa budget.

The four real-world tasks: banana in bowl, cube stacking, bridge building, and peg insertion.
MethodEasyMediumHardAvg. (%)
BananaCubeBridgePeg
π0.5
Vanilla7/103/102/100/1030.0
FLARE7/104/103/100/1035.0
VLA-JEPA8/103/104/102/1042.5
AttenA+7/103/103/102/1037.5
FUMA (ours)9/108/108/104/1072.5
GR00T N1.6
Vanilla7/102/102/100/1027.5
FLARE6/103/103/100/1030.0
VLA-JEPA7/104/104/101/1040.0
AttenA+7/103/103/101/1035.0
FUMA (ours)8/107/106/103/1060.0

Successful rollouts out of 10 per task; Avg. is the mean success rate.

Most methods do well on placing a banana in a bowl. Cube stacking and bridge building need precise placement, where small action differences change the outcome, and FUMA leads clearly on both. On peg insertion FUMA is also best, by a smaller margin, which may reflect how well the learned world model captures the subtle alignment and contact changes that decide insertion.

Does FUMA reduce the errors that matter?

We analyze the geometry matrix GG for π0.5 on RoboCasa. An action chunk is 32 seven-dimensional actions, so G∈R224×224G \in \mathbb{R}^{224 \times 224}; its eigenvectors qkq_k are ordered from most to least sensitive.

(a) Mean squared action error along each eigenvector: FUMA is 26 to 32 percent lower than Vanilla on the most sensitive directions and 9 to 14 percent higher on the least sensitive ones. (b) Perturbing demonstration actions along high-eigenvalue directions lowers grasp success and raises end-effector error the most. (c) A perturbation along q1 makes the gripper miss the tomato (91 mm error), while an eight-times-larger perturbation along q210 still grasps it (19 mm).
(a) Mean squared action error along each qkq_k (top) and its change from Vanilla to FUMA (bottom). (b) Grasp success and end-effector error when demonstration actions are perturbed along qkq_k. (c) Executions perturbed along q1q_1 (×1) and q210q_{210} (×8), labeled with the final end-effector error. Bands: 95% bootstrap intervals.

Where errors shrink. Over 3,000 training states from 24 tasks, FUMA's action errors are lower than Vanilla's along the high-eigenvalue directions and higher along the low-eigenvalue ones, while its success rate is 8.4 points higher.

Why those directions. Perturbing demonstration actions at 125 pre-grasp states, perturbations along high-eigenvalue directions cut grasp success and raise end-effector error the most at the same magnitude. A nominal push along q1q_1 makes the gripper miss the tomato; one eight times larger along q210q_{210} still grasps it.

Ablations

All variants keep the backbone's original objective and are evaluated on RoboCasa. Replacing GG with the identity (plain MSE on the action estimate) stays at Vanilla level. A single fixed average GG, computed once before training, already gives +3.4 and needs no world-model passes during training. An RGB world model (Wan2.2) helps less than the latent one, and targeting the encoded observed future (GT Latent) is worse than targeting the prediction under the demonstration action (GT Action), which does not penalize the policy for world-model error.

MethodλX-VLAMolmoAct2π0.5GR00T N1.6Average
Vanilla–40.861.060.068.757.6
(a) Variants of GG
Identity matrix1.040.80.059.3−1.760.5+0.568.9+0.257.3−0.3
Fixed average matrix1.045.5+4.764.1+3.164.5+4.570.0+1.361.0+3.4
RGB world model1.044.2+3.462.8+1.864.0+4.071.2+2.560.5+2.9
FUMA w/ GT latent1.045.8+5.062.3+1.366.0+6.071.2+2.561.3+3.7
(b) Training objectives
LFUMA\mathcal{L}_{\mathrm{FUMA}} only–43.9+3.158.5−2.564.3+4.367.9−0.858.6+1.0
LVanilla+λLFUMA\mathcal{L}_{\mathrm{Vanilla}}+\lambda\mathcal{L}_{\mathrm{FUMA}}0.547.2+6.461.4+0.466.6+6.669.5+0.861.1+3.5
2.047.5+6.764.8+3.866.5+6.571.9+3.262.6+5.0
4.046.9+6.166.1+5.167.3+7.372.0+3.363.0+5.4
FUMA w/ GT action (ours)1.048.3+7.564.5+3.568.4+8.472.1+3.463.3+5.7

Success rate (%) on RoboCasa; small numbers are the change relative to Vanilla. Gray λ values are fixed to the default.

Keep the native loss

FUMA alone is on par with Vanilla (58.6 vs 57.6), and combining the two beats either alone on every backbone and every tested λ. Early in training the policy's actions can be far from the demonstrations, where the world model is less reliable, so FUMA alone starts slower; the flow-matching term anchors training to the demonstrations and removes that delay.

Every λ in {0.5, 1, 2, 4} improves over Vanilla on all four backbones (averages 61.1–63.3) with no consistent trend, so the weight needs no precise tuning. We use λ = 1.

Success rate over 10k training steps for pi0.5 on RoboCasa. The Vanilla plus FUMA curve is highest throughout, ending near 65 percent; FUMA alone starts slowest but is above Vanilla alone from about 7k steps on.
Success over training (π0.5, RoboCasa, 1,200 episodes per snapshot). Bands: binomial 95% intervals.

Citation

BibTeX will be added with the arXiv release.