SVFM
Soft-Value Flow Matching for Offline Reinforcement Learning
1KAIST 2Hanyang University 3Korea University
*Equal contribution †Corresponding authors
Paper (PDF) arXiv Code BibTeX
TL;DR SVFM improves a multi-step flow policy with a learned critic, without backpropagating through sampling. It evaluates the critic at endpoints of a fixed behavior process, not the current actor, and supervises the actor with the gradient of the resulting soft value.
Behavior data
Over-estimated OOD region
Critic queries

Abstract
Flow-matching policies represent multimodal actions, but improving them with a learned critic in offline reinforcement learning faces two difficulties: backpropagating through multi-step sampling is costly and unstable, and critic queries on actor-generated actions can expose the policy to out-of-distribution value errors. We introduce Soft-Value Flow Matching (SVFM), which trains a multi-step actor with the gradient of a conditional soft value for KL-regularized policy improvement. Starting from noised data actions, SVFM samples stochastic continuations of a fixed behavior process and evaluates the critic at their endpoints, keeping the queries used for guidance independent of the current actor. A log-mean-exp of these values provides a Monte Carlo target for a soft-value network, whose gradient guides actor updates without trajectory backpropagation. We further propose an adaptive temperature that scales with the spread of critic values across continuations. SVFM achieves higher average success rates than the compared baselines on OGBench, LIBERO, and LIBERO-Plus. It also improves real-world manipulation with both task-specific flow policies and a pretrained VLA model.
Where the critic is queried matters
Flow and diffusion policies generate actions through many integration steps, so improving them with a critic usually means either backpropagating through the whole sampling trajectory, keeping the actor fixed and adding test-time computation, or distilling into a one-step policy. Recent methods avoid trajectory backpropagation by building guidance from the actor's own samples.
That choice has a cost in offline RL. A critic learned from a fixed dataset can over-estimate actions the data never covers. When guidance comes from actor samples, every actor update also moves where the critic is evaluated, possibly toward those over-estimated actions. In the toy task above, queries on actor samples drift into the over-estimated region; queries from fixed behavior continuations stay near the data.
Method
The Doob h-transform says the optimal correction to a behavior process, for KL-regularized policy improvement, is the gradient of a conditional soft value: a log-mean-exp of critic values over the behavior process's possible endpoints. SVFM learns that function directly from behavior continuations and regresses the actor onto the guided velocity.
-
Behavior continuations
At a data-path anchor , re-integrate the learned behavior field as a stochastic process with independent Brownian paths to get terminal actions . A parameter scales the noise, so continuations stay anchor-specific instead of collapsing to a common mean shift at small .
-
Soft-value regression
Score the endpoints with the critic and fit a soft-value network to their log-mean-exp:
-
Guided flow matching
Regress the actor velocity onto the data velocity plus the scaled soft-value gradient. No trajectory backpropagation is needed:
-
Adaptive temperature
With a constant , how strongly the soft value favors high-value continuations changes as the critic's scale changes. SVFM sets the temperature from the spread of critic values across continuations, , with a dimensionless gain :
The resulting guidance is invariant to positive affine rescaling of .
Offline RL on OGBench
Ten OGBench domains (five tasks each) under two protocols: the standard setting of FQL (8 seeds) and the advanced setting of QAM (12 seeds), which adds action chunking for manipulation, a 10-critic ensemble, and pessimistic value learning.


Offline-to-online fine-tuning
Each method trains offline for 1M gradient updates, then fine-tunes for 1M online environment steps (8 seeds). SVFM keeps improving with online interaction and reaches the highest final average, with the largest gain on puzzle-4x4.

Adaptive vs. constant temperature
Under the standard setting (default tasks, 3 seeds), adaptive temperature averages 56%, above the best single constant temperature (45%) and the best constant chosen per domain in hindsight (51%). It ranks first on 6 of 10 domains.
Fine-tuning a VLA: LIBERO and LIBERO-Plus
We fine-tune π0.5 from a behavior-cloned policy trained on 5 demonstrations per task. SVFM's critic sees the demonstrations and 2,000 policy rollouts, failures included. Best-of-N and QGF use a critic trained on the same data for the same number of updates. LIBERO is evaluated with 50 episodes per task, LIBERO-Plus on 10,030 perturbed tasks.
The largest gain over BC is on LIBERO-Long (52% → 71%). On LIBERO-Plus, SVFM is best or tied for best under every perturbation type.
Per-suite and per-perturbation results
| BC | Filtered BC | QGF | BoN | Q-VGM | SVFM (ours) | |
|---|---|---|---|---|---|---|
| LIBERO (by task suite) | ||||||
| Spatial | 92±1 | 96±1 | 91±0 | 92±0 | 92±2 | 95±1 |
| Object | 78±1 | 81±1 | 95±2 | 81±2 | 86±1 | 90±2 |
| Goal | 85±0 | 90±1 | 83±1 | 88±2 | 93±1 | 89±1 |
| Long | 52±0 | 60±2 | 52±2 | 46±2 | 47±1 | 71±2 |
| Average | 77±0 | 82±0 | 80±1 | 77±0 | 80±1 | 86±1 |
| LIBERO-Plus (by perturbation) | ||||||
| Camera | 52±1 | 55±0 | 50±0 | 49±1 | 52±1 | 61±0 |
| Robot | 62±1 | 64±1 | 59±0 | 59±0 | 60±1 | 66±1 |
| Language | 69±0 | 71±1 | 69±1 | 67±2 | 70±1 | 75±1 |
| Light | 79±2 | 79±1 | 81±1 | 78±1 | 82±1 | 86±1 |
| Background | 74±1 | 79±1 | 76±2 | 76±1 | 77±1 | 83±1 |
| Noise | 63±1 | 73±0 | 57±0 | 59±0 | 66±1 | 73±1 |
| Layout | 69±0 | 71±1 | 71±1 | 69±0 | 72±1 | 75±0 |
| Average | 66±0 | 69±0 | 65±0 | 64±0 | 67±0 | 73±1 |
Real-world manipulation
Four tasks, 50 teleoperated demonstrations each, 20 trials per method and task.

Task-specific flow policies
ResNet-18 encoders with an MLP flow-matching actor. SVFM reaches 64% on average and matches or exceeds every baseline on every task; on cube stacking, Flow BC fails all 20 trials while SVFM succeeds in half.
Pretrained VLA: GR00T N1.7
Fine-tuned jointly on all four tasks. SVFM trains for 10K updates from a GR00T policy behavior-cloned for 10K updates; BC+ instead behavior-clones for 20K updates, matching SVFM's total actor updates.
| BC+ | BoN | SVFM (ours) | ||||
|---|---|---|---|---|---|---|
| Task | SR (%) | Path (m) | SR (%) | Path (m) | SR (%) | Path (m) |
| Carrot in pot | 95 | 1.21 | 65 | 1.54 | 95 | 0.89 |
| Bowl stacking | 100 | 0.61 | 100 | 0.60 | 100 | 0.57 |
| Triple bowl stacking | 90 | 1.44 | 75 | 1.28 | 100 | 1.36 |
| Cube stacking | 5 | 1.65 | 5 | 0.86 | 100 | 0.78 |
| Average | 73 | 1.23 | 61 | 1.07 | 99 | 0.90 |
Success rate over 20 trials per task; median end-effector path length over successful trials (lower is shorter). Bold marks the best per row.
Smoother, more direct motion
Among successful trials, Flow BC shows irregular back-and-forth motion, while SVFM follows smoother, more consistent paths through grasp and release. Four successful trials per method from the same initial placement:
Carrot in pot


Bowl stacking


Citation
TBA