SVFM

SVFM

Soft-Value Flow Matching for Offline Reinforcement Learning

Dohyun Lee1* Junhyeong Hong2* Theo Choi2 Kinam Kim1 Jaegul Choo1 Kyungjae Lee3† Yoonseon Oh2†

1KAIST 2Hanyang University 3Korea University

*Equal contribution †Corresponding authors

TL;DR SVFM improves a multi-step flow policy with a learned critic, without backpropagating through sampling. It evaluates the critic at endpoints of a fixed behavior process, not the current actor, and supervises the actor with the gradient of the resulting soft value.

Behavior data Over-estimated OOD region Critic queries
Four panels of a 2D toy task. Critic queries used for actor supervision fall in an over-estimated out-of-distribution region for FQL (19.1%), QAM (12.6%), and Q-Flow (39.3%), but only 1.3% of the time for SVFM.
FQLQAMQ-FlowSVFM (ours)
Critic queries for actor supervision when the critic over-estimates an OOD region in a toy task. Queries on actor samples enter that region, whereas SVFM evaluates the critic there much less often.

Abstract

Flow-matching policies represent multimodal actions, but improving them with a learned critic in offline reinforcement learning faces two difficulties: backpropagating through multi-step sampling is costly and unstable, and critic queries on actor-generated actions can expose the policy to out-of-distribution value errors. We introduce Soft-Value Flow Matching (SVFM), which trains a multi-step actor with the gradient of a conditional soft value for KL-regularized policy improvement. Starting from noised data actions, SVFM samples stochastic continuations of a fixed behavior process and evaluates the critic at their endpoints, keeping the queries used for guidance independent of the current actor. A log-mean-exp of these values provides a Monte Carlo target for a soft-value network, whose gradient guides actor updates without trajectory backpropagation. We further propose an adaptive temperature that scales with the spread of critic values across continuations. SVFM achieves higher average success rates than the compared baselines on OGBench, LIBERO, and LIBERO-Plus. It also improves real-world manipulation with both task-specific flow policies and a pretrained VLA model.

Where the critic is queried matters

Flow and diffusion policies generate actions through many integration steps, so improving them with a critic usually means either backpropagating through the whole sampling trajectory, keeping the actor fixed and adding test-time computation, or distilling into a one-step policy. Recent methods avoid trajectory backpropagation by building guidance from the actor's own samples.

That choice has a cost in offline RL. A critic learned from a fixed dataset can over-estimate actions the data never covers. When guidance comes from actor samples, every actor update also moves where the critic is evaluated, possibly toward those over-estimated actions. In the toy task above, queries on actor samples drift into the over-estimated region; queries from fixed behavior continuations stay near the data.

Method

The Doob h-transform says the optimal correction to a behavior process, for KL-regularized policy improvement, is the gradient of a conditional soft value: a log-mean-exp of critic values over the behavior process's possible endpoints. SVFM learns that function directly from behavior continuations and regresses the actor onto the guided velocity.

From a noised data action, K stochastic behavior continuations reach terminal actions; the critic scores them; a log-mean-exp gives the soft-value target; the soft value's gradient guides the flow actor. anchor xt​ (1−t)x0 + t·a K behavior continuations (fixed uφ) Q = 0.9Q = 0.4 Q = 0.7Q = 0.1 soft-value target λ log mean exp(Q/λ) soft-value network Vψ(s, xt, t) regress onto target ∇xV flow actor vθ​ target: (a − x0) + βt∇V
The critic only sees endpoints of the fixed behavior process, so actor updates never move the queries used for supervision. Toy Q values are illustrative.
  1. Behavior continuations

    At a data-path anchor xt=(1−t)x0+tax_t=(1-t)x_0+ta, re-integrate the learned behavior field uϕu_\phi as a stochastic process with KK independent Brownian paths to get terminal actions a(1),…,a(K)a^{(1)},\dots,a^{(K)}. A parameter κ∈(0,1]\kappa\in(0,1] scales the noise, so continuations stay anchor-specific instead of collapsing to a common mean shift at small tt.

  2. Soft-value regression

    Score the endpoints with the critic and fit a soft-value network to their log-mean-exp:

    V^K(s,xt,t)=λlog⁡(1K∑j=1KeQ(s,a(j))/λ),LV(ψ)=E[(Vψ(s,xt,t)−sg[V^K])2]\widehat V_K(s,x_t,t)=\lambda\log\Big(\frac{1}{K}\sum_{j=1}^{K}e^{Q(s,a^{(j)})/\lambda}\Big),\qquad \mathcal L_V(\psi)=\mathbb E\big[(V_\psi(s,x_t,t)-\mathrm{sg}[\widehat V_K])^2\big]
  3. Guided flow matching

    Regress the actor velocity onto the data velocity plus the scaled soft-value gradient. No trajectory backpropagation is needed:

    Lactor(θ)=E ∥vθ(xt,t)−sg[(a−x0)+βt ∇xVψ(s,xt,t)]∥2,βt=κ21−ttλ\mathcal L_{\mathrm{actor}}(\theta)=\mathbb E\,\Big\|v_\theta(x_t,t)-\mathrm{sg}\big[(a-x_0)+\beta_t\,\nabla_x V_\psi(s,x_t,t)\big]\Big\|^2,\qquad \beta_t=\kappa^2\frac{1-t}{t\lambda}
  4. Adaptive temperature

    With a constant λ\lambda, how strongly the soft value favors high-value continuations changes as the critic's scale changes. SVFM sets the temperature from the spread of critic values across continuations, sQ=Ext[stdj Q(s,a(j))]s_Q=\mathbb E_{x_t}[\mathrm{std}_j\,Q(s,a^{(j)})], with a dimensionless gain gg:

    λ=κ2sQg⟹βt=g (1−t)t sQ\lambda=\frac{\kappa^2 s_Q}{g}\quad\Longrightarrow\quad \beta_t=\frac{g\,(1-t)}{t\,s_Q}

    The resulting guidance is invariant to positive affine rescaling of QQ.

Exactness. With exact components and fixed λ,κ\lambda,\kappa, integrating x˙t=u(xt,t)+κ21−ttλ∇xVλ,κ(s,xt,t)\dot x_t=u(x_t,t)+\kappa^2\frac{1-t}{t\lambda}\nabla_xV_{\lambda,\kappa}(s,x_t,t) from x0∼N(0,I)x_0\sim\mathcal N(0,I) produces samples from π⋆(a∣s)∝πβ(a∣s)exp⁡(Q(s,a)/λ)\pi^\star(a\mid s)\propto\pi_\beta(a\mid s)\exp(Q(s,a)/\lambda), the KL-regularized optimum (Proposition 1 in the paper).

Offline RL on OGBench

Ten OGBench domains (five tasks each) under two protocols: the standard setting of FQL (8 seeds) and the advanced setting of QAM (12 seeds), which adds action chunking for manipulation, a 10-critic ensemble, and pessimistic value learning.

Table 1: OGBench success rates under the standard setting. SVFM averages 57%, versus 53% for Q-Flow, 44% for FQL, and 42% for CEP.
OGBench success rates (%) under the standard setting, averaged over the 5 tasks of each domain, 8 seeds. Bold marks the best result per row.
Table 2: OGBench success rates under the advanced setting. SVFM averages 56%, versus 47% for Q-Flow and 44% for QAM-E.
OGBench success rates (%) under the advanced setting, averaged over the 5 tasks of each domain, 12 seeds. Bold marks the best result per row. “-sparse” denotes sparse-reward variants.

Offline-to-online fine-tuning

Each method trains offline for 1M gradient updates, then fine-tunes for 1M online environment steps (8 seeds). SVFM keeps improving with online interaction and reaches the highest final average, with the largest gain on puzzle-4x4.

Success rate over 2M training steps (1M offline, then 1M online) on antmaze-giant, humanoidmaze-medium, antsoccer, cube-double, puzzle-4x4, and the average. SVFM reaches about 93% average at the end, above Q-Flow, FQL, FBRAC, and IFQL.

Adaptive vs. constant temperature

Under the standard setting (default tasks, 3 seeds), adaptive temperature averages 56%, above the best single constant temperature (45%) and the best constant chosen per domain in hindsight (51%). It ranks first on 6 of 10 domains.

Constant λ = 0.127%
Constant λ = 0.545%
Constant λ = 1.038%
Constant λ = 3.022%
Best constant per domain51%
Adaptive λ (ours)56%

Fine-tuning a VLA: LIBERO and LIBERO-Plus

We fine-tune π0.5 from a behavior-cloned policy trained on 5 demonstrations per task. SVFM's critic sees the demonstrations and 2,000 policy rollouts, failures included. Best-of-N and QGF use a critic trained on the same data for the same number of updates. LIBERO is evaluated with 50 episodes per task, LIBERO-Plus on 10,030 perturbed tasks.

LIBERO (average of 4 suites)
BC77%
Filtered BC82%
QGF80%
BoN77%
Q-VGM80%
SVFM (ours)86%
LIBERO-Plus (average of 7 perturbations)
BC66%
Filtered BC69%
QGF65%
BoN64%
Q-VGM67%
SVFM (ours)73%

The largest gain over BC is on LIBERO-Long (52% → 71%). On LIBERO-Plus, SVFM is best or tied for best under every perturbation type.

Per-suite and per-perturbation results
BCFiltered BCQGFBoNQ-VGMSVFM (ours)
LIBERO (by task suite)
Spatial92±196±191±092±092±295±1
Object78±181±195±281±286±190±2
Goal85±090±183±188±293±189±1
Long52±060±252±246±247±171±2
Average77±082±080±177±080±186±1
LIBERO-Plus (by perturbation)
Camera52±155±050±049±152±161±0
Robot62±164±159±059±060±166±1
Language69±071±169±167±270±175±1
Light79±279±181±178±182±186±1
Background74±179±176±276±177±183±1
Noise63±173±057±059±066±173±1
Layout69±071±171±169±072±175±0
Average66±069±065±064±067±073±1

Real-world manipulation

Four tasks, 50 teleoperated demonstrations each, 20 trials per method and task.

The four real-world tasks: carrot in pot, bowl stacking, triple bowl stacking, and cube stacking.

Task-specific flow policies

ResNet-18 encoders with an MLP flow-matching actor. SVFM reaches 64% on average and matches or exceeds every baseline on every task; on cube stacking, Flow BC fails all 20 trials while SVFM succeeds in half.

Flow BCBoNQ-FlowSVFM (ours)
20
55
65
80
Carrot in pot
40
50
95
95
Bowl stacking
0
0
30
30
Triple bowl stacking
0
20
25
50
Cube stacking
15
31
54
64
Average

Pretrained VLA: GR00T N1.7

Fine-tuned jointly on all four tasks. SVFM trains for 10K updates from a GR00T policy behavior-cloned for 10K updates; BC+ instead behavior-clones for 20K updates, matching SVFM's total actor updates.

BC+BoNSVFM (ours)
TaskSR (%)Path (m)SR (%)Path (m)SR (%)Path (m)
Carrot in pot951.21651.54950.89
Bowl stacking1000.611000.601000.57
Triple bowl stacking901.44751.281001.36
Cube stacking51.6550.861000.78
Average731.23611.07990.90

Success rate over 20 trials per task; median end-effector path length over successful trials (lower is shorter). Bold marks the best per row.

Smoother, more direct motion

Among successful trials, Flow BC shows irregular back-and-forth motion, while SVFM follows smoother, more consistent paths through grasp and release. Four successful trials per method from the same initial placement:

Flow BC SVFM (ours)

Carrot in pot

Flow BC end-effector trajectories for carrot in pot
Flow BC
SVFM end-effector trajectories for carrot in pot
SVFM

Bowl stacking

Flow BC end-effector trajectories for bowl stacking
Flow BC
SVFM end-effector trajectories for bowl stacking
SVFM
Successful end-effector trajectories overlaid on a photograph of each scene for illustration. Each policy is trained once, so we treat these as qualitative evidence.

Citation

TBA