VLA Zero to One: Algorithms, Selection, and Evidence¶

English · 简体中文 · Specialization home

This is the canonical VLA track. Its goal is not “run a large checkpoint once,” but understand every contract between synchronized robot data and closed-loop action, implement the principal algorithm families, and know when a VLA is the wrong choice.

1. Exit contract¶

You complete this track only when you can:

define observation, language, state, action, timing, and episode semantics without ambiguity;
implement a matched behavior-cloning baseline before using a pretrained VLA;
derive discrete-token, regression, diffusion, and flow-matching action objectives;
choose an action family from task multimodality, latency, data, and compute constraints;
run receding-horizon closed-loop evaluation with safety filters;
prove whether language affects behavior through controlled ablations;
report task success, failures, latency, uncertainty, and distribution shifts separately.

2. Problem formulation¶

At time \(t\), define the context

\[ c_t = (I_{t-k:t}^{1:V},\; q_{t-k:t},\; \ell,\; m_t), \]

where \(I^{1:V}\) are synchronized camera observations, \(q\) is robot state, \(\ell\) is a task instruction or goal, and \(m_t\) contains masks and embodiment metadata. A VLA policy predicts an action or action chunk:

\[ \pi_\theta(A_t \mid c_t), \qquad A_t=[a_t,\ldots,a_{t+H-1}]. \]

This equation hides the most common failures. Every symbol requires a contract:

Contract	Questions that must be answered
Images	Which cameras, exposure, crop, color order, timestamps, and missing-frame policy?
State	Joint or task space? Position, velocity, force? Which frame, units, ordering, and rate?
Language	Episode-level or step-level? Template, paraphrase, goal image, or absent?
Action	Absolute or delta? Joint, end-effector, gripper, velocity, or torque? Which frame and horizon?
Timing	Observation-to-action delay, control rate, chunk prediction rate, and execution horizon?
Boundary	Reset, success, failure, timeout, intervention, and padding masks?

If these are not explicit, training loss cannot be interpreted.

Principle · Representation is only half the system. The policy fuses visual, language, and state context, but a robot adapter, rate contract, command bounds, and re-observation determine whether its output becomes a valid closed loop.

3. Data pipeline¶

3.1 Minimum episode schema¶

Each step should retain:

episode_id, step_id, timestamp
images[camera_name]
robot_state, state_names, state_units
action, action_names, action_units, action_frame
instruction_or_goal, task_id
success, terminal, timeout, intervention
embodiment_id, calibration_id, dataset_version

Store the observation that was available when the action was chosen, not a later camera frame. Split by episode—and by object/scene/task/embodiment when testing that shift—rather than randomly splitting individual frames.

3.2 Normalization¶

Fit statistics on the training split only. Record the transform and its inverse. For an action dimension \(j\), a robust bounded mapping may use dataset quantiles or known physical limits, but the choice must preserve units and saturation behavior. Verify a round trip:

\[ a \xrightarrow{N} \tilde a \xrightarrow{N^{-1}} \hat a, \qquad \|a-\hat a\| < \varepsilon. \]

Never assume two robot datasets share action semantics because their vectors have the same length.

3.3 Coverage audit¶

Report task, object, pose, background, camera, trajectory length, failure/intervention, and action-range coverage. Language diversity is not demonstrated by many paraphrases of identical motions; physical diversity is not demonstrated by many frames from one episode.

4. Algorithms by component¶

4.1 Visual representation¶

Choice	Use when	Main risk	Required ablation
Small CNN trained from scratch	One controlled task and enough matched images	Overfits appearance	Background/viewpoint shift
Frozen pretrained vision encoder	Robot data is limited	Features may ignore contact geometry	Frozen vs partially tuned
VLM visual tower	Language grounding is central	Semantic features may lose fine geometry	Language and spatial perturbations
Video/history encoder	Occlusion, velocity, or contact phase matters	Temporal leakage and latency	Single-frame vs history

Start with the smallest representation that exposes the task signal. More semantic pretraining does not replace calibration, depth, proprioception, or force sensing.

4.2 Language conditioning¶

Common mechanisms are token concatenation with attention, cross-attention, FiLM-style modulation, or a goal embedding. The important test is causal, not architectural:

correct instruction;
paraphrased instruction;
swapped instruction from another task;
empty or masked instruction;
visually identical scene with a different goal.

If correct and swapped instructions behave the same, do not claim language grounding.

4.3 Multimodal fusion¶

Fusion	Strength	Limitation
Concatenate pooled embeddings + MLP	Simple and fast	Weak fine-grained token interaction
Encoder Transformer	Flexible cross-modal attention	Quadratic token cost
Decoder-only token stream	Reuses autoregressive language modeling	Ordering, masks, and decoding latency matter
Separate VLM + action expert	Preserves semantic backbone while specializing control	Interface and optimization become more complex

4.4 Action representation¶

Choose semantics before choosing a neural head.

Representation	Appropriate when	Failure to watch
Joint position/delta	Fixed embodiment, reliable joint servo	Cross-robot incompatibility
End-effector delta + gripper	Manipulation with an IK/controller layer	Frame and IK singularity errors
Joint velocity	Smooth rate control	Integration drift and bounds
Torque	Dynamics-rich low-level control	High safety and model requirements
Canonical cross-embodiment action	Multi-robot data	Adapter hides infeasible commands

Rotation requires an explicit representation and loss. Do not apply Euclidean MSE to wrapped Euler angles without handling discontinuities.

4.5 Direct regression and chunked behavior cloning¶

For a deterministic continuous chunk:

\[ \mathcal L_{\text{reg}} = \frac{1}{H}\sum_{h=0}^{H-1} \|a_{t+h}-\hat a_{t+h}\|_1 \]

or MSE. It is the first baseline because it is cheap, diagnosable, and latency-friendly. Its load-bearing assumption is that one central prediction represents the conditional action distribution. When demonstrations contain incompatible valid actions, regression may average them.

ACT is an action-chunking generative policy developed for fine-grained bimanual imitation. Treat “chunking” and “ACT” separately: many models predict chunks without using ACT’s full CVAE formulation.

4.6 Discrete action tokens¶

Quantize each normalized action dimension or encode action vectors into discrete codes, then minimize token cross-entropy:

\[ \mathcal L_{\text{token}} = -\sum_n \log p_\theta(z_n\mid z_{<n},c_t). \]

This integrates naturally with autoregressive VLM training, as in RT-2 and the original OpenVLA. The costs are quantization error, token ordering choices, and sequential decoding latency. Always report decode-to-action round-trip error.

4.7 Continuous parallel action chunks¶

A continuous head predicts the whole chunk in parallel. OpenVLA-OFT is evidence that a continuous action representation, parallel decoding, and action chunking can materially change the fine-tuning and inference trade-off of an autoregressive VLA. Do not transfer its reported throughput or benchmark gain to another implementation without reproducing its setup.

4.8 Diffusion action model¶

For a simplified noise-prediction formulation:

\[ x_\tau=\sqrt{\bar\alpha_\tau}A+\sqrt{1-\bar\alpha_\tau}\epsilon, \qquad \mathcal L_{\text{diff}}=\|\epsilon-\epsilon_\theta(x_\tau,\tau,c_t)\|^2. \]

Diffusion Policy motivates this family for multimodal, high-dimensional action distributions with receding-horizon control. Its disadvantages are iterative inference and additional schedule/sampler choices. Compare under equal observation, data, chunk, and execution budgets.

4.9 Flow matching action expert¶

In a simple conditional flow-matching path, sample noise \(x_0\), a data action chunk \(x_1=A\), and time \(\tau\):

\[ x_\tau=(1-\tau)x_0+\tau x_1,\quad u_\tau=x_1-x_0,\quad \mathcal L_{\text{FM}}=\|v_\theta(x_\tau,\tau,c_t)-u_\tau\|^2. \]

Inference numerically integrates the learned vector field. π0 and SmolVLA use flow-based continuous action generation, but their architecture and training details are not interchangeable. Sampling speed must be measured end to end; “flow” is not a universal latency guarantee.

5. How to choose an algorithm¶

Step 1 · Does language change the required action?¶

No: start with BC, ACT, or Diffusion Policy. A VLA adds unnecessary capacity and a false language story.
Yes: add language to the matched baseline, then test swapped/absent instructions.

Step 2 · Is the conditional action distribution multimodal?¶

Low or unknown: continuous chunk regression first.
Clearly multimodal: compare diffusion or flow matching against regression; retain multiple samples and task outcomes.

Step 3 · What is the latency contract?¶

Hard real-time budget: prefer parallel regression/chunking initially; measure preprocessing, model, postprocessing, network, and queueing together.
Soft budget: generative heads are candidates, but still use receding-horizon execution.

Step 4 · What data and compute exist?¶

Situation	Recommended first experiment
Task-specific data, limited compute	Small chunked BC/ACT; language only if tasks vary
Multi-task data, one GPU	Fine-tune a compact/continuous-chunk VLA and keep a from-scratch baseline
Heterogeneous multi-robot data	Explicit embodiment adapters, per-robot normalization, and per-robot metrics
Large compute and strong language diversity	Compare discrete, continuous, and flow heads under matched data

Step 5 · Do not select by parameter count alone¶

Select using task semantics, data compatibility, control rate, action multimodality, reproducibility, license, and deployment constraints. The repository selector makes these assumptions visible:

python scripts/select_vla_wam_algorithm.py \
  --goal multimodal-action \
  --compute single-gpu \
  --data task-specific \
  --latency soft

6. Zero-to-one build sequence¶

Stage	Build	Promotion gate
0	Task and interface contract	Frames, units, rate, success, stop conditions reviewed
1	State-only BC	Tiny-set overfit and closed-loop state baseline pass
2	Vision BC	Image/action synchronization and unseen-initial-state evaluation pass
3	Language-conditioned baseline	Correct language beats swapped/absent language
4	Action chunks	Receding-horizon execution beats or matches one-step policy
5	Generative head	Multimodal benefit appears in task outcomes, not only likelihood
6	Pretrained VLA fine-tuning	Matched from-scratch baseline and data receipt retained
7	Shift suite	Object, pose, scene, language, and embodiment shifts reported separately
8	Deployment rehearsal	Latency, bounds, watchdog, shadow mode, and rollback pass

Suggested repository path:

python scripts/run_knowledge_map.py --path-to learning-vla
python scripts/run_pipeline.py --run vla-policy --dry-run
python scripts/run_pipeline.py --run vla-policy

The included PushCube VLA is a teaching baseline. It does not reproduce OpenVLA, π0, or SmolVLA pretraining.

7. Training and fine-tuning protocol¶

Validate schema and normalization on a few samples.
Overfit one tiny subset; failure indicates a data/objective/implementation bug.
Freeze train/validation/test episode identities before tuning.
Record backbone initialization, frozen layers, optimizer groups, precision, accumulation, and augmentation.
Compare full tuning, partial tuning, and parameter-efficient tuning only under matched data and steps.
Select checkpoints by a declared validation rule, not by test performance.
Evaluate closed loop with fixed resets, seeds or trial allocation, and intervention rules.

The official LeRobot SmolVLA guide recommends task-specific fine-tuning and gives one dataset/training starting point. Treat its episode and step counts as framework guidance for that setup, not a universal sample-complexity law.

8. Evaluation matrix¶

Axis	Required measurement
Offline	Loss by action dimension/horizon; decode error; validation split identity
Closed loop	Task success, progress, time, interventions, constraint violations
Language	Correct, paraphrased, swapped, absent, and contradictory instruction
Vision	Camera removal, occlusion, background, viewpoint, and lighting shifts
Action	Bounds, smoothness, saturation, chunk overlap, and execution delay
Generalization	Separate task, object, scene, embodiment, and motion shifts
Systems	End-to-end latency distribution, memory, dropped frames, queue age
Statistics	Episodes/trials, seed policy, confidence interval, and failure counts

Offline action error can improve while task success worsens. Language-conditioned success can improve because tasks correlate with backgrounds. Report these failure paths rather than hiding them in a mean.

9. Failure localization¶

Symptom	First checks
Good training loss, no task success	Time alignment, normalization inverse, action frame, reset distribution
Jerky or delayed motion	Queue age, chunk overlap, rate mismatch, network latency
Ignores language	Task imbalance, visual shortcut, swapped/absent-language ablation
Averages two strategies	Multimodal demonstrations; compare diffusion/flow head
Fails at contact	Camera geometry, proprio/force feedback, action rate, controller—not only VLA scale
Cross-robot collapse	Per-embodiment adapters, units, masks, action feasibility, pooled metrics
Great offline result, weak shift result	Leakage, narrow coverage, checkpoint selection, unreported OOD

10. Research questions worth testing¶

Does language provide causal task disambiguation beyond visual shortcuts?
Which action representation transfers across embodiments without hiding infeasible commands?
Does a generative action head improve closed-loop multimodal behavior at a fixed latency budget?
Which layers should adapt under small in-domain datasets?
Does history improve partial observability, or only leak action timing?
Can uncertainty predict failure early enough for a safe fallback?

For each question, define a matched baseline, one changed factor, a frozen shift suite, and a result that would reject the hypothesis.

11. Primary sources¶

Reported paper results are source claims. This track provides an algorithm and evidence curriculum, not a claim that the repository reproduces those systems.