Part 5 · Borrowing from VLAs: action tokens and action chunks5편 · VLA에서 빌려 온 것: 액션 토큰과 액션 청크第 5 篇 · 向 VLA 借鉴：动作 token 与动作块

Part 4 ended with a diagnosis. The words matter on one frame per episode, the first, which is 2% of the training rows. On that frame the policy answered with the average of the two sentences’ answers. On every other frame the image already showed where to go.

Large vision-language-action models (VLAs) don’t output actions the way that policy did. RT-2 and OpenVLA turn each action dimension into one of 256 tokens and train with cross-entropy. OpenVLA-OFT and ACT predict a chunk of several future actions at once. This post borrows those two output formats and nothing else. The encoders keep their design, the data and the parameter budget stay as they were, and the image layer shrinks from 128 to 104 numbers to keep the parameter counts comparable. There is still no pretrained model; that arrives in Season 4.

How these experiments were run

This post is a screen, not a final result, so the rules are lighter than Part 4’s and are stated up front:

• One training seed per model (seed 0). Part 4 used three.
• Validation only: 10 pairs, 20 flights. The test split is untouched.
• Same training recipe as Part 4: 20 epochs, keep the last, dataset v0.2 (100 training pairs).
• Same size: every model has 80–100% of Part 4’s 480,901 parameters.

Single-pair differences are noise at this sample size, so the post gives intervals where it matters.

1. What changed: only the output

Tokens. For each axis, the training speeds are split into 256 equal bins between their 1st and 99th percentiles, as OpenVLA does. The head outputs 256 scores per axis instead of one number. Training asks it to score the expert’s bin highest (cross-entropy); flying uses the centre of its top bin (model code):

def set_action_bins(self, actions):            # actions: (N, 4) training speeds / limits
    for i, d in enumerate((0, 1)):             # vx, vy; vz and yaw rate are always 0 here
        lo, hi = torch.quantile(actions[:, d], torch.tensor([0.01, 0.99]))
        self.bin_edges[i] = torch.linspace(lo, hi, 257)

def tokenize(self, actions):                   # training target: which bin
    return torch.stack([torch.bucketize(actions[:, d], self.bin_edges[i, 1:-1])
                        for i, d in enumerate((0, 1))], dim=1)

def detokenize(self, logits):                  # flying: the centre of the most likely bin
    centres = (self.bin_edges[:, :-1] + self.bin_edges[:, 1:]) / 2
    return centres.gather(1, logits.argmax(-1).T).T

Chunks. The head predicts the next 8 steps at once. A shared trunk produces one feature, a learned embedding for each step k is added, and one decoder turns each into bins plus a Stop logit. The loss covers every future step that exists in the episode (training code):

logits_h, stops_h = model.heads(feats)[2]          # (B, 8, 2, 256), (B, 8)
target = model.tokenize(chunk_action[idx])         # expert's next 8 actions, as bins
ce = cross_entropy(logits_h.flatten(0, 2), target.flatten(), reduction="none")
motion = (ce.view(B, 8, 2).mean(-1) * mask).sum() / mask.sum()   # mask: steps past the end
stop = (bce(stops_h, chunk_stop[idx]) * mask).sum() / mask.sum()
loss = motion + stop

The model is still asked every 0.2 s, and only the first step is flown. So the chunk changes what the model is trained to predict, not how it flies. Section 3 also tests flying all 8 steps.

2. Why tokens fly better: averaging versus choosing

On the first frame, the image and the state are identical for both sentences of a pair, and their labels differ. Regression with a Huber loss is pulled toward the middle. Over all 200 training first frames, the Part 4 model is closer to its own sentence’s answer in exactly 100 and to the other sentence’s answer in the other 100. It sits on average 0.03 m/s from the midpoint.

A token head can’t place its answer between two bins it was never shown. It has to put its probability somewhere, and it mostly puts it on one expert answer. It picks the right one in 135 of 200 training first frames. Its median error is 0.0005 m/s, but its mean is 0.066 m/s, because the wrong picks are a whole answer away. Training pair 0, sentence B in the figure is such a wrong pick.

In flight, a regression policy that starts in the middle tends to hover between the pillars. A token policy commits to a side. That alone moves the numbers:

This is the multimodality problem that Diffusion Policy and Behavior Transformers were built for: when two answers are both right, a model that averages them is wrong for both. Here the two answers come from two sentences rather than from two demonstrators, but the loss can’t tell the difference on that frame.

3. Results: the chunk is the best end-to-end model so far

• Tokens alone don’t make the drone choose. With 256 bins, 7 of 10 pairs still go to the same pillar for both sentences. The token head commits, but on most first frames it commits to whatever the image suggests.
• Tokens + an 8-step chunk is the best end-to-end model: 4 pairs split the right way, and pair success is 3/10. One plausible reason: predicting 1.6 s ahead forces the representation to encode where the flight is going, not just the next step. Its 95% interval is 0.11–0.60, which overlaps Part 4’s 0/10 (0–0.28). With one seed and ten pairs this is a lead, not a proven gain.
• Flying all 8 predicted steps before asking again flies about as well (9/20) but listens less (1 correct pair). My guess: re-asking every step lets each new image correct the course, and flying 8 steps blind gives that up.

The first three validation pairs, unselected, show how far from solved this is:

4. More states: it reacts, then it can’t stop

Part 4’s diagnosis was that the words decide only on frames the policy rarely sees. Part 6’s world model was trained on three times as many states: expert flights plus exploration flights (noisy expert, random walks, overshoots past the goal). So I gave the chunk policy the same 400 exploration flights (19,896 states), each state labelled with what the expert would do from there toward a target picked at random for that episode, with the matching sentence. For exploration rows only the current step is a label; their future chunk steps are masked, because a random walk isn’t the expert’s plan.

On training frames, the swap test now looks like the expert. Changing only the sentence moves the first command by a median 0.196 m/s (the expert: 0.200), and it moves in the right direction in 180 of 200 frames. In closed loop it splits 5 of 10 pairs the right way, the most of any end-to-end model (cross-attention + counterfactual relabelling also reaches 5). But 8 of 20 flights overshoot the target and leave the room, success falls to 6, and pair success is 0.

I didn’t separate the causes. Three candidates:

• Stop is diluted. The exploration rows contain only 6 Stop examples, so Stop becomes a rarer event in the training mix.
• The bins got coarser. The bin range comes from the training speeds, and exploration widens the vy range from about ±0.17 to ±0.42 m/s. Each bin is 2.4 times wider, which is coarse near the goal.
• Recovery labels near the goal. Overshoot flights teach large corrections exactly where the expert usually slows down.

With exploration states the first-frame reaction matched the expert’s on training frames (median 0.196 vs 0.200 m/s), but pair success fell from 3/10 to 0/10. The experiment changes what the policy reacts to without changing how it stops. That split, choosing versus flying, is what Part 6 builds in on purpose.

What this part does not show

• One training seed and ten pairs. Wilson 95% for 3/10 is 0.11–0.60. Single-pair differences, and the bin-count comparison in the appendix, are within noise.
• Validation was used both to screen models and to set the Stop threshold, so these are not generalisation estimates. The original test seeds were seen in an earlier round; final numbers need a fresh test split.
• The swap test runs on training frames. Near-zero token errors there may partly be memorisation.
• The language is easy. The two pillars always differ in colour, so one word decides.
• It is not a VLA. Only the output format is borrowed; there is no pretrained vision or language model.

Next

No end-to-end model splits more than 5 of 10 pairs the right way, and the best one completes 3. Part 6 tries the opposite design: a model that perceives where every target is without reading the words, a tiny selector that lets the words pick one, and a planner that flies there.

Appendix

A. Everything else I tried (validation, one seed)

model	params	correct / swapped / same-target pairs	success /20	pair success /10
Part 4 regression	480,901	0 / 0 / 10	3	0
+ counterfactual relabelling	480,901	2 / 3 / 5	0	0
+ counterfactual + FiLM	492,517	4 / 3 / 3	0	0
+ target-position aux head	493,773	0 / 0 / 10	8	0
+ cross-attention	420,997	0 / 0 / 10	0	0
+ cross-attention + counterfactual	420,997	5 / 3 / 2	0	0
256-bin tokens	469,609	3 / 0 / 7	8	1
64-bin tokens	471,289	1 / 1 / 8	7	1
1024-bin tokens	475,609	2 / 0 / 8	4	0
256-bin + paired goal loss	480,555	2 / 0 / 8	7	0
256-bin + cross-attention	477,289	1 / 0 / 9	9	0
256-bin + chunk 8, fly step 1	470,633	4 / 1 / 5	8	3
256-bin + chunk 8, fly all 8	470,633	1 / 0 / 9	9	1
chunk 8 + exploration states	470,633	5 / 2 / 3	6	0
chunk 8 + exploration + aux head	468,909	5 / 2 / 3	2	0

Notes:

• Counterfactual relabelling adds, for every training state, the expert's action toward the other target with the other sentence. The words start to matter, but correct and swapped pairs are about equal, and flying collapses. Part of that is a confound: the added rows contain no Stop positives, while the Stop loss weight stayed at its original value.
• Aux head: an extra output trained to predict where every colour's target is, from the simulator's true positions (the same labels Part 6's world model gets). It flies better (8/20) and listens no more.
• FiLM and cross-attention let the sentence modulate the image features (FiLM scales and shifts the convolution channels; cross-attention uses the sentence as a query over 6 × 8 image patches). The FiLM row and the aux-head row predate the 80–100% parameter rule (FiLM 102%, aux 103%).
• Bin count: 64, 256 and 1024 bins are within one seed's noise of each other.

python -m dronevla.train --data data/v0.2 --epochs 20 --select last --seed 0 \
    --action-bins 256 --img-dim 104 --out runs/bc_tok256_v0.2_s0
python -m dronevla.train --data data/v0.2 --epochs 20 --select last --seed 0 \
    --action-bins 256 --img-dim 104 --chunk 8 --out runs/bc_tokchunk8_v0.2_s0
python -m dronevla.train --data data/v0.2 --epochs 20 --select last --seed 0 \
    --action-bins 256 --img-dim 104 --chunk 8 --explore data/explore_v0.2/train \
    --out runs/bc_tokchunk8_explore_v0.2_s0
bash scripts/run_tokens_s0.sh && bash scripts/run_tokens2_s0.sh   # evaluate (and train if missing)
bash scripts/run_explore_s0.sh
python scripts/season2_summary.py                 # every number in this post (needs all run_*_s0.sh)
python scripts/figures/blog05.py                  # figures
bash scripts/figures/render_blog05_videos.sh      # GIFs

References: what I took from each

• OpenVLA (Kim et al., 2024) and RT-2 (Brohan et al., 2023) discretise each action dimension into 256 bins and predict them as tokens. → The token head and its 1st–99th percentile bins.
• OpenVLA-OFT (Kim, Finn and Liang, 2025) and ACT (Zhao et al., 2023) predict chunks of future actions. → The 8-step chunk. Here only the first step is flown.
• Diffusion Policy (Chi et al., 2023) and Behavior Transformers (Shafiullah et al., 2022) show that averaging a multimodal action distribution fails. → The explanation for section 2.
• Stop Regressing (Farebrother et al., 2024) finds classification losses train value networks better than regression. → Why switching the loss alone was worth testing.
• Causal Confusion in Imitation Learning (de Haan et al., 2019) describes policies that latch onto the wrong cue. → The image shortcut from Part 4, which tokens don’t remove.

Summary (STAR)