Part 4 · The policy that did not listen4편 · 말을 듣지 않은 정책第 4 篇 · 不听指令的策略

Three parts of setup lead to this one: a simulator, a scored task, and 100 pairs of flights in which the same image comes with two different sentences. Now the first policy. I expected it to be bad at flying. I didn’t expect it to fly toward a pillar and stop there while having no idea which one.

1. The model: small on purpose

Three inputs, one output, no pretrained parts:

• Image: four strided convolutions shrink 96 x 128 x 3 to a 6 x 8 x 64 map, then to 128 numbers.
• Sentence: each word gets a learned 32-number vector, and the vectors are averaged. With only 15 distinct words in the dataset, this is the simplest text encoder that can still tell “red” from “blue”.
• State: the 11 numbers from Part 2’s contract, normalised with training-set statistics.

They are joined into 192 numbers, and a small MLP outputs four velocities (squashed by tanh and scaled to the limits) plus one Stop logit. Total: 480,901 parameters. It runs in 2.7 ms per decision on the CPU (median), far inside the 200 ms budget (simplified below; model code):

def forward(self, rgb_u8, proprio, text_vec):
    """rgb_u8 (B, 96, 128, 3) uint8; proprio (B, 11) normalised; text_vec (B, 32)."""
    x = rgb_u8.permute(0, 3, 1, 2).float().div(255.0).sub(0.5)       # to (B, 3, 96, 128)
    feats = torch.cat([self.img_fc(self.cnn(x)),                      # (B, 128)
                       text_vec,                                      # (B, 32)
                       self.prop(proprio)], dim=1)                    # (B, 32)
    out = self.head(feats)                                            # (B, 5)
    return torch.tanh(out[:, :4]), out[:, 4]                          # velocities, Stop logit

2. Training: behaviour cloning on CPU

Behaviour cloning means copying the expert: for every recorded frame, predict the action the expert took. Two losses are added together: a Huber loss on the four velocities, and a binary cross-entropy on Stop, weighted up because only 6.7% of rows are Stop (full code):

huber = nn.SmoothL1Loss(beta=0.1)
bce = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(13.9))   # from training rows only

def train_step(idx):
    rgb, prop, ids, act, stp = prep.batch(train, idx)       # act is scaled by the limits
    m, logit = model(rgb, prop, model.encode_text(ids))
    loss = huber(m, act) + bce(logit, stp)
    opt.zero_grad()
    loss.backward()
    opt.step()

Settings: Adam, learning rate 10⁻³, batch 16, 9,952 training rows. Each seed takes 7–8.5 minutes on the laptop CPU.

The evaluation rules, fixed before running

An earlier round of experiments taught me these rules by breaking them, so this time they were written down first:

1. Decisions use validation only. The 10 test pairs aren’t touched, and no number in this post comes from them. (Those test seeds were looked at during the earlier, discarded round, so final numbers will come from a fresh test split.)
2. Three training seeds. With 10 validation pairs, one seed’s luck can look like an effect.
3. One stopping rule for every run: train 20 epochs and keep the last. Picking the “best” epoch by validation loss favoured whichever epoch the Stop term happened to like.
4. “Which target did it go for” = the one it came closest to. Scoring by the final position misreads flights that end outside the room.

The Stop threshold is the only thing tuned, on validation rows.

3. Results: it flies, it doesn’t listen

The expert splits all 10 pairs. The policy split 0, 2 and 0 of them: for almost every layout it flew to the same pillar whichever sentence it was given.

The middle panel follows from that. Within a pair, each sentence names a different pillar. A policy that flies to the same pillar for both is right for exactly one of the two sentences, so any word-blind policy scores exactly 10 of 20, as Part 2 predicted. Seeds 0 and 2 are that case. Seed 1’s two split pairs land on 10 too: it split one pair the right way round and one the wrong way round. And no seed completed both sentences of any pair: pair success was 0/10 for all three seeds.

The top view shows it directly:

Its most common failures are stopping away from both hover points and stopping at the wrong pillar (appendix A). It does fly toward the pillars and it does stop. What it lacks is the one decision the sentence exists for.

4. Why: the words matter for one frame

My hypothesis was that the image makes the words unnecessary, and I tested it directly. From every recorded training state, I asked each trained policy for its action twice: once with the episode’s own sentence, and once with its pair partner’s. Then I compared how much the sideways command changed with how much the expert’s changes (measurement script).

A. On the identical first frame, swapping the sentence moves the expert’s sideways command by 0.200 m/s. It moves the policy’s by 0.004–0.007 m/s, 30 to 50 times less, for all three seeds. At later steps the gap is just as small. The policy has nearly disconnected the words. (The expert’s later-step bars are computed actions: from each logged state, what the expert would do for the other sentence. Those states never occur with the other sentence in the data, which is why its gap grows later in the episode.)

B. On the training set, the policy matches the expert to about 0.01 m/s on every step except the first, where its error is 0.10 m/s. That’s half of the 0.2 m/s gap between the two sentences’ labels, consistent with predicting the average of the two.

The mechanism is simple:

• On the first frame, both sentences share one image, and only the words say left or right. That frame is 2% of the training rows.
• On every later frame, the drone has already started moving to one side (its heading never changes; only its position does). The image itself shows which way it is going, so the image predicts the label. That’s 98% of the rows.
• Behaviour cloning minimises the average loss over all rows. The cheapest solution is to read the image and average over the 2% it can’t explain. In closed loop, the first move is then made without the words, and every frame after it confirms whichever way the drone happened to lean.

That is why the counterfactual pairs exist. Without them, this policy’s success rate could have looked like a language effect; with them, the word-blindness is visible.

What this part does not show

• 10 validation pairs is a small sample; a policy that listened only a little could hide in it.
• One small architecture and one way of encoding words. A bigger text pathway could behave differently; that is a question for the next season, not a result here.
• Simulation only, two pillars, two sentence templates.

Next

Season 2 asks whether the shortcut can be removed. One idea is to give the words a reason to matter on every frame. Another is to split the problem in two: first perceive where every target is, then choose the one the sentence names.

Appendix

for s in 0 1 2; do
  python -m dronevla.train --data data/v0.2 --epochs 20 --select last --seed $s --out runs/bc_v0.2_s$s
done
python -m dronevla.evaluate --data data/v0.2 --split val \
    --policy runs/bc_v0.2_s0 --policy runs/bc_v0.2_s1 --policy runs/bc_v0.2_s2 \
    --out reports/eval_v0.2_val.json
python scripts/language_use.py reports/eval_v0.2_val.json
python scripts/instruction_sensitivity.py --data data/v0.2 --split train \
    --runs runs/bc_v0.2_s0 runs/bc_v0.2_s1 runs/bc_v0.2_s2 \
    --out reports/instruction_sensitivity_v0.2_train.json
bash scripts/figures/render_blog04_videos.sh
python scripts/figures/blog02_04.py

References: what I took from each

• RT-2 (Google DeepMind) represents actions as text tokens inside a vision-language model. → That’s one end of the action-output spectrum; this policy is the opposite end.
• SmolVLA (Hugging Face) pairs a small VLM with a flow-matching action expert that outputs action chunks. → The target for Season 4, and why the (5,) contract will need a mapping step.
• A Recipe for Training Neural Networks (Karpathy) says to become one with the data and verify the evaluation before trusting a model. → Pair success and the swap test are how this project checks the policy instead of trusting the loss.

Summary (STAR)