Part 6 · Separating perception from choice: a world model and a planner6편 · 인지와 선택을 분리하기: 월드 모델과 플래너第 6 篇 · 把感知与选择分开：世界模型与规划器

Parts 4 and 5 kept one network responsible for everything: reading the image, reading the words, and flying. The words kept losing to the image. This part tests a different division of labour, stated as a design rule:

The sentence should only choose the target. The image and the state should do the flying.

So the words get exactly one job and no path to the motors. A world model learns where every target is without ever seeing a sentence. A tiny selector turns the sentence into a colour. A planner, which is not learned, flies to the chosen target.

1. The system: perceive, select, plan

1. Perceive (learned, no words). An encoder of the same kind as the BC policy’s turns the image and state into 64 numbers, z. A small head reads out, for each of the four colours, the offset from the drone to that colour’s hover point. The head was trained against the simulator’s true positions; those positions are training labels only, never inputs.
2. Select (learned, 340 parameters). Word embeddings, averaged, then one linear layer to four colours. It is 100% correct on the train and validation sentences. In this task one colour word decides the target, so that is expected, not a result.
3. Filter (rule). Single-frame estimates are noisy (about 0.17 m near the goal), and acting on them raw makes the commands jitter. The filter predicts the offset forward by the last command and blends in each new measurement at 30%.
4. Plan (rule). The cross-entropy method (CEM): sample 64 sequences of 8 commands (1.6 s), score each by where it would leave the drone relative to the goal, keep the best 8, resample around them, four rounds. Fly the first command and plan again 0.2 s later.
5. Stop (rule). The expert’s own rule: Stop when the estimated goal is within 0.12 m and the drone is slower than 0.08 m/s.

The planner is short (full code):

def plan(self, g0, displacement):        # g0: filtered goal offset (2,), body frame
    mean, std = self.mean, np.full((8, 2), 0.3)
    for _ in range(4):
        A = cap(mean + std * rng.standard_normal((64, 8, 2)))   # 64 sequences of 8 commands
        off = g0 + displacement(A)                              # where the goal would be
        cost = (off ** 2).sum(-1).mean(1)                       # stay close to it
        cost += 0.5 * jerk(A) + 1.0 * (A[:, -1] ** 2).sum(-1)   # smooth, arrive slowly
        elite = A[np.argsort(cost)[:8]]
        mean, std = elite.mean(0), elite.std(0) + 1e-3
    return elite[0][0]                                          # fly step 1, re-plan next tick

integrator = lambda A: -np.cumsum(A, axis=1) * 0.2               # variant (A)

The only difference between the two variants is displacement. (A) assumes the drone moves exactly as commanded. (B) starts from z, applies the world model’s learned dynamics z’ = LN(z + f(z, a)) step by step, and reads the head after each step.

All planner settings were tuned with the true target positions on training scenes, before any learned variant was run on validation.

2. What the world model learned

The world model has 574,515 parameters and was trained on CPU for 12 epochs (36 minutes). Its training data was the 100 demonstration layouts plus exploration flights over the same layouts: noisy expert, random walks, and overshoots past the goal. That is 30,448 rows, three times the states the BC policies saw. The loss unrolls the dynamics 4 steps from each start state with the recorded commands (full code):

z = model.encode(rgb[:, 0], state[:, 0])
loss = heads_loss(z, k=0)                                  # state + every colour's offset
for k in range(1, 5):
    z = model.step(z, command[:, k - 1])                   # z' = LN(z + f(z, a))
    target = model.encode(rgb[:, k], state[:, k]).detach() # what the encoder sees at t+k
    loss += 0.9 ** k * (((z - target) ** 2).mean() + heads_loss(z, k))

• A, perception is good enough to plan with. On unseen layouts the estimated goal position is off by a median 0.15–0.26 m, under the 0.4 m goal radius in 68–92% of frames. Training on 100 scenes instead of 20 roughly halved the error. The bottleneck was the number of distinct scenes, not the model.
• B, the learned dynamics are worse than doing nothing clever. Starting from the same perceived position, rolling the learned dynamics forward 2 s gives 0.39 m error on expert flights and 0.57 m on random walks. Simply integrating the commands stays near 0.30 m. In this simulator the low-level controller tracks commands closely, so an integrator is almost perfect dynamics.
• C, and the learned dynamics are biased by direction. Hold one command for 1 s: the model predicts forward and left moves about right, but backward moves twice as far as commanded and right moves at 0.58 of the commanded distance. Most training commands point forward, toward the targets.

3. Results

(A) World model + integrator planner.

• All 10 pairs split the right way (no end-to-end model splits more than 5 correctly; Fisher’s exact test for 10/10 vs 5/10: p ≈ 0.03).
• Success is 15/20 and pair success 6/10, 95% interval 0.31–0.83. Against the best end-to-end model’s 3/10 (0.11–0.60) that difference is not yet beyond chance.
• It reaches the 0.4 m goal region in all 20 flights.
• The five misses all come after it has reached the region: two stops just outside it and three collisions. The same planner fed the true target position succeeds 20/20, so what is left is perception error close to the target.

(B) The same system planning through the learned dynamics scores 0/20: 13 timeouts and 5 collisions. The GIF above shows why. CEM searches for commands that look good according to the model, and panel C showed the model exaggerates some directions. The planner keeps choosing the commands whose effect the model gets most wrong, and the drone jitters. Two things make this worse: the model was trained to unroll 4 steps and is asked for 8, and most of its training commands point forward, while a planner tries every direction.

4. Is this a fair comparison? No, so here is what it does and doesn’t explain

The modular system differs from the BC policies in more than its structure:

Neither the labels nor the states, alone or together, gave an end-to-end policy the modular system’s 10 of 10 correctly split pairs. What’s left is the structure: the words can only choose, and nothing downstream can ignore that choice. I can’t yet separate how much comes from the selector and how much from the rule-based Stop.

Two smaller differences:

• Randomness. The planner’s random seed differs between the two flights of a pair; BC evaluation is deterministic.
• Training budget. The world model trained for 36 minutes, about 4–5 times a Part 4 BC run.

What this part does not show

• One training seed and ten pairs. 6/10 has a 95% interval of 0.31–0.83.
• The selector’s job is trivial here. Two pillars always differ in colour, so one word decides. Relational instructions (“the pillar left of the red one”) would need a real grounding model.
• The integrator works because the simulator is kind. Its controller tracks commands almost perfectly. With wind, drag or a real airframe, the dynamics would have to be learned or identified, and section 3 shows that naively learned dynamics are a liability for a sampling planner.
• Validation only. It was used for screening; final numbers need a fresh test split.

Next

Season 2 ends with a working answer to Part 4’s question, for this two-pillar, one-colour-word task: yes, the words can be made to decide, if they are only allowed to decide. Season 3 turns to engineering. One evaluation contract will run the same policies in Python and C++, so every later change (ONNX, INT8, a C++ runner) is measured against the same numbers.

Appendix

python -m dronevla.world_model train --data data/v0.2 --explore data/explore_v0.2 --out runs/wm_v1
python -m dronevla.world_model eval --run runs/wm_v1 --out reports/wm_v1_prediction.json
python scripts/wm_diagnostics.py --run runs/wm_v1 --out reports/wm_v1_diagnostics.json
python -m dronevla.planner train-selector --data data/v0.2 --out runs/goal_selector
python -m dronevla.evaluate --data data/v0.2 --split val --policy plan-oracle \
    --policy plan:runs/wm_v1:integrator --policy plan:runs/wm_v1:wm \
    --out reports/eval_planner_val.json
python scripts/figures/blog06.py
bash scripts/figures/render_blog06_videos.sh

References: what I took from each

• World Models (Ha and Schmidhuber, 2018) and PlaNet (Hafner et al., 2019) learn a compact latent state and its dynamics from images. → The encoder, latent z and residual dynamics.
• PETS (Chua et al., 2018) plans with CEM through learned dynamics. → The planner. Its use of model ensembles to avoid exploiting model error is exactly what variant (B) lacks.
• When to Trust Your Model (Janner et al., 2019) shows how model error compounds over long rollouts. → Why the planner was given an integrator first, and why (B) fails at an 8-step horizon after 4-step training.
• TD-MPC (Hansen et al., 2022) plans in a latent space learned for control. → A direction for making (B) work, not tried here.
• CLIPort (Shridhar et al., 2021) separates what (language-conditioned semantics) from where (spatial precision). → The same split, in its simplest possible form.

Summary (STAR)