Part 3 · Same scene, different words: a counterfactual dataset3편 · 같은 장면, 다른 말: counterfactual 데이터셋 만들기第 3 篇 · 同一场景，不同指令：反事实数据集

Part 2 fixed the task and the scorer. Now the policy needs examples to learn from. The question of this series sets one hard requirement on the data: listening has to be necessary. If one image could only ever lead to one target, a policy could ignore the words and still match every label. So every example in this dataset comes as a pair.

1. How one pair is made

1. A seed picks a layout: two pillars of different colours, box or cylinder, somewhere in front of the drone. Layouts that would make a bad test are thrown away and resampled. Examples: hover points closer than 1.5 m, or a straight path to one pillar that brushes the other.
2. One start snapshot is shared by both flights. Everything random (the layout, the position-noise stream) comes from the seed, never from which target is named. That’s why the two first frames are byte-identical.
3. The scripted expert flies both targets through the same action adapter, controller and physics a policy would use. It is an oracle: it reads true positions, which a policy never gets. That’s fine for a teacher, as long as the dataset says so (it does).
4. The recorder writes every 0.2 s: the PNG image, the 11-number state, the action actually applied, the Stop label, and, in separate priv_* columns that policies never read, the true position for later analysis.

The sentences come from two human-written templates, with no language model involved: “Go to the {colour} {shape} and stop.” and “Approach the {colour} {shape}, then hold position.” Both flights of a pair use the same template, so the only difference between them is the colour and shape words.

2. What is in v0.2

Splits are by layout, not by frame. If frames from one flight landed in both training and validation, a model could memorise the scene and look good for the wrong reason. Here, both flights of a pair and every frame of a flight stay in one split, and no layout appears twice. The validation and test seeds are the same as in the first, smaller version of the dataset, so results stay comparable across versions. One caveat: those test seeds were looked at during an earlier, discarded round of experiments, so final test numbers will come from a fresh test split.

Two label properties shape everything in Part 4:

• Stop is rare. Only 6.7% of action rows say “stop”. The training loss weights those rows up, with the weight computed from training data only.
• Most of the time the expert flies at full speed. 75% of commands are at the 0.5 m/s cap. The interesting part of each label is its direction, and at the very start that direction depends only on the sentence.

The colour-by-side table is reported, not enforced: green goals were on the left 42 times and on the right 26. The splits are small, so a balanced design would have to come from the generator, not from filtering afterwards.

3. Proving the files are what they claim

A dataset that changes silently invalidates every result built on it. So the recorder writes a manifest: the code commit, the simulator commit, the full task configuration and its hash, the camera parameters, a SHA-256 for every non-image file, and one hash over all 12,198 images. Re-running the command on the same machine reproduces the images byte for byte.

Then a validator reads everything back from disk and runs 14 checks: files present and hashes matching, images decoding to 96 x 128 x 3, every pair complete and identical at the start, splits disjoint, every sentence naming its goal, actions aligned with observations. Here is the pair check, slightly simplified (full code):

for pid, eps in by_pair.items():
    a, b = sorted(eps, key=lambda e: e["goal_index"])           # goal 0 and goal 1
    for key in ("split", "layout_id", "snapshot_seed", "first_rgb_sha256", "instruction_family"):
        if a[key] != b[key]:
            pair_problems.append((pid, f"{key} differs"))       # same start, same first image
    if a["instruction"] == b["instruction"]:
        pair_problems.append((pid, "same instruction for both goals"))
    if load_steps(root, a["episode_id"])[0]["proprio"] != load_steps(root, b["episode_id"])[0]["proprio"]:
        pair_problems.append((pid, "first proprio differs"))    # same state, too
check("pairs_complete_and_identical_at_start", not pair_problems, ...)

4. Two bugs that only reading the output caught

Both happened while I was building the first version of the generator. Each is now covered by a test.

A good layout was rejected: the visibility check rendered from inside the drone. One candidate layout was thrown away because “a target covers 0 pixels at the start”. The real first frame showed it clearly, at 346 pixels. Hypothesis: the check and the recording render from different places. Confirmation: the check rendered from the nominal start position. Position noise leaves the hovering drone a few centimetres away from there, so the check’s camera sat inside the real drone’s arm and propeller, which filled the whole view. Fix: render the check from the pose of the actual first observation. The layout is now test pair 5, with both targets clearly visible.

The teacher was “speeding” 37% of the time. The expert commanded exactly 0.5 m/s. Stored as float32, that became 0.5000001 m/s, so the adapter flagged a cap on 37% of the expert’s actions. Numerically harmless, but it would have reported a teacher that never exceeds the limit as clipping more than a third of the time. The expert now commands 0.5 x (1 − 10⁻⁵): zero flags.

What this dataset can’t support

• Two pillars in an empty room, yaw fixed, flat motion. It says nothing about open-vocabulary language or real scenes.
• 10 validation and 10 test layouts are few, so success rates will carry wide intervals.
• The two pillars always differ in colour, so colour alone always decides the target.

Next

Part 4 trains a policy on these 100 pairs, and it does not listen.

Appendix

python -m dronevla.record --out data/v0.2 --pairs 100 10 10 --version 0.2.0    # ~9 min on CPU
python scripts/inspect_episode.py --dataset data/v0.2 --episode val-004-g1      # every frame + action arrow
python scripts/figures/blog02_04.py

References: what I took from each

• Datasheets for Datasets (Gebru et al.) proposes a standard record of a dataset’s purpose, composition and collection. → The recorder writes a datasheet with every v0.2 number in this post computed from the files.
• LeRobotDataset stores episodes as Parquet plus media with metadata. → I kept a simple Parquet + PNG layout now and will convert when SmolVLA needs it.

Summary (STAR)