Submersivity — XR-guided underwater garbage detection

Overview

From submarine capture to XR guidance — the full perception and mapping chain in one view.

Open pipeline PDF

Pipeline diagram: RC submarine captures underwater frames, YOLOv8 detects garbage, COLMAP maps detections in Peter Street Basin, and XR goggles display location and direction. — System overview (Fig. 2): design and build the RC submarine, acquire underwater frames, detect garbage with a fine-tuned YOLOv8 model, localize hits on the Peter Street Basin map, and display the nearest target on Vuzix Smart Swim XR goggles.

End-to-end pipeline

01HardwareBuild a submarine that survives the basin

02AcquisitionGet frames off the submarine and into the model

03Computer visionTrain a detector that works in murky water

04GeometryGive every detection a real place in the basin

05InterfacePut the next target inside a pair of swim goggles

Aerial photo and photogrammetry-generated site plan of Peter Street Basin in Toronto with dimensions in metres. — Peter Street Basin, Toronto: aerial view and the photogrammetry-generated site plan used as the basin reference frame. Adapted from the Peter Street Basin Master Plan 2026.

Key numbers

Site: Peter Street Basin, Toronto (~60 m × 30 m)
Platform: ESP-DIVE RC submarine, XIAO ESP32-S3 Sense
Detector: YOLOv8s, single class, 12,177 images
Precision: 92.1% on the combined validation set
Mapping: COLMAP SfM + ArUco, 796 / 823 frames registered
Display: Vuzix Smart Swim XR goggles (Android)

What this is, in plain terms

Peter Street Basin is an outdoor pool of Lake Ontario water in downtown Toronto, about 60 m by 30 m, connected to the inner harbour by a narrow channel under Queens Quay. It collects litter, and the litter sinks where nobody can see it.

This project is the full chain from “nobody knows what is down there” to “swim 4 metres that way and pick it up.” We built a small submarine that carries a camera underwater, trained a detector to recognise garbage in murky water, reconstructed the basin in 3D so every detection has real coordinates, and pushed the nearest target onto a head-up display inside swim goggles.

The result is a human-in-the-loop system: the machine handles perception and geometry, the person handles the hands-in-the-water part.

The five stages

Each stage has its own page with decisions, specs, code locations, and what still needs work.

01Hardware

Build a submarine that survives the basin

A battery-powered, dual-ballast RC submersible carries the camera under the surface and keeps its radio link above it.

Open stage 1

02Acquisition

Get frames off the submarine and into the model

A host-side capture loop pulls the submarine's video stream, timestamps frames, and hands them to inference.

Open stage 2

03Computer vision

Train a detector that works in murky water

YOLOv8s, fine-tuned as a single-class “garbage” detector on 12,177 terrestrial and underwater images, reaching 92.1% precision.

Open stage 3

04Geometry

Give every detection a real place in the basin

COLMAP structure-from-motion plus ArUco ground-control points turn camera poses into basin-aligned coordinates for each detection.

Open stage 4

05Interface

Put the next target inside a pair of swim goggles

A custom Android application on Vuzix Smart Swim goggles shows the nearest target's class, range, heading cue, and a reduced map of the basin.

Open stage 5

Results at a glance

Field deployment figures from Peter Street Basin — detections, basin map, and the XR head-up display in operation.

Two underwater frames with bounding boxes: packaging debris in turbid water, and a small distant detection near the basin floor. — Detections from the Peter Street Basin deployment: packaging/plastic debris in turbid light (left) and a small, distant, low-contrast object near the basin floor (right).

Plot of the perimeter survey path around Peter Street Basin with accumulated garbage detections in basin coordinates. — A representative survey trajectory in basin coordinates with accumulated detections. The path is derived from the COLMAP sparse reconstruction of 796 registered frames, starting at the southeast corner; this run logged 130 detector hits.

Head-up display through the Vuzix Smart Swim lens: a compass arrow, a 4.1 m range readout, the class label OTHER, and a small basin map. — The XR head-up display in operation: heading arrow, 4.1 m range readout, target class, and a miniaturised Peter Street Basin map with registered detections — all inside the small monocular display window.

Paper and credits

XR-Guided Garbage Detection with an RC Submarine for Peter Street Basin Cleanup

Chris McGale, Michel A. Herrera Viyella, Chan Park, Daniel Bros, Alexander Vicol, Steve Mann (corresponding author)

Department of Electrical and Computer Engineering (ECE), University of Toronto
Department of Mechanical and Industrial Engineering (MIE), University of Toronto

Presented at the Mersivity / WaterHCI Symposium 2026. Carried out with the Friends of Peter Street Basin.