A custom model for the wand tip — from 59 to 397 images, and the 0.08 threshold that rejected real wands

Data volume vs. error (from training logs)

Outputs are presence (is there a wand) plus normalized tip and grip coordinates. Error is mean L2 distance in normalized coordinates on validation data — 0.1 means off by a tenth of the image width.

• 2026-07-20: 59 images. An MLP (64px) scored 0.652 mean error — barely usable — while a CNN (224px) on the same 59 scored 0.190–0.210.
• 2026-07-20: 0.151 at 123 images.
• 2026-07-26: 0.085–0.090 at 250 images.
• 2026-09-13: presence output added. 0.101 at 320 images and 0.101 at 397 (current). Presence accuracy 100%.

Going from 59 to 250 images more than halved the error, but 250 → 397 made it slightly worse. We added the presence output and “no wand in frame” images at the same time, which we suspect is the cause — but we have not isolated it.

Also, in every version the tip error is larger than the grip error (currently tip 0.129 / grip 0.073). The grip sits in the hand and is stable; the tip is thin, blends into backgrounds, and moves fastest when the wand swings. The tip is exactly where light or spells should come from — the point we need most is the hardest.

The bug: a threshold stricter than the model’s error

Predictions are discarded as “not a wand” unless the tip–grip distance (stick length) clears a threshold. While that threshold was 0.08, production users saw “杖が見つかりません” (wand not found) even with the wand in frame.

The model then was trained on 250 images with mean keypoint error around 0.09. So against a model whose tip and grip could each be off by 0.09, we were discarding anything with less than 0.08 between the two points — a check finer than the model’s own precision. Tilting the wand away or stepping back shortens it on screen and makes correct detections even likelier to fall under. On 2026-08-09 we lowered it to 0.04.

1export const WAND_INFERENCE_INTERVAL_MS = 125;2 3/** Minimum tip–grip length (normalized) to treat as a wand hit. */4export const WAND_MIN_STICK_LENGTH = 0.04;   // 0.08 → 0.04（2026-08-09）5 6/**7 * Models trained before the presence output always report confidence=1,8 * so this only gates once a presence-aware model is deployed.9 */10export const WAND_MIN_CONFIDENCE = 0.5;

A stick-length gate was always an indirect way to judge “no wand in frame”, since a regression model emits some coordinates even with no wand. Later, to fix false positives when no wand was present, we added the presence output. The Worker handles both: 5-output models (presence + tip xy + grip xy) use presence; older 4-output models are treated as confidence = 1.

The inference pipeline

Input is 224px. A naive resize distorts the aspect ratio and with it the tip position of a long, thin wand, so we letterbox (scale with padding) and map outputs back to source coordinates with unletterboxPoint(). Inference runs in a Worker; the main thread only receives results (Vol.11).

1const letterboxed = letterboxResize(pixels, inputSize, box);   // 224px、余白付き2const values = await model.predict(preprocessed).data();3// 5-output models (presence + tip/grip) started shipping after the4// "no wand in frame" false-positive fix; 4-output models predate it.5if (values.length >= 5) {6  const confidence = clamp01(values[0]);7  const tip = unletterboxPoint(values[1], values[2], inputSize, box);8  const grip = unletterboxPoint(values[3], values[4], inputSize, box);9  post({ type: 'prediction', id: data.id, tip, grip, confidence });10}

Like the twelve-sign CNN, this pipeline trains locally, ships as static files (a uint8-quantized graph-model, about 2.5 MB), and infers in the browser. Production servers neither train nor infer.