Classifying the twelve hand signs with a CNN — from RandomForest to a 6,267-image CNN

How the model evolved (from training logs)

These come from the training logs in training/artifacts. Every accuracy figure is on a validation split of a random 20% held out from the training data.

• RandomForest (2026-05-18): 132-dim landmark features. 88.6% at 1,268 samples, 97.5% at 2,400.
• MLP (2026-05-20 to 24): 64px hand crops in sklearn. Around 86–88% from 2,401 to 3,433 samples.
• CNN (2026-06-28): transfer learning on a frozen ImageNet-pretrained MobileNetV2 base, 128px input. 99.1% on the same 3,433 samples.
• CNN (2026-09-22, current): 99.4% on 6,267 samples. Per-class counts are skewed, from 253 for Hitsuji (Ram) to 771 for Ushi (Ox).

On the same 3,433 samples, MLP vs. CNN went from 87.9% to 99.1% — the difference spatial structure makes is unambiguous.

We do not read the 99%-range as real-world accuracy, though. The validation split is drawn at random from consecutive frames of the same capture sessions, so near-duplicates of training images land in validation. Accuracy for a first-time user in an unfamiliar room’s lighting should be lower.

Inference rate: every 125 ms, paused during a jutsu

A sign only means something after being held for a few hundred milliseconds, so there is no need to infer every frame. tick() enforces HAND_SIGN_INFERENCE_INTERVAL_MS = 125 (~8 fps) internally, and callers just call it every frame. Labels under 0.45 softmax confidence are dropped.

The other saving: no inference while a jutsu is playing. Recognizing a sign while Rasengan spins on your palm is useless, and the CNN and the VFX (Three.js with bloom) fight over the same main thread and GPU. Session hooks pass pauseInference into tick(), and Call Studio got the same “pause the CNN while a jutsu plays” change on 2026-09-23.

1export const HAND_SIGN_INFERENCE_INTERVAL_MS = 125; // ~8 fps2export const HAND_SIGN_CONFIDENCE_THRESHOLD = 0.45;3export const HAND_SIGN_MODEL_TARGET_BYTES = 1 * 1024 * 1024; // 目標 1MB4export const HAND_SIGN_MODEL_MAX_BYTES = 5 * 1024 * 1024;    // 警告 5MB5 6// useRasenganSession: 術の発動中は推論を止める7const pauseInference = trigger.getPhase() === 'rasenganActive';8classifier.tick(sourceVideo, inferenceLandmarks, now, pauseInference, false);

Where we got stuck: the tab froze with no error at all

In August 2026, loading a CNN retrained for the fox sign (chainsaw_kon) made the tab completely unresponsive — no error, no console output, just frozen.

The cause was in the Keras → TF.js conversion. The training script exports via legacy Keras (TF_USE_LEGACY_KERAS=1), and that path emits the frozen MobileNetV2 base as a nested Functional model. To make it loadable in TF.js we patch the graph by hoisting nested layers to the top level — but the patch rewrote only the referenced layer name and left the node_index (often 1) in place. A hoisted layer is called only once (node 0 only), so the reference pointed at a node that would never exist, and TF.js’s topological model construction looped forever trying to resolve it.

1for (const connection of node) {2  if (Array.isArray(connection) && connection[0] === fromName) {3    connection[0] = toName;4    // 引き上げたレイヤーは node 0 しか持たない。古い index を残すと5    // 存在しないノードを指し、TF.js のモデル構築がエラーを出さずに止まる6    if (resetNodeIndex && connection.length >= 2) {7      connection[1] = 0;8    }9  }10}

The fix went into both the browser-side patch (tfjsModelLoad.ts) and the Python export step (train_hand_sign_model.py) — fixing only one would bring it back with the next retrained model. The existing twelve-sign and wand models were exported from Keras 3, never took this path, and were unaffected.