Vol.11
Keeping the main thread alive — a model load that froze for 10 minutes, 120 fps rendering, and a HUD in the recording
A model load that froze for 10+ minutes with no error
Back when the wand keypoint model loaded in TF.js Layers format, model construction sometimes ran past 10 minutes without finishing or throwing (useWandKeypointDetector’s comment says “Layers-era hangs could exceed 10 minutes with no error”). Like the node_index bug in Vol.4, TF.js graph construction can answer broken input with an infinite loop rather than an exception.
Three layers of defense. Load in a Web Worker so graph construction cannot freeze the main thread. Add a 90-second timeout that shows “wand model load timed out (90s)”. Fall back to main-thread loading only if the Worker fails. Alongside that came uint8 quantization, conversion to graph-model format, and a load-progress display.
1// Prefer Worker so graph construction does not freeze the page.2// Cap wait — Layers-era hangs could exceed 10 minutes with no error.3const LOAD_TIMEOUT_MS = 90_000;4 5try {6 const client = new WandModelWorkerClient();7 await withTimeout(client.load(modelPath, metadataPath, trackProgress));8 finishLoaded(Math.round(performance.now() - loadStarted), lastProgress, 'Wand model loaded (worker)');9 return;10} catch (workerError) {11 console.warn('[wand] worker load failed, falling back to main thread', workerError);12}13// ↓ メインスレッドで WandKeypointDetector を読み込む（同じタイムアウト付き）Do not feed Hands at the render rate of 120 fps
Call Studio’s render loop can exceed 120 fps depending on the setup (per the hook’s comment). Sending to Hands every frame drags rendering down, and MediaPipe only returns around 15 fps anyway, so most sends are wasted. Hands sends are throttled to HANDS_SEND_INTERVAL_MS = 70 (~14 fps), and frames in between are interpolated with damp (Vol.6).
Before heavy synchronous work, yieldToMain() hands rendering and input back to the browser. In a visible tab it chains setTimeout(0) inside requestAnimationFrame; in background tabs or Workers, where rAF can stall, it uses setTimeout(0) alone.
Keep the perf HUD out of the loop — and out of the recording
The Performance HUD shows under a DEV build, a ?perf query, or a localStorage toggle. It never joins the app’s rAF loop and only snapshots every 500 ms. Render FPS and hand-tracking FPS are separate counters; tracking FPS counts each parseTrackedHandsFromResults call — so we can tell whether rendering or inference is the bottleneck.
Call Studio has another constraint: OBS captures this screen, so anything drawn over the camera ends up in the recording. During the Tasks Vision migration on 2026-09-27, recognition status and occlusion / seal debug moved off the camera overlay into a side panel (getDebugHud). The same day, the landmark debug HUD and send-path probes added for the migration investigation were removed, keeping only the HUD showing detectForVideo time.
What building it taught us
- Always put a timeout on model loading. TF.js can respond to a broken graph with an infinite loop instead of an exception.
- Decouple inference send rate from render frame rate. On high-refresh displays, sending every frame wastes most of it.
- On a screen that gets captured, where debug UI lives is a design decision. Everything drawn on the camera overlay ends up in the recording.
- Remove HUDs and probes added for an investigation once it is solved. Keep only the numbers you will keep watching (detect time).