All 21 landmarks collapsed to one point on Chrome — moving from legacy Hands to Tasks Vision

What happened

anime-ar-stock derives palm center, fingertips, and hand orientation from the 21 points and uses them for (1) the crop region fed to the sign CNN and (2) the anchor that keeps Rasengan or Chidori attached to the hand. When the landmarks collapse to one point, the crop has zero area and the anchor sticks to a single spot. Hands keeps reporting “hand detected”, so nothing throws.

The commit message records it as: “Legacy Hands 0.4 collapses all 21 landmarks to one point in Chrome 153; HandLandmarker 0.10.32 does not.”

Isolating it: we suspected how frames were fed

The first suspect was not MediaPipe but how we fed it frames. The header comment of lib/nativeHandsVideo.ts still holds the hypothesis: “Chrome WebGL texImage2D(<video>) samples the CSS compositor (object-fit, layout box, DPR), so landmarks do not match videoWidth×videoHeight.” We tried, in order:

• Pin the display <video>’s width / height attributes and CSS size to the real resolution, with object-fit: fill and transform: none (pinHandsVideoIntrinsicSize).
• clone() the track into an invisible off-screen <video> (opacity 0.04, z-index -1) and feed that one to Hands.
• On Chrome, send a canvas with the decoded frame drawn into it instead of the <video> (2D canvas drawImage reads the decoded frame, not the compositor).
• Put plain HTML pages — mediapipe-hands-raw.html and mediapipe-tasks-hand-raw.html under public/debug/ — that bypass the React app entirely, and feed legacy Hands and HandLandmarker the same camera to compare raw coordinates.

The comparison pages settled it. Even with no app code in the path, only legacy Hands collapsed while HandLandmarker was fine — so we could drop the “fix how we feed frames” theory. Workarounds 1–3 above only ever told us that they did not help; the root cause is still unidentified.

Migration options

We migrated in two steps. On 2026-09-27 Call Studio (the admin capture page) moved to Tasks Vision first, and public pages moved to the same adapter later that day. The default is HAND_TRACKING_BACKEND = 'tasks'; appending ?hands=legacy rolls back without a rebuild.

Three differences the adapter absorbs

1async send(input) {2  const image = input.image;3  if (!(image instanceof HTMLVideoElement) || !landmarker) return;4  // ① まだフレームが無い video を渡さない5  if (image.readyState < HTMLMediaElement.HAVE_CURRENT_DATA || image.videoWidth <= 0) return;6 7  // ② detectForVideo は timestamp が単調増加でないと例外になる8  let timestamp = performance.now();9  if (timestamp <= lastTimestamp) timestamp = lastTimestamp + 1;10  lastTimestamp = timestamp;11 12  const result = landmarker.detectForVideo(image, timestamp);13 14  // ③ HandLandmarker には selfieMode が無いので、x の反転を自前でやる15  const flipX = options.selfieMode === true;16  const multiHandLandmarks = copyAndMaybeFlipX(result.landmarks ?? [], flipX);17 18  // 旧 Hands と同じ形で返す。Left/Right はここでは決めない19  onResults?.({ multiHandLandmarks });20}

Forget ③ and, on selfie-view experiences only, the effect moves mirror-opposite to the hand. The handedness HandLandmarker returns is shown in the debug HUD and otherwise ignored. Side is still decided inside parseTrackedHandsFromResults from the x of the palm center (mean of landmarks 0, 5, 9, 13, 17): one hand, x < 0.5 is Right; two hands, sort by x and screen-left is Right — because the selfie view is mirrored, the user’s right hand appears on screen-left.

Re-reading the code turned up a caveat. The adapter only passes numHands to HandLandmarker. The modelComplexity (0 on touch devices, 1 elsewhere) and minDetectionConfidence (0.55 / 0.7) that each hook sets via setOptions are not used on the Tasks Vision path — per-device tuning we thought was active has been a no-op since the migration.