Vol.10
Depth from a person mask — never start Selfie Segmentation at the same time as Hands
Where it is used
- Shiki Fujin (useShikifujinSession): the person’s silhouette occludes the death god (shinigami.glb, 6.3 MB) so it appears to stand behind them.
- Hand Occlusion (vision/handOcclusionMask.ts): in Call Studio’s Rasengan and others, hides the VFX where the hand comes in front of it.
Jutsu that do not need person compositing never load the Segmentation script. When it does load, it checks window.SelfieSegmentation and script[data-mediapipe-selfie-segmentation] to avoid double insertion.
Where we got stuck: simultaneous init makes WASM Modules collide
Initializing MediaPipe’s legacy solutions (Hands / Pose / Selfie Segmentation) simultaneously makes their internal WASM Modules collide. Shiki Fujin uses all three — Hands for signs, Pose for where the person is, Segmentation for occlusion — so it runs a three-stage chain: Pose waits for handsReady, Segmentation waits for poseReady. The code comment says it plainly: “this only serializes load order; there is no functional dependency.” The order itself means nothing; the point is never running them at once.
Hand Occlusion built this lesson in from the start: until tick()’s third argument handsSettled is true (Hands finished loading, or is disabled), it does not even begin loading the script or initializing the model.
1/**2 * 重要: MediaPipe は Hands と Selfie Segmentation を同時に初期化すると内部 WASM Module が3 * 衝突する（useShikifujinSession.ts が poseReady を待ってから Segmentation を起動している4 * のと同じ理由）。そのため tick() の第3引数 handsSettled が true になるまでは5 * スクリプト読み込み/モデル初期化そのものを開始しない。6 */Reusing the full-body mask for hand occlusion
- Pros
- Could cut cleanly down to finger outlines.
- Cons
- Adds cost to find, train, and ship a model. A third inference on top of Hands and Segmentation.
- Pros
- No new infrastructure. Enough for the requirement: the VFX should look hidden when the hand is in front.
- Cons
- Every person pixel is in the mask, so on its own it cannot tell hand from body in depth.
Why: Depth (what is in front of the VFX) comes from a depth proxy built from landmarks; the mask is only a 2D gate saying “a person is here”. With roles split, the full-body mask’s coarseness stops mattering.
Mask processing varies by quality. Against a 60 fps render loop, low / standard send every 4th frame and high / ultra every 2nd (15–30 fps equivalent). Results are downscaled into a 128–256px canvas, feathered with a 3–5px blur, and blended over the previous result with globalAlpha (0.5–0.35) for temporal smoothing. There is no pixel readback (getImageData) at all, so latency to OBS does not grow.
What building it taught us
- Initialize MediaPipe’s legacy solutions one at a time; simultaneous start collides their WASM Modules. It recurs with every new combination, so design in a “wait for the previous one” argument from the start.
- Do not make the mask carry depth. Landmarks for depth, the mask as a 2D “person here” gate — split that way, the full-body mask was enough.
- When video goes out to OBS or similar, avoid pixel readback like getImageData. Even mask smoothing can be done with canvas compositing alone.