Vol.1
We started with server inference — how anime-ar-stock moved fully into the browser
Browser — public AR experience
These four stages share one browser. TF.js runs the inference.
Firebase Hosting
Hosting itself does not infer. It delivers the model; the browser above runs it.
Cloud Run
There is no Camera → Cloud Run arrow. The public AR loop does not call this service.
What this article concludes
anime-ar-stock did not start as a browser-only app. The May 2026 prototype cropped the hand out of the camera frame, sent that image to the server, and classified the sign in Python (an sklearn MLP) behind Express.
Used as AR, what showed up was not the model’s inference speed by itself, but the per-call round trip: crop → encode → POST → infer in Python → JSON back. That round trip sat on the gap between forming a sign and seeing the jutsu. We have not measured the split — encode vs HTTP vs Python.
That is why, on 2026-06-28, we moved the Keras CNN to TF.js. The public experience now stays in the browser: camera → MediaPipe hand detection → TF.js sign recognition → Three.js VFX. Firebase Hosting serves HTML, JS, model.json, and assets. Hosting itself does not run inference. The visitor’s browser does, on the model Hosting delivered.
The architecture we actually went through
This is the order recorded in the commit history and the training logs under training/artifacts. Accuracy belongs in Vol.4. Here, the question is where inference ran.
- 2026-05-18: RandomForest on 132-dimensional features built from MediaPipe landmarks (1,268 → 2,400 samples). Artifacts: training/artifacts/rf-20260518T*.json.
- 2026-05-20 to 24: sklearn MLP on 64px hand crops (imageSize is 64 in mlp-20260520T*.json; the last train in that series is 2026-05-24). The browser POSTed imageBase64 to /api/predict-hand-sign-image, and Express forwarded it to a local Python service.
- 2026-06-28: a 128px-input Keras CNN converted to TF.js and inferred on device (2665e95, which also added the Firebase deploy pipeline). That day’s log, cnn-20260628T083326Z.json, has imageSize 128.
- Today: the sklearn + Python API path is still in the tree. isLocalDevBackendInferenceEnabled() returns true only when import.meta.env.DEV, so production builds never take it. Production metadata.json is backend: "tfjs-cnn", imageSize: 128.
What went wrong with server inference
A sign only means something after it is held for a few hundred milliseconds, so we already treated a few classifications per second as enough (the interval constant is Vol.4). What hurt in use was that each of those calls carried a round trip: crop, encode, POST, Python infers, JSON comes back. Network jitter went straight into the time from forming a sign to the screen reacting.
The other mismatch was how the page is published. Public play pages need no login (we removed the login requirement on 2026-09-06, 95aff58). A free page anyone can open, with inference requests growing in proportion to opens, sat badly with server-side inference cost. Crops of the camera reaching the server also add something we have to explain.
How many milliseconds of that round trip made the experience worse is not in the logs. What the code records is only the call interval. The image path on 2026-05-21 (4014229) used INFERENCE_INTERVAL_MS = 150. Today HAND_SIGN_INFERENCE_INTERVAL_MS = 125, and the comment targets about 8 fps. Both are throttles, not the time one call takes. Rows marked “not measured” below are left that way.
| Item | Server inference (May 2026) | Browser TF.js (public experience now) |
|---|---|---|
| Input | 64px crop (MLP imageSize) | 128px crop (InputLayer in model.json, and metadata) |
| Where it runs | Express → Python (sklearn MLP) | TF.js in the browser |
| Network round trip | Yes. POST the hand image | Not in the public experience loop |
| Call interval | 150 ms (the throttle then). Achieved fps was not measured | 125 ms (current throttle; target about 8 fps). Time per call was not measured |
| Where the model lives | joblib on the server (training/artifacts) | Hosting serves about 2.4 MB; the browser loads it |
| Who pays for inference | The server (per request) | The visitor’s device |
| Camera image leaves the device | Yes (imageBase64) | Not in the public loop. Only the DEV sklearn path still does |
| Latency split | Not measured (encode, HTTP, Python inference, sign-to-HUD) | Not measured (CNN alone, crop, sign-to-HUD). The public loop has no HTTP / Python hop |
The options we weighed
- Pros
- Use sklearn from Python as-is. Swapping the model means dropping a new joblib on the server.
- Cons
- A round trip per inference. Cost scales with visitors. Crops of the camera feed reach the server.
- Pros
- No per-inference round trip. Compute stays on the visitor’s device. Hosting only delivers files; TF.js in the browser runs inference.
- Cons
- Every visitor downloads the model. We absorb device spread (GPU, heat) ourselves. Keras → TF.js conversion has traps (Vol.4).
- Pros
- Full access to camera, inference, and audio APIs.
- Cons
- Adds an install step and loses the “open a link and try it” entry point. We never built this.
Why: MediaPipe was already in the browser, so moving the CNN there closes camera-to-render on one device. We took B on 2026-06-28. The round trip was what showed up in use; Cloud Run cost and “no Python in the production container” were part of the same move. We never split how much each one weighed.
The current setup
firebase.json is the boundary between the public experience and everything else. frontend/dist is served as static files from Firebase Hosting. Only /api/**, /admin, and /admin/** are rewritten to Cloud Run (service anime-ar-engine in asia-northeast1). /models/** is Cache-Control: public, max-age=3600.
Camera → hand detection → sign recognition → VFX does not go through that rewrite. Hosting delivers HTML, JS, model.json, and assets. The Hosting process does not infer. TF.js in the browser that loaded model.json does.
What production Cloud Run actually serves is Stripe checkout and the GA4 admin dashboard. The Express routes that save hand crops and wand annotations live in the same process, but with NODE_ENV=production they return 403 unless ENABLE_TRAINING_API is set. Production wand annotations do not go through Cloud Run: the admin UI writes Firebase Storage and Firestore (usesFirebaseWandAnnotations() is true in PROD).
One frame of the public experience stays in the browser, in this order. Runtime differences inside Hands are Vol.3. The CNN’s 125 ms throttle and the pause during a jutsu are Vol.4. This is only the call order.
1hands.onResults((results) => {2 const trackedHands = parseTrackedHandsFromResults(results);3 classifier.tick(video, landmarks, now, pauseInference);4 layer.renderFrame(anchor, now);5});What going browser-only costs
Inference requests from the public experience to the server are gone. The load moved to the visitor’s device: compute, the model download, GPU/CPU spread, heat, and the first-load download of 3D assets.
File sizes under public/models (size after transfer compression was not measured):
- Twelve-sign CNN (hand-signs): weights 2,273,356 bytes (2.27 MB) + model.json 152,608 bytes (149 KB) ≈ 2.4 MB
- Wand keypoint model (wand): weights 2,371,569 bytes (2.37 MB) + model.json 111,566 bytes (109 KB) ≈ 2.5 MB
- Edo Tensei character GLBs: character1 37.0 MB / character2 16.7 MB / character3 18.6 MB
The sign CNN’s ~2.4 MB is already over HAND_SIGN_MODEL_TARGET_BYTES (1 MB) in handSignInferenceConfig.ts, and under the 5 MB warning in scripts/validate-deploy-models.sh. The wand model is a similar 2.5 MB. One GLB is 16.7–37.0 MB. After the move into the browser, first load is starting to be dominated by 3D assets, not the ML models.
What building it taught us
- For sign recognition, the experience changed on whether each call needed a network round trip, more than on whether inference was heavy. Even a few times a second, a round trip was enough reason to move it on device. We made that call without measuring the round trip in milliseconds.
- We kept the server-inference path, gated to DEV. While collecting training data, swapping the model on the Python side is faster.
- Moving into the browser shifts server inference cost onto client compute, the model download, GPU/CPU spread, heat, and the 3D asset download. At current file sizes, the GLBs (37.0 / 16.7 / 18.6 MB) are starting to dominate first load, ahead of the CNN (~2.4 MB) and the wand model (~2.5 MB).
- Cloud Run did not stay for inference. Putting secret-bearing work (Stripe, the admin dashboard) on the server keeps secrets out of the frontend. The training-data APIs are off in production.
- The server bill did not simply disappear. What changed is where we put the responsibility for compute, network, and asset delivery.