Casting by voice and getting the sound into recordings — when mic and speaker fight over one audio session

Speech recognition: spelling variants and dropped sessions

The constructor resolves as SpeechRecognition ?? webkitSpeechRecognition, listening with continuous = true, interimResults = true, lang = ja-JP. Browsers without the API get null, and the experience falls back to signs or buttons only.

The Web Speech API returns 螺旋丸, らせんがん, ラセンガン, 螺旋がん, 螺旋眼 and more (the hook’s comment records the spellings we actually observed). We normalize with NFKC → strip whitespace, punctuation, and long-vowel marks → katakana to hiragana, and check romaji with a separate regex on the raw transcript. Interim and final results both match the same utterance, so casting is debounced to once per 800 ms.

Recovery is split by how sessions actually ended. A natural end (onend) restarts after 250 ms. When screen recording or another app grabs the mic, the session dies with audio-capture / aborted, and we restart after 400 ms. If start() itself fails, the delay doubles each time (min 400 ms, max 2,000 ms). Only not-allowed does not restart; it tells the user to allow the microphone separately from the camera.

1recognition.onerror = (event) => {2  const code = event.error ?? 'unknown';3  if (code === 'not-allowed' || code === 'service-not-allowed') {4    setError('マイク許可が必要です（カメラとは別にマイクを許可してください）');5    shouldListenRef.current = false;6    return;7  }8  // 画面収録や他アプリがマイクを使うと audio-capture / aborted で切れる。聞き続ける。9  if (code === 'audio-capture' || code === 'aborted' || code === 'network') {10    scheduleRestart(400);11  }12};13recognition.onend = () => {14  if (shouldListenRef.current) scheduleRestart(250);15};

The problem: SFX missing from iPhone screen recordings

anime-ar-stock’s jutsu are meant to be captured with iPhone screen recording and posted to social media. But the SFX at the time were synthesized live with the Web Audio API (AudioContext), and recordings had everything except the SFX. The seal “don” sound — seal_don.mp3 played via <audio> — did make it into recordings.

Over the next two days we tried, in order:

• 8/28: route AudioContext output through a MediaStreamDestination into <audio>.srcObject → not captured.
• 8/28: assume the page’s audio session is stuck in ambient; loop a silent WAV through <audio> from the first tap to force playback → not captured.
• 8/29: render each SFX with OfflineAudioContext at trigger time and play it from a Blob URL <audio> → still not captured.
• 8/29: pre-render the SFX to WAV in a Node script (scripts/generate-purple-audio.mjs) using the same DSP as the browser, served as static files → captured.

What we confirmed on device: iOS screen recording (ReplayKit) does not capture AudioContext-originated audio — not via MediaStreamDestination, not via a Blob URL — while a real network-served file played via <audio src> is always captured. Rasengan’s SFX were already .wav files, so we stopped routing them through AudioContext and play them directly.

Where we got stuck: fixing recording broke voice input

Midway through 8/29, voice casting for Hollow Purple stopped responding and the blue/red spawn sounds went silent. Our suspects were two recording workarounds: a “probe” that played and paused five audio files simultaneously on the Start tap to unlock iOS playback, and the silent WAV loop. The theory: both churned the audio session at exactly the moment SpeechRecognition was acquiring the mic. Neither was helping recording anyway, so both came out.

Then the blue/red spawn sounds went completely silent. Those two SFX are played with new Audio().play() from inside the MediaPipe loop, not inside the Start click. iOS refuses playback outside a user gesture, so removing every probe removed the playback unlock itself. The final fix restored only the pattern already working for the seal “don”: one Audio element at volume 0, play → pause → rewind.

A gap remains. Continuous sounds modulated per frame — the fusion drone, the flight whoosh — cannot be pre-rendered every frame, so they stay on AudioContext and are still missing from screen recordings.