PiAgentUI/docs/superpowers/plans/2026-06-02-piagentui-stt-realtime.md

PiAgentUI STT Realtime Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add backend-proxied realtime speech-to-text so microphone dictation appears live in the message composer and remains editable after stop.

Architecture: Extend the existing STT settings/types with realtime endpoint fields, add a dedicated /api/stt/realtime WebSocket proxy in the extension server, and update the voice hook/InputBox composition model to merge partial/final transcripts without duplicate text. Preserve the current file-upload transcription path for mode: 'file'.

Tech Stack: React 19, TypeScript, Vitest/jsdom, Node http, ws, MediaRecorder, PiAgentUI SQLite settings store.

---

File map

• Modify src/types/api/stt.ts: add realtime settings and realtime event types shared by frontend.
• Modify extensions/piagentui-settings-store.ts: persist optional realtimeEndpoint and realtimeModel through the existing JSON settings namespace.
• Modify extensions/piagentui-server.ts: sanitize realtime settings, validate updates, route /api/stt/realtime in WebSocket upgrade handling, and proxy client/provider events.
• Modify extensions/piagentui-server.test.ts: add focused backend realtime tests.
• Modify src/api/stt.ts: add a helper to build an authenticated realtime WebSocket URL.
• Modify src/api/stt.test.ts: test realtime WebSocket URL generation with token auth.
• Modify src/hooks/useVoiceTranscription.ts: support mode: 'realtime', websocket lifecycle, chunk sending, partial/final callbacks.
• Modify src/hooks/useVoiceTranscription.test.tsx: test realtime hook behavior with mocked WebSocket and MediaRecorder.
• Modify src/features/chat/InputBox.tsx: add live dictation composition refs/callbacks and pass them to InputToolbar.
• Modify src/features/chat/InputBox.test.tsx: test partial replacement, final consolidation, append, and replace behavior.
• Modify src/features/chat/input/InputToolbar.tsx: pass realtime delta callback to VoiceInputButton.
• Modify src/features/chat/input/VoiceInputButton.tsx: pass realtime delta callback to hook and expose busy/error states without changing visual structure.
• Modify src/features/chat/input/VoiceInputButton.test.tsx: ensure the hook receives both final and delta callbacks.
• Modify src/features/settings/components/SttSettings.tsx: expose Mode, Realtime endpoint, and Realtime model controls.
• Modify src/features/settings/components/SttSettings.test.tsx: verify realtime settings can be displayed and saved.

Task 1: Extend STT settings and frontend URL helper

Files:

• Modify: src/types/api/stt.ts

• Modify: src/api/stt.ts

• Test: src/api/stt.test.ts

• Step 1: Write failing API/types test

Add to src/api/stt.test.ts:

it('builds an authenticated realtime STT websocket URL', () => {
  const url = getSttRealtimeUrl()

  expect(url).toBe('ws://127.0.0.1:58785/api/stt/realtime?token=token-123')
})

Update the import:

import { getSttRealtimeUrl, getSttSettings, transcribeAudio, updateSttSettings } from './stt'

• Step 2: Run test to verify RED

Run:

npm run test:run -- src/api/stt.test.ts

Expected: FAIL because getSttRealtimeUrl is not exported.

• Step 3: Implement minimal types and helper

In src/types/api/stt.ts, extend SttSettings and UpdateSttSettingsRequest:

realtimeEndpoint: string
realtimeModel: string

  realtimeEndpoint?: string
  realtimeModel?: string

Add event types:

export type SttRealtimeDelta =
  | { type: 'transcript.partial'; text: string }
  | { type: 'transcript.final'; text: string }
  | { type: 'error'; error: string }
  | { type: 'ready' }
  | { type: 'closed' }

In src/api/stt.ts, add:

export function getSttRealtimeUrl(): string {
  const baseUrl = serverStore
    .getActiveBaseUrl()
    .replace(/^http:/, 'ws:')
    .replace(/^https:/, 'wss:')
  const token = serverStore.getActiveToken()
  const url = new URL('/api/stt/realtime', baseUrl)
  if (token) url.searchParams.set('token', token)
  return url.toString()
}

• Step 4: Run test to verify GREEN

Run:

npm run test:run -- src/api/stt.test.ts

Expected: PASS.

Task 2: Persist and sanitize realtime STT settings

Files:

• Modify: extensions/piagentui-settings-store.ts

• Modify: extensions/piagentui-server.ts

• Modify: src/features/settings/components/SttSettings.tsx

• Test: extensions/piagentui-server.test.ts

• Test: src/features/settings/components/SttSettings.test.tsx

• Step 1: Write failing settings tests

In src/features/settings/components/SttSettings.test.tsx, extend SETTINGS:

  realtimeEndpoint: '/realtime/transcriptions',
  realtimeModel: 'gpt-4o-mini-transcribe',

Add test:

it('edits realtime transcription settings', async () => {
  render(<SttSettings />)

  fireEvent.change(await screen.findByLabelText('Mode'), { target: { value: 'realtime' } })
  fireEvent.change(screen.getByLabelText('Realtime endpoint'), {
    target: { value: '/v1/realtime/transcriptions' },
  })
  fireEvent.change(screen.getByLabelText('Realtime model'), {
    target: { value: 'gpt-4o-realtime-preview' },
  })
  fireEvent.click(screen.getByRole('button', { name: 'Save transcription settings' }))

  await waitFor(() =>
    expect(updateSttSettings).toHaveBeenCalledWith(
      expect.objectContaining({
        mode: 'realtime',
        realtimeEndpoint: '/v1/realtime/transcriptions',
        realtimeModel: 'gpt-4o-realtime-preview',
      }),
    ),
  )
})

In extensions/piagentui-server.test.ts, inside the existing STT settings section, add expectations after saving STT:

expect(savedSttSettings).toMatchObject({
  realtimeEndpoint: '/realtime/transcriptions',
  realtimeModel: 'gpt-4o-mini-transcribe',
})
expect(savedSttSettings.apiKey).toBeUndefined()

• Step 2: Run tests to verify RED

Run:

npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts

Expected: FAIL because mode/realtime fields are not rendered/sanitized yet.

• Step 3: Implement settings fields

In extensions/piagentui-settings-store.ts, add optional fields to StoredSttSettings:

  realtimeEndpoint?: string
  realtimeModel?: string

In extensions/piagentui-server.ts, update defaults in getEffectiveSttSettings():

      realtimeEndpoint: '/realtime/transcriptions',
      realtimeModel: 'gpt-4o-mini-transcribe',

Update sanitizeSttSettings():

      realtimeEndpoint: effective.realtimeEndpoint || '/realtime/transcriptions',
      realtimeModel: effective.realtimeModel || effective.transcriptionModel || 'gpt-4o-mini-transcribe',

Update updateSttSettings(update) validation:

if (update.realtimeEndpoint !== undefined) {
  if (typeof update.realtimeEndpoint !== 'string' || !update.realtimeEndpoint.startsWith('/')) {
    return { ok: false, error: 'Invalid realtimeEndpoint' }
  }
  next.realtimeEndpoint = update.realtimeEndpoint
}
if (update.realtimeModel !== undefined) {
  if (typeof update.realtimeModel !== 'string' || !update.realtimeModel.trim()) {
    return { ok: false, error: 'Invalid realtimeModel' }
  }
  next.realtimeModel = update.realtimeModel.trim()
}

In src/features/settings/components/SttSettings.tsx, include in saveSettings() payload:

          realtimeEndpoint: settings.realtimeEndpoint,
          realtimeModel: settings.realtimeModel,

Add controls in the grid:

;<label className="space-y-1">
  <span className="text-[length:var(--fs-xs)] font-medium text-text-300">Mode</span>
  <SettingsSelect
    wrapperClassName="w-full"
    value={settings.mode}
    onChange={event => setSettings({ ...settings, mode: event.target.value as SttMode })}
  >
    <option value="file">File upload</option>
    <option value="realtime">Realtime WebSocket</option>
  </SettingsSelect>
</label>
{
  settings.mode === 'realtime' && (
    <>
      <label className="space-y-1">
        <span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime endpoint</span>
        <input
          className={inputClass}
          value={settings.realtimeEndpoint}
          onChange={event => setSettings({ ...settings, realtimeEndpoint: event.target.value })}
          placeholder="/realtime/transcriptions"
        />
      </label>
      <label className="space-y-1">
        <span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime model</span>
        <input
          className={inputClass}
          value={settings.realtimeModel}
          onChange={event => setSettings({ ...settings, realtimeModel: event.target.value })}
          placeholder="gpt-4o-realtime-preview"
        />
      </label>
    </>
  )
}

Also import SttMode.

• Step 4: Run tests to verify GREEN

Run:

npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts

Expected: PASS.

Task 3: Backend realtime WebSocket proxy

Files:

• Modify: extensions/piagentui-server.ts

• Test: extensions/piagentui-server.test.ts

• Step 1: Write failing backend realtime test

Add a test to extensions/piagentui-server.test.ts near existing STT tests. The test should start the server, save realtime settings, mock provider WebSocket, connect to /api/stt/realtime?token=${token}, and verify normalized events. Use this shape:

it('proxies realtime STT websocket events without exposing the API key', async () => {
  const saveSttRes = await fetch(`${baseUrl}/api/settings/stt`, {
    method: 'POST',
    headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      enabled: true,
      providerKind: 'openai-compatible',
      mode: 'realtime',
      baseUrl: 'https://api.openai.test/v1',
      apiKeyRequired: false,
      realtimeEndpoint: '/realtime/transcriptions',
      realtimeModel: 'gpt-4o-realtime-preview',
    }),
  })
  expect(saveSttRes.status).toBe(200)

  const ws = new WebSocket(`ws://127.0.0.1:${port}/api/stt/realtime?token=${token}`)
  const messages: any[] = []
  ws.on('message', data => messages.push(JSON.parse(String(data))))

  await waitFor(() => expect(messages.some(message => message.type === 'ready')).toBe(true))
  ws.send(JSON.stringify({ type: 'audio.append', audio: Buffer.from('audio').toString('base64') }))
  ws.send(JSON.stringify({ type: 'audio.commit' }))

  await waitFor(() => expect(messages.some(message => message.type === 'transcript.final')).toBe(true))
  expect(messages).toContainEqual({ type: 'transcript.final', text: 'olá realtime' })

  ws.close()
})

If current tests do not expose a provider WebSocket mock, add a small local WebSocketServer test provider bound to an available port and set baseUrl to http://127.0.0.1:<providerPort>/v1.

• Step 2: Run test to verify RED

Run:

npm run test:run -- extensions/piagentui-server.test.ts

Expected: FAIL because /api/stt/realtime is not routed.

• Step 3: Implement backend route and proxy helpers

In extensions/piagentui-server.ts, add helper types near other STT helpers:

type SttRealtimeClientMessage =
  | { type: 'audio.append'; audio: string }
  | { type: 'audio.commit' }
  | { type: 'session.close' }

Add URL helper:

function buildRealtimeProviderUrl(settings: StoredSttSettings): string {
  const base = (settings.baseUrl || 'https://api.openai.com/v1').replace(/\/+$/, '')
  const endpoint = settings.realtimeEndpoint || '/realtime/transcriptions'
  const wsBase = base.replace(/^http:/, 'ws:').replace(/^https:/, 'wss:')
  return `${wsBase}${endpoint}`
}

Add normalizer:

function normalizeRealtimeProviderMessage(raw: string): { type: string; text?: string; error?: string } | null {
  let event: any
  try {
    event = JSON.parse(raw)
  } catch {
    return null
  }
  const partialText = event.text_delta || event.delta || event.partial || event.transcript?.partial
  if (typeof partialText === 'string') return { type: 'transcript.partial', text: partialText }
  const finalText = event.text || event.final || event.transcript?.text
  if (typeof finalText === 'string') return { type: 'transcript.final', text: finalText }
  const error = event.error?.message || event.error
  if (typeof error === 'string') return { type: 'error', error }
  return null
}

Add handler:

function handleSttRealtimeWebSocket(client: WebSocket, req: http.IncomingMessage) {
  const auth = checkAuth(req)
  if (!auth.ok) {
    client.send(JSON.stringify({ type: 'error', error: 'Unauthorized' }))
    client.close()
    return
  }

  const settings = getEffectiveSttSettings()
  if (!settings.enabled || settings.mode !== 'realtime') {
    client.send(JSON.stringify({ type: 'error', error: 'Realtime STT is not enabled' }))
    client.close()
    return
  }
  if (settings.apiKeyRequired && !settings.apiKey) {
    client.send(JSON.stringify({ type: 'error', error: 'STT API key is not configured' }))
    client.close()
    return
  }
  if (!validateSttBaseUrl(settings.baseUrl)) {
    client.send(JSON.stringify({ type: 'error', error: 'Invalid STT baseUrl' }))
    client.close()
    return
  }

  const headers: Record<string, string> = {}
  if (settings.apiKey) headers.Authorization = `Bearer ${settings.apiKey}`
  const provider = new WebSocket(buildRealtimeProviderUrl(settings), { headers })

  const closeBoth = () => {
    if (provider.readyState === WebSocket.OPEN || provider.readyState === WebSocket.CONNECTING) provider.close()
    if (client.readyState === WebSocket.OPEN) client.close()
  }

  provider.on('open', () => {
    provider.send(
      JSON.stringify({ type: 'session.configure', model: settings.realtimeModel || settings.transcriptionModel }),
    )
    if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'ready' }))
  })
  provider.on('message', data => {
    const normalized = normalizeRealtimeProviderMessage(String(data))
    if (normalized && client.readyState === WebSocket.OPEN) client.send(JSON.stringify(normalized))
  })
  provider.on('error', () => {
    if (client.readyState === WebSocket.OPEN)
      client.send(JSON.stringify({ type: 'error', error: 'Realtime STT provider failed' }))
  })
  provider.on('close', () => {
    if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'closed' }))
  })

  client.on('message', data => {
    if (provider.readyState !== WebSocket.OPEN) return
    let message: SttRealtimeClientMessage
    try {
      message = JSON.parse(String(data))
    } catch {
      client.send(JSON.stringify({ type: 'error', error: 'Invalid realtime message' }))
      return
    }
    provider.send(JSON.stringify(message))
  })
  client.on('close', closeBoth)
  client.on('error', closeBoth)
}

In server.on('upgrade'), route by pathname before generic connection handling:

const pathname = new URL(request.url || '/', 'http://127.0.0.1').pathname
wss!.handleUpgrade(request, socket, head, (ws: WebSocket) => {
  if (pathname === '/api/stt/realtime') {
    handleSttRealtimeWebSocket(ws, request)
    return
  }
  wss!.emit('connection', ws, request)
})

• Step 4: Run backend test to verify GREEN

Run:

npm run test:run -- extensions/piagentui-server.test.ts

Expected: PASS.

Task 4: Realtime hook lifecycle

Files:

• Modify: src/hooks/useVoiceTranscription.ts

• Test: src/hooks/useVoiceTranscription.test.tsx

• Step 1: Write failing realtime hook test

Add mocked WebSocket class in src/hooks/useVoiceTranscription.test.tsx:

class MockWebSocket extends EventTarget {
  static instances: MockWebSocket[] = []
  readyState = WebSocket.CONNECTING
  sent: string[] = []
  onopen: (() => void) | null = null
  onmessage: ((event: MessageEvent) => void) | null = null
  onclose: (() => void) | null = null
  onerror: (() => void) | null = null

  constructor(public readonly url: string) {
    super()
    MockWebSocket.instances.push(this)
  }

  send(data: string) {
    this.sent.push(data)
  }

  close() {
    this.readyState = WebSocket.CLOSED
    this.onclose?.()
  }

  open() {
    this.readyState = WebSocket.OPEN
    this.onopen?.()
  }

  receive(message: unknown) {
    this.onmessage?.({ data: JSON.stringify(message) } as MessageEvent)
  }
}

Add test:

it('streams realtime transcription deltas while recording', async () => {
  vi.mocked(getSttSettings).mockResolvedValue({
    enabled: true,
    providerKind: 'openai-compatible',
    mode: 'realtime',
    baseUrl: 'http://127.0.0.1:8022/v1',
    apiKeyConfigured: false,
    apiKeyRequired: false,
    transcriptionEndpoint: '/audio/transcriptions',
    transcriptionModel: 'model-file',
    realtimeEndpoint: '/realtime/transcriptions',
    realtimeModel: 'model-realtime',
    insertMode: 'append',
  })
  Object.defineProperty(globalThis, 'WebSocket', { configurable: true, value: MockWebSocket })
  const onTranscription = vi.fn()
  const onTranscriptionDelta = vi.fn()
  const { result } = renderHook(() => useVoiceTranscription({ onTranscription, onTranscriptionDelta }))

  await act(async () => {
    await result.current.startRecording()
  })
  const ws = MockWebSocket.instances[0]
  act(() => {
    ws.open()
    ws.receive({ type: 'ready' })
  })
  expect(result.current.state).toBe('recording')

  act(() => ws.receive({ type: 'transcript.partial', text: 'olá' }))
  expect(onTranscriptionDelta).toHaveBeenCalledWith({ type: 'transcript.partial', text: 'olá' }, 'append')

  await act(async () => result.current.stopRecording())
  expect(ws.sent.some(data => JSON.parse(data).type === 'audio.commit')).toBe(true)
})

• Step 2: Run test to verify RED

Run:

npm run test:run -- src/hooks/useVoiceTranscription.test.tsx

Expected: FAIL because hook does not accept onTranscriptionDelta or realtime mode.

• Step 3: Implement realtime hook minimally

In src/hooks/useVoiceTranscription.ts:

• import getSttRealtimeUrl and SttRealtimeDelta;
• extend options:

  onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void

Add refs:

const wsRef = useRef<WebSocket | null>(null)
const realtimeInsertModeRef = useRef<SttInsertMode>('append')

In startRecording, after config validation, branch:

if (activeSettings.mode === 'realtime') {
  await startRealtimeRecording(activeSettings)
  return
}

Implement startRealtimeRecording(activeSettings):

const startRealtimeRecording = useCallback(
  async (activeSettings: SttSettings) => {
    if (typeof WebSocket === 'undefined') {
      setError('Realtime voice transcription is not supported in this browser')
      setState('error')
      return
    }
    realtimeInsertModeRef.current = activeSettings.insertMode
    const ws = new WebSocket(getSttRealtimeUrl())
    wsRef.current = ws
    ws.onmessage = event => {
      let message: SttRealtimeDelta
      try {
        message = JSON.parse(String(event.data))
      } catch {
        return
      }
      if (message.type === 'error') {
        setError(message.error)
        setState('error')
        return
      }
      if (message.type === 'ready') return
      onTranscriptionDelta?.(message, realtimeInsertModeRef.current)
    }
    ws.onerror = () => {
      setError('Realtime voice transcription failed')
      setState('error')
    }
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
    streamRef.current = stream
    const mimeType = MediaRecorder.isTypeSupported?.('audio/webm') ? 'audio/webm' : undefined
    const recorder = new MediaRecorder(stream, mimeType ? { mimeType } : undefined)
    recorderRef.current = recorder
    recorder.ondataavailable = async event => {
      if (event.data.size <= 0 || ws.readyState !== WebSocket.OPEN) return
      const audio = await event.data.arrayBuffer()
      ws.send(JSON.stringify({ type: 'audio.append', audio: btoa(String.fromCharCode(...new Uint8Array(audio))) }))
    }
    recorder.start(500)
    setState('recording')
  },
  [onTranscriptionDelta],
)

Update stopRecording:

if (wsRef.current?.readyState === WebSocket.OPEN) {
  wsRef.current.send(JSON.stringify({ type: 'audio.commit' }))
  wsRef.current.close()
  wsRef.current = null
  cleanupStream()
  setState('idle')
  return
}

• Step 4: Run hook tests to verify GREEN

Run:

npm run test:run -- src/hooks/useVoiceTranscription.test.tsx

Expected: PASS.

Task 5: InputBox live transcript composition

Files:

• Modify: src/features/chat/InputBox.tsx

• Modify: src/features/chat/input/InputToolbar.tsx

• Modify: src/features/chat/input/VoiceInputButton.tsx

• Test: src/features/chat/InputBox.test.tsx

• Test: src/features/chat/input/VoiceInputButton.test.tsx

• Step 1: Write failing InputBox tests

Update the InputToolbar mock in src/features/chat/InputBox.test.tsx so tests can invoke voice callbacks:

let latestVoiceDelta: ((event: { type: string; text?: string }, insertMode: SttInsertMode) => void) | undefined

vi.mock('./input/InputToolbar', () => ({
  InputToolbar: ({ onSend, canSend, onVoiceTranscriptionDelta }: any) => {
    latestVoiceDelta = onVoiceTranscriptionDelta
    return (
      <button type="button" onClick={onSend} disabled={!canSend}>
        send
      </button>
    )
  },
}))

Add tests:

it('shows realtime voice partials without duplicating previous partial text', () => {
  render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
  const textarea = screen.getByRole('textbox') as HTMLTextAreaElement
  fireEvent.change(textarea, { target: { value: 'Resumo:' } })

  act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá' }, 'append'))
  expect(textarea.value).toBe('Resumo: olá')

  act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá mundo' }, 'append'))
  expect(textarea.value).toBe('Resumo: olá mundo')
})

it('keeps final realtime transcription editable in the composer', () => {
  render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
  const textarea = screen.getByRole('textbox') as HTMLTextAreaElement

  act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: 'olá' }, 'replace'))
  act(() => latestVoiceDelta?.({ type: 'transcript.final', text: 'olá mundo' }, 'replace'))
  fireEvent.change(textarea, { target: { value: `${textarea.value}!` } })

  expect(textarea.value).toBe('olá mundo!')
})

• Step 2: Run tests to verify RED

Run:

npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx

Expected: FAIL because onVoiceTranscriptionDelta is not wired.

• Step 3: Implement composition

In VoiceInputButton.tsx:

import type { SttInsertMode, SttRealtimeDelta } from '../../../types/api/stt'

interface VoiceInputButtonProps {
  disabled?: boolean
  onTranscription: (text: string, insertMode: SttInsertMode) => void
  onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
}

Pass to hook:

const { state, isAvailable, error, startRecording, stopRecording } = useVoiceTranscription({
  onTranscription,
  onTranscriptionDelta,
})

In InputToolbar.tsx, extend props and pass through:

  onVoiceTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void

<VoiceInputButton
  disabled={controlsDisabled}
  onTranscription={onVoiceTranscription}
  onTranscriptionDelta={onVoiceTranscriptionDelta}
/>

In InputBox.tsx, import SttRealtimeDelta, add refs:

const voiceDraftBaseRef = useRef<string | null>(null)
const voiceDraftFinalRef = useRef('')
const voiceDraftPartialRef = useRef('')

Add helper:

const composeVoiceDraft = useCallback((insertMode: SttInsertMode) => {
  const base = voiceDraftBaseRef.current ?? ''
  const body = `${voiceDraftFinalRef.current}${voiceDraftPartialRef.current}`.trim()
  if (!body) return base
  if (insertMode === 'replace' || !base.trim()) return body
  const needsSpace = !base.endsWith(' ') && !base.endsWith('\n')
  return `${base}${needsSpace ? ' ' : ''}${body}`
}, [])

Add delta callback:

const handleVoiceTranscriptionDelta = useCallback(
  (event: SttRealtimeDelta, insertMode: SttInsertMode) => {
    if (event.type !== 'transcript.partial' && event.type !== 'transcript.final') return
    if (voiceDraftBaseRef.current === null) {
      voiceDraftBaseRef.current = insertMode === 'replace' ? '' : text
      voiceDraftFinalRef.current = ''
      voiceDraftPartialRef.current = ''
    }
    if (event.type === 'transcript.partial') {
      voiceDraftPartialRef.current = event.text
    } else {
      voiceDraftFinalRef.current = `${voiceDraftFinalRef.current} ${event.text}`.trim()
      voiceDraftPartialRef.current = ''
    }
    setText(composeVoiceDraft(insertMode))
    requestAnimationFrame(() => {
      const textarea = textareaRef.current
      if (!textarea) return
      textarea.focus()
      textarea.setSelectionRange(textarea.value.length, textarea.value.length)
    })
  },
  [composeVoiceDraft, text],
)

Pass to toolbar:

onVoiceTranscriptionDelta = { handleVoiceTranscriptionDelta }

In existing handleVoiceTranscription, clear voice refs before final file-mode insert:

voiceDraftBaseRef.current = null
voiceDraftFinalRef.current = ''
voiceDraftPartialRef.current = ''

• Step 4: Run InputBox tests to verify GREEN

Run:

npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx

Expected: PASS.

Task 6: Final verification and build

Files:

• Verify all touched files from Tasks 1–5.

• Step 1: Format only touched files

Run:

node node_modules/prettier/bin/prettier.cjs --write src/types/api/stt.ts src/api/stt.ts src/api/stt.test.ts src/hooks/useVoiceTranscription.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/InputToolbar.tsx src/features/chat/input/VoiceInputButton.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-settings-store.ts extensions/piagentui-server.ts extensions/piagentui-server.test.ts

Expected: Prettier completes without errors.

• Step 2: Run focused tests

Run:

npm run test:run -- src/api/stt.test.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts

Expected: all selected test files pass.

• Step 3: Run typechecks

Run:

npm run typecheck
npm run typecheck:extensions

Expected: both commands exit 0.

• Step 4: Build frontend dist

Run:

npm run build

Expected: tsc -b && vite build exits 0 and updates dist/.

• Step 5: Check whitespace

Run:

git diff --check

Expected: no output.

• Step 6: Runtime smoke test

After restarting PiAgentUI:

1. Settings → Voice → enable STT.
2. Set Mode to Realtime WebSocket.
3. Configure a compatible realtime STT server.
4. Open a chat session.
5. Click microphone.
6. Speak a short sentence.
7. Confirm partial text appears in the message field before stopping.
8. Click stop.
9. Confirm the final text remains editable and is not sent automatically.

Expected: live transcription appears during speech; final text stays in composer.