PiAgentUI/docs/superpowers/plans/2026-06-02-piagentui-stt-realtime.md

# PiAgentUI STT Realtime Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add backend-proxied realtime speech-to-text so microphone dictation appears live in the message composer and remains editable after stop.

**Architecture:** Extend the existing STT settings/types with realtime endpoint fields, add a dedicated `/api/stt/realtime` WebSocket proxy in the extension server, and update the voice hook/InputBox composition model to merge partial/final transcripts without duplicate text. Preserve the current file-upload transcription path for `mode: 'file'`.

**Tech Stack:** React 19, TypeScript, Vitest/jsdom, Node `http`, `ws`, `MediaRecorder`, PiAgentUI SQLite settings store.

---

## File map

- Modify `src/types/api/stt.ts`: add realtime settings and realtime event types shared by frontend.
- Modify `extensions/piagentui-settings-store.ts`: persist optional `realtimeEndpoint` and `realtimeModel` through the existing JSON settings namespace.
- Modify `extensions/piagentui-server.ts`: sanitize realtime settings, validate updates, route `/api/stt/realtime` in WebSocket upgrade handling, and proxy client/provider events.
- Modify `extensions/piagentui-server.test.ts`: add focused backend realtime tests.
- Modify `src/api/stt.ts`: add a helper to build an authenticated realtime WebSocket URL.
- Modify `src/api/stt.test.ts`: test realtime WebSocket URL generation with token auth.
- Modify `src/hooks/useVoiceTranscription.ts`: support `mode: 'realtime'`, websocket lifecycle, chunk sending, partial/final callbacks.
- Modify `src/hooks/useVoiceTranscription.test.tsx`: test realtime hook behavior with mocked WebSocket and MediaRecorder.
- Modify `src/features/chat/InputBox.tsx`: add live dictation composition refs/callbacks and pass them to `InputToolbar`.
- Modify `src/features/chat/InputBox.test.tsx`: test partial replacement, final consolidation, append, and replace behavior.
- Modify `src/features/chat/input/InputToolbar.tsx`: pass realtime delta callback to `VoiceInputButton`.
- Modify `src/features/chat/input/VoiceInputButton.tsx`: pass realtime delta callback to hook and expose busy/error states without changing visual structure.
- Modify `src/features/chat/input/VoiceInputButton.test.tsx`: ensure the hook receives both final and delta callbacks.
- Modify `src/features/settings/components/SttSettings.tsx`: expose `Mode`, `Realtime endpoint`, and `Realtime model` controls.
- Modify `src/features/settings/components/SttSettings.test.tsx`: verify realtime settings can be displayed and saved.

## Task 1: Extend STT settings and frontend URL helper

**Files:**

- Modify: `src/types/api/stt.ts`
- Modify: `src/api/stt.ts`
- Test: `src/api/stt.test.ts`

- [x] **Step 1: Write failing API/types test**

Add to `src/api/stt.test.ts`:

```ts
it('builds an authenticated realtime STT websocket URL', () => {
  const url = getSttRealtimeUrl()

  expect(url).toBe('ws://127.0.0.1:58785/api/stt/realtime?token=token-123')
})
```

Update the import:

```ts
import { getSttRealtimeUrl, getSttSettings, transcribeAudio, updateSttSettings } from './stt'
```

- [x] **Step 2: Run test to verify RED**

Run:

```bash
npm run test:run -- src/api/stt.test.ts
```

Expected: FAIL because `getSttRealtimeUrl` is not exported.

- [x] **Step 3: Implement minimal types and helper**

In `src/types/api/stt.ts`, extend `SttSettings` and `UpdateSttSettingsRequest`:

```ts
realtimeEndpoint: string
realtimeModel: string
```

```ts
  realtimeEndpoint?: string
  realtimeModel?: string
```

Add event types:

```ts
export type SttRealtimeDelta =
  | { type: 'transcript.partial'; text: string }
  | { type: 'transcript.final'; text: string }
  | { type: 'error'; error: string }
  | { type: 'ready' }
  | { type: 'closed' }
```

In `src/api/stt.ts`, add:

```ts
export function getSttRealtimeUrl(): string {
  const baseUrl = serverStore
    .getActiveBaseUrl()
    .replace(/^http:/, 'ws:')
    .replace(/^https:/, 'wss:')
  const token = serverStore.getActiveToken()
  const url = new URL('/api/stt/realtime', baseUrl)
  if (token) url.searchParams.set('token', token)
  return url.toString()
}
```

- [x] **Step 4: Run test to verify GREEN**

Run:

```bash
npm run test:run -- src/api/stt.test.ts
```

Expected: PASS.

## Task 2: Persist and sanitize realtime STT settings

**Files:**

- Modify: `extensions/piagentui-settings-store.ts`
- Modify: `extensions/piagentui-server.ts`
- Modify: `src/features/settings/components/SttSettings.tsx`
- Test: `extensions/piagentui-server.test.ts`
- Test: `src/features/settings/components/SttSettings.test.tsx`

- [ ] **Step 1: Write failing settings tests**

In `src/features/settings/components/SttSettings.test.tsx`, extend `SETTINGS`:

```ts
  realtimeEndpoint: '/realtime/transcriptions',
  realtimeModel: 'gpt-4o-mini-transcribe',
```

Add test:

```ts
it('edits realtime transcription settings', async () => {
  render(<SttSettings />)

  fireEvent.change(await screen.findByLabelText('Mode'), { target: { value: 'realtime' } })
  fireEvent.change(screen.getByLabelText('Realtime endpoint'), {
    target: { value: '/v1/realtime/transcriptions' },
  })
  fireEvent.change(screen.getByLabelText('Realtime model'), {
    target: { value: 'gpt-4o-realtime-preview' },
  })
  fireEvent.click(screen.getByRole('button', { name: 'Save transcription settings' }))

  await waitFor(() =>
    expect(updateSttSettings).toHaveBeenCalledWith(
      expect.objectContaining({
        mode: 'realtime',
        realtimeEndpoint: '/v1/realtime/transcriptions',
        realtimeModel: 'gpt-4o-realtime-preview',
      }),
    ),
  )
})
```

In `extensions/piagentui-server.test.ts`, inside the existing STT settings section, add expectations after saving STT:

```ts
expect(savedSttSettings).toMatchObject({
  realtimeEndpoint: '/realtime/transcriptions',
  realtimeModel: 'gpt-4o-mini-transcribe',
})
expect(savedSttSettings.apiKey).toBeUndefined()
```

- [ ] **Step 2: Run tests to verify RED**

Run:

```bash
npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
```

Expected: FAIL because mode/realtime fields are not rendered/sanitized yet.

- [ ] **Step 3: Implement settings fields**

In `extensions/piagentui-settings-store.ts`, add optional fields to `StoredSttSettings`:

```ts
  realtimeEndpoint?: string
  realtimeModel?: string
```

In `extensions/piagentui-server.ts`, update defaults in `getEffectiveSttSettings()`:

```ts
      realtimeEndpoint: '/realtime/transcriptions',
      realtimeModel: 'gpt-4o-mini-transcribe',
```

Update `sanitizeSttSettings()`:

```ts
      realtimeEndpoint: effective.realtimeEndpoint || '/realtime/transcriptions',
      realtimeModel: effective.realtimeModel || effective.transcriptionModel || 'gpt-4o-mini-transcribe',
```

Update `updateSttSettings(update)` validation:

```ts
if (update.realtimeEndpoint !== undefined) {
  if (typeof update.realtimeEndpoint !== 'string' || !update.realtimeEndpoint.startsWith('/')) {
    return { ok: false, error: 'Invalid realtimeEndpoint' }
  }
  next.realtimeEndpoint = update.realtimeEndpoint
}
if (update.realtimeModel !== undefined) {
  if (typeof update.realtimeModel !== 'string' || !update.realtimeModel.trim()) {
    return { ok: false, error: 'Invalid realtimeModel' }
  }
  next.realtimeModel = update.realtimeModel.trim()
}
```

In `src/features/settings/components/SttSettings.tsx`, include in `saveSettings()` payload:

```ts
          realtimeEndpoint: settings.realtimeEndpoint,
          realtimeModel: settings.realtimeModel,
```

Add controls in the grid:

```tsx
;<label className="space-y-1">
  <span className="text-[length:var(--fs-xs)] font-medium text-text-300">Mode</span>
  <SettingsSelect
    wrapperClassName="w-full"
    value={settings.mode}
    onChange={event => setSettings({ ...settings, mode: event.target.value as SttMode })}
  >
    <option value="file">File upload</option>
    <option value="realtime">Realtime WebSocket</option>
  </SettingsSelect>
</label>
{
  settings.mode === 'realtime' && (
    <>
      <label className="space-y-1">
        <span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime endpoint</span>
        <input
          className={inputClass}
          value={settings.realtimeEndpoint}
          onChange={event => setSettings({ ...settings, realtimeEndpoint: event.target.value })}
          placeholder="/realtime/transcriptions"
        />
      </label>
      <label className="space-y-1">
        <span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime model</span>
        <input
          className={inputClass}
          value={settings.realtimeModel}
          onChange={event => setSettings({ ...settings, realtimeModel: event.target.value })}
          placeholder="gpt-4o-realtime-preview"
        />
      </label>
    </>
  )
}
```

Also import `SttMode`.

- [ ] **Step 4: Run tests to verify GREEN**

Run:

```bash
npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
```

Expected: PASS.

## Task 3: Backend realtime WebSocket proxy

**Files:**

- Modify: `extensions/piagentui-server.ts`
- Test: `extensions/piagentui-server.test.ts`

- [ ] **Step 1: Write failing backend realtime test**

Add a test to `extensions/piagentui-server.test.ts` near existing STT tests. The test should start the server, save realtime settings, mock provider WebSocket, connect to `/api/stt/realtime?token=${token}`, and verify normalized events. Use this shape:

```ts
it('proxies realtime STT websocket events without exposing the API key', async () => {
  const saveSttRes = await fetch(`${baseUrl}/api/settings/stt`, {
    method: 'POST',
    headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      enabled: true,
      providerKind: 'openai-compatible',
      mode: 'realtime',
      baseUrl: 'https://api.openai.test/v1',
      apiKeyRequired: false,
      realtimeEndpoint: '/realtime/transcriptions',
      realtimeModel: 'gpt-4o-realtime-preview',
    }),
  })
  expect(saveSttRes.status).toBe(200)

  const ws = new WebSocket(`ws://127.0.0.1:${port}/api/stt/realtime?token=${token}`)
  const messages: any[] = []
  ws.on('message', data => messages.push(JSON.parse(String(data))))

  await waitFor(() => expect(messages.some(message => message.type === 'ready')).toBe(true))
  ws.send(JSON.stringify({ type: 'audio.append', audio: Buffer.from('audio').toString('base64') }))
  ws.send(JSON.stringify({ type: 'audio.commit' }))

  await waitFor(() => expect(messages.some(message => message.type === 'transcript.final')).toBe(true))
  expect(messages).toContainEqual({ type: 'transcript.final', text: 'olá realtime' })

  ws.close()
})
```

If current tests do not expose a provider WebSocket mock, add a small local `WebSocketServer` test provider bound to an available port and set `baseUrl` to `http://127.0.0.1:<providerPort>/v1`.

- [ ] **Step 2: Run test to verify RED**

Run:

```bash
npm run test:run -- extensions/piagentui-server.test.ts
```

Expected: FAIL because `/api/stt/realtime` is not routed.

- [ ] **Step 3: Implement backend route and proxy helpers**

In `extensions/piagentui-server.ts`, add helper types near other STT helpers:

```ts
type SttRealtimeClientMessage =
  | { type: 'audio.append'; audio: string }
  | { type: 'audio.commit' }
  | { type: 'session.close' }
```

Add URL helper:

```ts
function buildRealtimeProviderUrl(settings: StoredSttSettings): string {
  const base = (settings.baseUrl || 'https://api.openai.com/v1').replace(/\/+$/, '')
  const endpoint = settings.realtimeEndpoint || '/realtime/transcriptions'
  const wsBase = base.replace(/^http:/, 'ws:').replace(/^https:/, 'wss:')
  return `${wsBase}${endpoint}`
}
```

Add normalizer:

```ts
function normalizeRealtimeProviderMessage(raw: string): { type: string; text?: string; error?: string } | null {
  let event: any
  try {
    event = JSON.parse(raw)
  } catch {
    return null
  }
  const partialText = event.text_delta || event.delta || event.partial || event.transcript?.partial
  if (typeof partialText === 'string') return { type: 'transcript.partial', text: partialText }
  const finalText = event.text || event.final || event.transcript?.text
  if (typeof finalText === 'string') return { type: 'transcript.final', text: finalText }
  const error = event.error?.message || event.error
  if (typeof error === 'string') return { type: 'error', error }
  return null
}
```

Add handler:

```ts
function handleSttRealtimeWebSocket(client: WebSocket, req: http.IncomingMessage) {
  const auth = checkAuth(req)
  if (!auth.ok) {
    client.send(JSON.stringify({ type: 'error', error: 'Unauthorized' }))
    client.close()
    return
  }

  const settings = getEffectiveSttSettings()
  if (!settings.enabled || settings.mode !== 'realtime') {
    client.send(JSON.stringify({ type: 'error', error: 'Realtime STT is not enabled' }))
    client.close()
    return
  }
  if (settings.apiKeyRequired && !settings.apiKey) {
    client.send(JSON.stringify({ type: 'error', error: 'STT API key is not configured' }))
    client.close()
    return
  }
  if (!validateSttBaseUrl(settings.baseUrl)) {
    client.send(JSON.stringify({ type: 'error', error: 'Invalid STT baseUrl' }))
    client.close()
    return
  }

  const headers: Record<string, string> = {}
  if (settings.apiKey) headers.Authorization = `Bearer ${settings.apiKey}`
  const provider = new WebSocket(buildRealtimeProviderUrl(settings), { headers })

  const closeBoth = () => {
    if (provider.readyState === WebSocket.OPEN || provider.readyState === WebSocket.CONNECTING) provider.close()
    if (client.readyState === WebSocket.OPEN) client.close()
  }

  provider.on('open', () => {
    provider.send(
      JSON.stringify({ type: 'session.configure', model: settings.realtimeModel || settings.transcriptionModel }),
    )
    if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'ready' }))
  })
  provider.on('message', data => {
    const normalized = normalizeRealtimeProviderMessage(String(data))
    if (normalized && client.readyState === WebSocket.OPEN) client.send(JSON.stringify(normalized))
  })
  provider.on('error', () => {
    if (client.readyState === WebSocket.OPEN)
      client.send(JSON.stringify({ type: 'error', error: 'Realtime STT provider failed' }))
  })
  provider.on('close', () => {
    if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'closed' }))
  })

  client.on('message', data => {
    if (provider.readyState !== WebSocket.OPEN) return
    let message: SttRealtimeClientMessage
    try {
      message = JSON.parse(String(data))
    } catch {
      client.send(JSON.stringify({ type: 'error', error: 'Invalid realtime message' }))
      return
    }
    provider.send(JSON.stringify(message))
  })
  client.on('close', closeBoth)
  client.on('error', closeBoth)
}
```

In `server.on('upgrade')`, route by pathname before generic connection handling:

```ts
const pathname = new URL(request.url || '/', 'http://127.0.0.1').pathname
wss!.handleUpgrade(request, socket, head, (ws: WebSocket) => {
  if (pathname === '/api/stt/realtime') {
    handleSttRealtimeWebSocket(ws, request)
    return
  }
  wss!.emit('connection', ws, request)
})
```

- [ ] **Step 4: Run backend test to verify GREEN**

Run:

```bash
npm run test:run -- extensions/piagentui-server.test.ts
```

Expected: PASS.

## Task 4: Realtime hook lifecycle

**Files:**

- Modify: `src/hooks/useVoiceTranscription.ts`
- Test: `src/hooks/useVoiceTranscription.test.tsx`

- [ ] **Step 1: Write failing realtime hook test**

Add mocked WebSocket class in `src/hooks/useVoiceTranscription.test.tsx`:

```ts
class MockWebSocket extends EventTarget {
  static instances: MockWebSocket[] = []
  readyState = WebSocket.CONNECTING
  sent: string[] = []
  onopen: (() => void) | null = null
  onmessage: ((event: MessageEvent) => void) | null = null
  onclose: (() => void) | null = null
  onerror: (() => void) | null = null

  constructor(public readonly url: string) {
    super()
    MockWebSocket.instances.push(this)
  }

  send(data: string) {
    this.sent.push(data)
  }

  close() {
    this.readyState = WebSocket.CLOSED
    this.onclose?.()
  }

  open() {
    this.readyState = WebSocket.OPEN
    this.onopen?.()
  }

  receive(message: unknown) {
    this.onmessage?.({ data: JSON.stringify(message) } as MessageEvent)
  }
}
```

Add test:

```ts
it('streams realtime transcription deltas while recording', async () => {
  vi.mocked(getSttSettings).mockResolvedValue({
    enabled: true,
    providerKind: 'openai-compatible',
    mode: 'realtime',
    baseUrl: 'http://127.0.0.1:8022/v1',
    apiKeyConfigured: false,
    apiKeyRequired: false,
    transcriptionEndpoint: '/audio/transcriptions',
    transcriptionModel: 'model-file',
    realtimeEndpoint: '/realtime/transcriptions',
    realtimeModel: 'model-realtime',
    insertMode: 'append',
  })
  Object.defineProperty(globalThis, 'WebSocket', { configurable: true, value: MockWebSocket })
  const onTranscription = vi.fn()
  const onTranscriptionDelta = vi.fn()
  const { result } = renderHook(() => useVoiceTranscription({ onTranscription, onTranscriptionDelta }))

  await act(async () => {
    await result.current.startRecording()
  })
  const ws = MockWebSocket.instances[0]
  act(() => {
    ws.open()
    ws.receive({ type: 'ready' })
  })
  expect(result.current.state).toBe('recording')

  act(() => ws.receive({ type: 'transcript.partial', text: 'olá' }))
  expect(onTranscriptionDelta).toHaveBeenCalledWith({ type: 'transcript.partial', text: 'olá' }, 'append')

  await act(async () => result.current.stopRecording())
  expect(ws.sent.some(data => JSON.parse(data).type === 'audio.commit')).toBe(true)
})
```

- [ ] **Step 2: Run test to verify RED**

Run:

```bash
npm run test:run -- src/hooks/useVoiceTranscription.test.tsx
```

Expected: FAIL because hook does not accept `onTranscriptionDelta` or realtime mode.

- [ ] **Step 3: Implement realtime hook minimally**

In `src/hooks/useVoiceTranscription.ts`:

- import `getSttRealtimeUrl` and `SttRealtimeDelta`;
- extend options:

```ts
  onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
```

Add refs:

```ts
const wsRef = useRef<WebSocket | null>(null)
const realtimeInsertModeRef = useRef<SttInsertMode>('append')
```

In `startRecording`, after config validation, branch:

```ts
if (activeSettings.mode === 'realtime') {
  await startRealtimeRecording(activeSettings)
  return
}
```

Implement `startRealtimeRecording(activeSettings)`:

```ts
const startRealtimeRecording = useCallback(
  async (activeSettings: SttSettings) => {
    if (typeof WebSocket === 'undefined') {
      setError('Realtime voice transcription is not supported in this browser')
      setState('error')
      return
    }
    realtimeInsertModeRef.current = activeSettings.insertMode
    const ws = new WebSocket(getSttRealtimeUrl())
    wsRef.current = ws
    ws.onmessage = event => {
      let message: SttRealtimeDelta
      try {
        message = JSON.parse(String(event.data))
      } catch {
        return
      }
      if (message.type === 'error') {
        setError(message.error)
        setState('error')
        return
      }
      if (message.type === 'ready') return
      onTranscriptionDelta?.(message, realtimeInsertModeRef.current)
    }
    ws.onerror = () => {
      setError('Realtime voice transcription failed')
      setState('error')
    }
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
    streamRef.current = stream
    const mimeType = MediaRecorder.isTypeSupported?.('audio/webm') ? 'audio/webm' : undefined
    const recorder = new MediaRecorder(stream, mimeType ? { mimeType } : undefined)
    recorderRef.current = recorder
    recorder.ondataavailable = async event => {
      if (event.data.size <= 0 || ws.readyState !== WebSocket.OPEN) return
      const audio = await event.data.arrayBuffer()
      ws.send(JSON.stringify({ type: 'audio.append', audio: btoa(String.fromCharCode(...new Uint8Array(audio))) }))
    }
    recorder.start(500)
    setState('recording')
  },
  [onTranscriptionDelta],
)
```

Update `stopRecording`:

```ts
if (wsRef.current?.readyState === WebSocket.OPEN) {
  wsRef.current.send(JSON.stringify({ type: 'audio.commit' }))
  wsRef.current.close()
  wsRef.current = null
  cleanupStream()
  setState('idle')
  return
}
```

- [ ] **Step 4: Run hook tests to verify GREEN**

Run:

```bash
npm run test:run -- src/hooks/useVoiceTranscription.test.tsx
```

Expected: PASS.

## Task 5: InputBox live transcript composition

**Files:**

- Modify: `src/features/chat/InputBox.tsx`
- Modify: `src/features/chat/input/InputToolbar.tsx`
- Modify: `src/features/chat/input/VoiceInputButton.tsx`
- Test: `src/features/chat/InputBox.test.tsx`
- Test: `src/features/chat/input/VoiceInputButton.test.tsx`

- [ ] **Step 1: Write failing InputBox tests**

Update the `InputToolbar` mock in `src/features/chat/InputBox.test.tsx` so tests can invoke voice callbacks:

```ts
let latestVoiceDelta: ((event: { type: string; text?: string }, insertMode: SttInsertMode) => void) | undefined

vi.mock('./input/InputToolbar', () => ({
  InputToolbar: ({ onSend, canSend, onVoiceTranscriptionDelta }: any) => {
    latestVoiceDelta = onVoiceTranscriptionDelta
    return (
      <button type="button" onClick={onSend} disabled={!canSend}>
        send
      </button>
    )
  },
}))
```

Add tests:

```ts
it('shows realtime voice partials without duplicating previous partial text', () => {
  render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
  const textarea = screen.getByRole('textbox') as HTMLTextAreaElement
  fireEvent.change(textarea, { target: { value: 'Resumo:' } })

  act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá' }, 'append'))
  expect(textarea.value).toBe('Resumo: olá')

  act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá mundo' }, 'append'))
  expect(textarea.value).toBe('Resumo: olá mundo')
})

it('keeps final realtime transcription editable in the composer', () => {
  render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
  const textarea = screen.getByRole('textbox') as HTMLTextAreaElement

  act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: 'olá' }, 'replace'))
  act(() => latestVoiceDelta?.({ type: 'transcript.final', text: 'olá mundo' }, 'replace'))
  fireEvent.change(textarea, { target: { value: `${textarea.value}!` } })

  expect(textarea.value).toBe('olá mundo!')
})
```

- [ ] **Step 2: Run tests to verify RED**

Run:

```bash
npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx
```

Expected: FAIL because `onVoiceTranscriptionDelta` is not wired.

- [ ] **Step 3: Implement composition**

In `VoiceInputButton.tsx`:

```ts
import type { SttInsertMode, SttRealtimeDelta } from '../../../types/api/stt'

interface VoiceInputButtonProps {
  disabled?: boolean
  onTranscription: (text: string, insertMode: SttInsertMode) => void
  onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
}
```

Pass to hook:

```ts
const { state, isAvailable, error, startRecording, stopRecording } = useVoiceTranscription({
  onTranscription,
  onTranscriptionDelta,
})
```

In `InputToolbar.tsx`, extend props and pass through:

```ts
  onVoiceTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
```

```tsx
<VoiceInputButton
  disabled={controlsDisabled}
  onTranscription={onVoiceTranscription}
  onTranscriptionDelta={onVoiceTranscriptionDelta}
/>
```

In `InputBox.tsx`, import `SttRealtimeDelta`, add refs:

```ts
const voiceDraftBaseRef = useRef<string | null>(null)
const voiceDraftFinalRef = useRef('')
const voiceDraftPartialRef = useRef('')
```

Add helper:

```ts
const composeVoiceDraft = useCallback((insertMode: SttInsertMode) => {
  const base = voiceDraftBaseRef.current ?? ''
  const body = `${voiceDraftFinalRef.current}${voiceDraftPartialRef.current}`.trim()
  if (!body) return base
  if (insertMode === 'replace' || !base.trim()) return body
  const needsSpace = !base.endsWith(' ') && !base.endsWith('\n')
  return `${base}${needsSpace ? ' ' : ''}${body}`
}, [])
```

Add delta callback:

```ts
const handleVoiceTranscriptionDelta = useCallback(
  (event: SttRealtimeDelta, insertMode: SttInsertMode) => {
    if (event.type !== 'transcript.partial' && event.type !== 'transcript.final') return
    if (voiceDraftBaseRef.current === null) {
      voiceDraftBaseRef.current = insertMode === 'replace' ? '' : text
      voiceDraftFinalRef.current = ''
      voiceDraftPartialRef.current = ''
    }
    if (event.type === 'transcript.partial') {
      voiceDraftPartialRef.current = event.text
    } else {
      voiceDraftFinalRef.current = `${voiceDraftFinalRef.current} ${event.text}`.trim()
      voiceDraftPartialRef.current = ''
    }
    setText(composeVoiceDraft(insertMode))
    requestAnimationFrame(() => {
      const textarea = textareaRef.current
      if (!textarea) return
      textarea.focus()
      textarea.setSelectionRange(textarea.value.length, textarea.value.length)
    })
  },
  [composeVoiceDraft, text],
)
```

Pass to toolbar:

```tsx
onVoiceTranscriptionDelta = { handleVoiceTranscriptionDelta }
```

In existing `handleVoiceTranscription`, clear voice refs before final file-mode insert:

```ts
voiceDraftBaseRef.current = null
voiceDraftFinalRef.current = ''
voiceDraftPartialRef.current = ''
```

- [ ] **Step 4: Run InputBox tests to verify GREEN**

Run:

```bash
npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx
```

Expected: PASS.

## Task 6: Final verification and build

**Files:**

- Verify all touched files from Tasks 1–5.

- [ ] **Step 1: Format only touched files**

Run:

```bash
node node_modules/prettier/bin/prettier.cjs --write src/types/api/stt.ts src/api/stt.ts src/api/stt.test.ts src/hooks/useVoiceTranscription.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/InputToolbar.tsx src/features/chat/input/VoiceInputButton.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-settings-store.ts extensions/piagentui-server.ts extensions/piagentui-server.test.ts
```

Expected: Prettier completes without errors.

- [ ] **Step 2: Run focused tests**

Run:

```bash
npm run test:run -- src/api/stt.test.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
```

Expected: all selected test files pass.

- [ ] **Step 3: Run typechecks**

Run:

```bash
npm run typecheck
npm run typecheck:extensions
```

Expected: both commands exit 0.

- [ ] **Step 4: Build frontend dist**

Run:

```bash
npm run build
```

Expected: `tsc -b && vite build` exits 0 and updates `dist/`.

- [ ] **Step 5: Check whitespace**

Run:

```bash
git diff --check
```

Expected: no output.

- [ ] **Step 6: Runtime smoke test**

After restarting PiAgentUI:

1. Settings → Voice → enable STT.
2. Set `Mode` to `Realtime WebSocket`.
3. Configure a compatible realtime STT server.
4. Open a chat session.
5. Click microphone.
6. Speak a short sentence.
7. Confirm partial text appears in the message field before stopping.
8. Click stop.
9. Confirm the final text remains editable and is not sent automatically.

Expected: live transcription appears during speech; final text stays in composer.