wip: snapshot das alterações não commitadas (2026-10-20)
Origem: D:/Documents/Github/PiAgentUI/PiAgentUi Arquivos omitidos (segredos, bancos de dados, logs, >50 MB): 4
This commit is contained in:
1 parent
b3489e1ede
commit
98b5db029f
70 files changed
+9466
-9191
No files matched your search
@@ -0,0 +1,916 @@
|
||||
# PiAgentUI STT Realtime Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add backend-proxied realtime speech-to-text so microphone dictation appears live in the message composer and remains editable after stop.
|
||||
|
||||
**Architecture:** Extend the existing STT settings/types with realtime endpoint fields, add a dedicated `/api/stt/realtime` WebSocket proxy in the extension server, and update the voice hook/InputBox composition model to merge partial/final transcripts without duplicate text. Preserve the current file-upload transcription path for `mode: 'file'`.
|
||||
|
||||
**Tech Stack:** React 19, TypeScript, Vitest/jsdom, Node `http`, `ws`, `MediaRecorder`, PiAgentUI SQLite settings store.
|
||||
|
||||
---
|
||||
|
||||
## File map
|
||||
|
||||
- Modify `src/types/api/stt.ts`: add realtime settings and realtime event types shared by frontend.
|
||||
- Modify `extensions/piagentui-settings-store.ts`: persist optional `realtimeEndpoint` and `realtimeModel` through the existing JSON settings namespace.
|
||||
- Modify `extensions/piagentui-server.ts`: sanitize realtime settings, validate updates, route `/api/stt/realtime` in WebSocket upgrade handling, and proxy client/provider events.
|
||||
- Modify `extensions/piagentui-server.test.ts`: add focused backend realtime tests.
|
||||
- Modify `src/api/stt.ts`: add a helper to build an authenticated realtime WebSocket URL.
|
||||
- Modify `src/api/stt.test.ts`: test realtime WebSocket URL generation with token auth.
|
||||
- Modify `src/hooks/useVoiceTranscription.ts`: support `mode: 'realtime'`, websocket lifecycle, chunk sending, partial/final callbacks.
|
||||
- Modify `src/hooks/useVoiceTranscription.test.tsx`: test realtime hook behavior with mocked WebSocket and MediaRecorder.
|
||||
- Modify `src/features/chat/InputBox.tsx`: add live dictation composition refs/callbacks and pass them to `InputToolbar`.
|
||||
- Modify `src/features/chat/InputBox.test.tsx`: test partial replacement, final consolidation, append, and replace behavior.
|
||||
- Modify `src/features/chat/input/InputToolbar.tsx`: pass realtime delta callback to `VoiceInputButton`.
|
||||
- Modify `src/features/chat/input/VoiceInputButton.tsx`: pass realtime delta callback to hook and expose busy/error states without changing visual structure.
|
||||
- Modify `src/features/chat/input/VoiceInputButton.test.tsx`: ensure the hook receives both final and delta callbacks.
|
||||
- Modify `src/features/settings/components/SttSettings.tsx`: expose `Mode`, `Realtime endpoint`, and `Realtime model` controls.
|
||||
- Modify `src/features/settings/components/SttSettings.test.tsx`: verify realtime settings can be displayed and saved.
|
||||
|
||||
## Task 1: Extend STT settings and frontend URL helper
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/types/api/stt.ts`
|
||||
- Modify: `src/api/stt.ts`
|
||||
- Test: `src/api/stt.test.ts`
|
||||
|
||||
- [x] **Step 1: Write failing API/types test**
|
||||
|
||||
Add to `src/api/stt.test.ts`:
|
||||
|
||||
```ts
|
||||
it('builds an authenticated realtime STT websocket URL', () => {
|
||||
const url = getSttRealtimeUrl()
|
||||
|
||||
expect(url).toBe('ws://127.0.0.1:58785/api/stt/realtime?token=token-123')
|
||||
})
|
||||
```
|
||||
|
||||
Update the import:
|
||||
|
||||
```ts
|
||||
import { getSttRealtimeUrl, getSttSettings, transcribeAudio, updateSttSettings } from './stt'
|
||||
```
|
||||
|
||||
- [x] **Step 2: Run test to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/api/stt.test.ts
|
||||
```
|
||||
|
||||
Expected: FAIL because `getSttRealtimeUrl` is not exported.
|
||||
|
||||
- [x] **Step 3: Implement minimal types and helper**
|
||||
|
||||
In `src/types/api/stt.ts`, extend `SttSettings` and `UpdateSttSettingsRequest`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: string
|
||||
realtimeModel: string
|
||||
```
|
||||
|
||||
```ts
|
||||
realtimeEndpoint?: string
|
||||
realtimeModel?: string
|
||||
```
|
||||
|
||||
Add event types:
|
||||
|
||||
```ts
|
||||
export type SttRealtimeDelta =
|
||||
| { type: 'transcript.partial'; text: string }
|
||||
| { type: 'transcript.final'; text: string }
|
||||
| { type: 'error'; error: string }
|
||||
| { type: 'ready' }
|
||||
| { type: 'closed' }
|
||||
```
|
||||
|
||||
In `src/api/stt.ts`, add:
|
||||
|
||||
```ts
|
||||
export function getSttRealtimeUrl(): string {
|
||||
const baseUrl = serverStore
|
||||
.getActiveBaseUrl()
|
||||
.replace(/^http:/, 'ws:')
|
||||
.replace(/^https:/, 'wss:')
|
||||
const token = serverStore.getActiveToken()
|
||||
const url = new URL('/api/stt/realtime', baseUrl)
|
||||
if (token) url.searchParams.set('token', token)
|
||||
return url.toString()
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: Run test to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/api/stt.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 2: Persist and sanitize realtime STT settings
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `extensions/piagentui-settings-store.ts`
|
||||
- Modify: `extensions/piagentui-server.ts`
|
||||
- Modify: `src/features/settings/components/SttSettings.tsx`
|
||||
- Test: `extensions/piagentui-server.test.ts`
|
||||
- Test: `src/features/settings/components/SttSettings.test.tsx`
|
||||
|
||||
- [ ] **Step 1: Write failing settings tests**
|
||||
|
||||
In `src/features/settings/components/SttSettings.test.tsx`, extend `SETTINGS`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-mini-transcribe',
|
||||
```
|
||||
|
||||
Add test:
|
||||
|
||||
```ts
|
||||
it('edits realtime transcription settings', async () => {
|
||||
render(<SttSettings />)
|
||||
|
||||
fireEvent.change(await screen.findByLabelText('Mode'), { target: { value: 'realtime' } })
|
||||
fireEvent.change(screen.getByLabelText('Realtime endpoint'), {
|
||||
target: { value: '/v1/realtime/transcriptions' },
|
||||
})
|
||||
fireEvent.change(screen.getByLabelText('Realtime model'), {
|
||||
target: { value: 'gpt-4o-realtime-preview' },
|
||||
})
|
||||
fireEvent.click(screen.getByRole('button', { name: 'Save transcription settings' }))
|
||||
|
||||
await waitFor(() =>
|
||||
expect(updateSttSettings).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
mode: 'realtime',
|
||||
realtimeEndpoint: '/v1/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-realtime-preview',
|
||||
}),
|
||||
),
|
||||
)
|
||||
})
|
||||
```
|
||||
|
||||
In `extensions/piagentui-server.test.ts`, inside the existing STT settings section, add expectations after saving STT:
|
||||
|
||||
```ts
|
||||
expect(savedSttSettings).toMatchObject({
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-mini-transcribe',
|
||||
})
|
||||
expect(savedSttSettings.apiKey).toBeUndefined()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: FAIL because mode/realtime fields are not rendered/sanitized yet.
|
||||
|
||||
- [ ] **Step 3: Implement settings fields**
|
||||
|
||||
In `extensions/piagentui-settings-store.ts`, add optional fields to `StoredSttSettings`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint?: string
|
||||
realtimeModel?: string
|
||||
```
|
||||
|
||||
In `extensions/piagentui-server.ts`, update defaults in `getEffectiveSttSettings()`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-mini-transcribe',
|
||||
```
|
||||
|
||||
Update `sanitizeSttSettings()`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: effective.realtimeEndpoint || '/realtime/transcriptions',
|
||||
realtimeModel: effective.realtimeModel || effective.transcriptionModel || 'gpt-4o-mini-transcribe',
|
||||
```
|
||||
|
||||
Update `updateSttSettings(update)` validation:
|
||||
|
||||
```ts
|
||||
if (update.realtimeEndpoint !== undefined) {
|
||||
if (typeof update.realtimeEndpoint !== 'string' || !update.realtimeEndpoint.startsWith('/')) {
|
||||
return { ok: false, error: 'Invalid realtimeEndpoint' }
|
||||
}
|
||||
next.realtimeEndpoint = update.realtimeEndpoint
|
||||
}
|
||||
if (update.realtimeModel !== undefined) {
|
||||
if (typeof update.realtimeModel !== 'string' || !update.realtimeModel.trim()) {
|
||||
return { ok: false, error: 'Invalid realtimeModel' }
|
||||
}
|
||||
next.realtimeModel = update.realtimeModel.trim()
|
||||
}
|
||||
```
|
||||
|
||||
In `src/features/settings/components/SttSettings.tsx`, include in `saveSettings()` payload:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: settings.realtimeEndpoint,
|
||||
realtimeModel: settings.realtimeModel,
|
||||
```
|
||||
|
||||
Add controls in the grid:
|
||||
|
||||
```tsx
|
||||
;<label className="space-y-1">
|
||||
<span className="text-[length:var(--fs-xs)] font-medium text-text-300">Mode</span>
|
||||
<SettingsSelect
|
||||
wrapperClassName="w-full"
|
||||
value={settings.mode}
|
||||
onChange={event => setSettings({ ...settings, mode: event.target.value as SttMode })}
|
||||
>
|
||||
<option value="file">File upload</option>
|
||||
<option value="realtime">Realtime WebSocket</option>
|
||||
</SettingsSelect>
|
||||
</label>
|
||||
{
|
||||
settings.mode === 'realtime' && (
|
||||
<>
|
||||
<label className="space-y-1">
|
||||
<span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime endpoint</span>
|
||||
<input
|
||||
className={inputClass}
|
||||
value={settings.realtimeEndpoint}
|
||||
onChange={event => setSettings({ ...settings, realtimeEndpoint: event.target.value })}
|
||||
placeholder="/realtime/transcriptions"
|
||||
/>
|
||||
</label>
|
||||
<label className="space-y-1">
|
||||
<span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime model</span>
|
||||
<input
|
||||
className={inputClass}
|
||||
value={settings.realtimeModel}
|
||||
onChange={event => setSettings({ ...settings, realtimeModel: event.target.value })}
|
||||
placeholder="gpt-4o-realtime-preview"
|
||||
/>
|
||||
</label>
|
||||
</>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
Also import `SttMode`.
|
||||
|
||||
- [ ] **Step 4: Run tests to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 3: Backend realtime WebSocket proxy
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `extensions/piagentui-server.ts`
|
||||
- Test: `extensions/piagentui-server.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing backend realtime test**
|
||||
|
||||
Add a test to `extensions/piagentui-server.test.ts` near existing STT tests. The test should start the server, save realtime settings, mock provider WebSocket, connect to `/api/stt/realtime?token=${token}`, and verify normalized events. Use this shape:
|
||||
|
||||
```ts
|
||||
it('proxies realtime STT websocket events without exposing the API key', async () => {
|
||||
const saveSttRes = await fetch(`${baseUrl}/api/settings/stt`, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
enabled: true,
|
||||
providerKind: 'openai-compatible',
|
||||
mode: 'realtime',
|
||||
baseUrl: 'https://api.openai.test/v1',
|
||||
apiKeyRequired: false,
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-realtime-preview',
|
||||
}),
|
||||
})
|
||||
expect(saveSttRes.status).toBe(200)
|
||||
|
||||
const ws = new WebSocket(`ws://127.0.0.1:${port}/api/stt/realtime?token=${token}`)
|
||||
const messages: any[] = []
|
||||
ws.on('message', data => messages.push(JSON.parse(String(data))))
|
||||
|
||||
await waitFor(() => expect(messages.some(message => message.type === 'ready')).toBe(true))
|
||||
ws.send(JSON.stringify({ type: 'audio.append', audio: Buffer.from('audio').toString('base64') }))
|
||||
ws.send(JSON.stringify({ type: 'audio.commit' }))
|
||||
|
||||
await waitFor(() => expect(messages.some(message => message.type === 'transcript.final')).toBe(true))
|
||||
expect(messages).toContainEqual({ type: 'transcript.final', text: 'olá realtime' })
|
||||
|
||||
ws.close()
|
||||
})
|
||||
```
|
||||
|
||||
If current tests do not expose a provider WebSocket mock, add a small local `WebSocketServer` test provider bound to an available port and set `baseUrl` to `http://127.0.0.1:<providerPort>/v1`.
|
||||
|
||||
- [ ] **Step 2: Run test to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: FAIL because `/api/stt/realtime` is not routed.
|
||||
|
||||
- [ ] **Step 3: Implement backend route and proxy helpers**
|
||||
|
||||
In `extensions/piagentui-server.ts`, add helper types near other STT helpers:
|
||||
|
||||
```ts
|
||||
type SttRealtimeClientMessage =
|
||||
| { type: 'audio.append'; audio: string }
|
||||
| { type: 'audio.commit' }
|
||||
| { type: 'session.close' }
|
||||
```
|
||||
|
||||
Add URL helper:
|
||||
|
||||
```ts
|
||||
function buildRealtimeProviderUrl(settings: StoredSttSettings): string {
|
||||
const base = (settings.baseUrl || 'https://api.openai.com/v1').replace(/\/+$/, '')
|
||||
const endpoint = settings.realtimeEndpoint || '/realtime/transcriptions'
|
||||
const wsBase = base.replace(/^http:/, 'ws:').replace(/^https:/, 'wss:')
|
||||
return `${wsBase}${endpoint}`
|
||||
}
|
||||
```
|
||||
|
||||
Add normalizer:
|
||||
|
||||
```ts
|
||||
function normalizeRealtimeProviderMessage(raw: string): { type: string; text?: string; error?: string } | null {
|
||||
let event: any
|
||||
try {
|
||||
event = JSON.parse(raw)
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
const partialText = event.text_delta || event.delta || event.partial || event.transcript?.partial
|
||||
if (typeof partialText === 'string') return { type: 'transcript.partial', text: partialText }
|
||||
const finalText = event.text || event.final || event.transcript?.text
|
||||
if (typeof finalText === 'string') return { type: 'transcript.final', text: finalText }
|
||||
const error = event.error?.message || event.error
|
||||
if (typeof error === 'string') return { type: 'error', error }
|
||||
return null
|
||||
}
|
||||
```
|
||||
|
||||
Add handler:
|
||||
|
||||
```ts
|
||||
function handleSttRealtimeWebSocket(client: WebSocket, req: http.IncomingMessage) {
|
||||
const auth = checkAuth(req)
|
||||
if (!auth.ok) {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Unauthorized' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
|
||||
const settings = getEffectiveSttSettings()
|
||||
if (!settings.enabled || settings.mode !== 'realtime') {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Realtime STT is not enabled' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
if (settings.apiKeyRequired && !settings.apiKey) {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'STT API key is not configured' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
if (!validateSttBaseUrl(settings.baseUrl)) {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Invalid STT baseUrl' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
|
||||
const headers: Record<string, string> = {}
|
||||
if (settings.apiKey) headers.Authorization = `Bearer ${settings.apiKey}`
|
||||
const provider = new WebSocket(buildRealtimeProviderUrl(settings), { headers })
|
||||
|
||||
const closeBoth = () => {
|
||||
if (provider.readyState === WebSocket.OPEN || provider.readyState === WebSocket.CONNECTING) provider.close()
|
||||
if (client.readyState === WebSocket.OPEN) client.close()
|
||||
}
|
||||
|
||||
provider.on('open', () => {
|
||||
provider.send(
|
||||
JSON.stringify({ type: 'session.configure', model: settings.realtimeModel || settings.transcriptionModel }),
|
||||
)
|
||||
if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'ready' }))
|
||||
})
|
||||
provider.on('message', data => {
|
||||
const normalized = normalizeRealtimeProviderMessage(String(data))
|
||||
if (normalized && client.readyState === WebSocket.OPEN) client.send(JSON.stringify(normalized))
|
||||
})
|
||||
provider.on('error', () => {
|
||||
if (client.readyState === WebSocket.OPEN)
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Realtime STT provider failed' }))
|
||||
})
|
||||
provider.on('close', () => {
|
||||
if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'closed' }))
|
||||
})
|
||||
|
||||
client.on('message', data => {
|
||||
if (provider.readyState !== WebSocket.OPEN) return
|
||||
let message: SttRealtimeClientMessage
|
||||
try {
|
||||
message = JSON.parse(String(data))
|
||||
} catch {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Invalid realtime message' }))
|
||||
return
|
||||
}
|
||||
provider.send(JSON.stringify(message))
|
||||
})
|
||||
client.on('close', closeBoth)
|
||||
client.on('error', closeBoth)
|
||||
}
|
||||
```
|
||||
|
||||
In `server.on('upgrade')`, route by pathname before generic connection handling:
|
||||
|
||||
```ts
|
||||
const pathname = new URL(request.url || '/', 'http://127.0.0.1').pathname
|
||||
wss!.handleUpgrade(request, socket, head, (ws: WebSocket) => {
|
||||
if (pathname === '/api/stt/realtime') {
|
||||
handleSttRealtimeWebSocket(ws, request)
|
||||
return
|
||||
}
|
||||
wss!.emit('connection', ws, request)
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run backend test to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 4: Realtime hook lifecycle
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/hooks/useVoiceTranscription.ts`
|
||||
- Test: `src/hooks/useVoiceTranscription.test.tsx`
|
||||
|
||||
- [ ] **Step 1: Write failing realtime hook test**
|
||||
|
||||
Add mocked WebSocket class in `src/hooks/useVoiceTranscription.test.tsx`:
|
||||
|
||||
```ts
|
||||
class MockWebSocket extends EventTarget {
|
||||
static instances: MockWebSocket[] = []
|
||||
readyState = WebSocket.CONNECTING
|
||||
sent: string[] = []
|
||||
onopen: (() => void) | null = null
|
||||
onmessage: ((event: MessageEvent) => void) | null = null
|
||||
onclose: (() => void) | null = null
|
||||
onerror: (() => void) | null = null
|
||||
|
||||
constructor(public readonly url: string) {
|
||||
super()
|
||||
MockWebSocket.instances.push(this)
|
||||
}
|
||||
|
||||
send(data: string) {
|
||||
this.sent.push(data)
|
||||
}
|
||||
|
||||
close() {
|
||||
this.readyState = WebSocket.CLOSED
|
||||
this.onclose?.()
|
||||
}
|
||||
|
||||
open() {
|
||||
this.readyState = WebSocket.OPEN
|
||||
this.onopen?.()
|
||||
}
|
||||
|
||||
receive(message: unknown) {
|
||||
this.onmessage?.({ data: JSON.stringify(message) } as MessageEvent)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Add test:
|
||||
|
||||
```ts
|
||||
it('streams realtime transcription deltas while recording', async () => {
|
||||
vi.mocked(getSttSettings).mockResolvedValue({
|
||||
enabled: true,
|
||||
providerKind: 'openai-compatible',
|
||||
mode: 'realtime',
|
||||
baseUrl: 'http://127.0.0.1:8022/v1',
|
||||
apiKeyConfigured: false,
|
||||
apiKeyRequired: false,
|
||||
transcriptionEndpoint: '/audio/transcriptions',
|
||||
transcriptionModel: 'model-file',
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'model-realtime',
|
||||
insertMode: 'append',
|
||||
})
|
||||
Object.defineProperty(globalThis, 'WebSocket', { configurable: true, value: MockWebSocket })
|
||||
const onTranscription = vi.fn()
|
||||
const onTranscriptionDelta = vi.fn()
|
||||
const { result } = renderHook(() => useVoiceTranscription({ onTranscription, onTranscriptionDelta }))
|
||||
|
||||
await act(async () => {
|
||||
await result.current.startRecording()
|
||||
})
|
||||
const ws = MockWebSocket.instances[0]
|
||||
act(() => {
|
||||
ws.open()
|
||||
ws.receive({ type: 'ready' })
|
||||
})
|
||||
expect(result.current.state).toBe('recording')
|
||||
|
||||
act(() => ws.receive({ type: 'transcript.partial', text: 'olá' }))
|
||||
expect(onTranscriptionDelta).toHaveBeenCalledWith({ type: 'transcript.partial', text: 'olá' }, 'append')
|
||||
|
||||
await act(async () => result.current.stopRecording())
|
||||
expect(ws.sent.some(data => JSON.parse(data).type === 'audio.commit')).toBe(true)
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/hooks/useVoiceTranscription.test.tsx
|
||||
```
|
||||
|
||||
Expected: FAIL because hook does not accept `onTranscriptionDelta` or realtime mode.
|
||||
|
||||
- [ ] **Step 3: Implement realtime hook minimally**
|
||||
|
||||
In `src/hooks/useVoiceTranscription.ts`:
|
||||
|
||||
- import `getSttRealtimeUrl` and `SttRealtimeDelta`;
|
||||
- extend options:
|
||||
|
||||
```ts
|
||||
onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
|
||||
```
|
||||
|
||||
Add refs:
|
||||
|
||||
```ts
|
||||
const wsRef = useRef<WebSocket | null>(null)
|
||||
const realtimeInsertModeRef = useRef<SttInsertMode>('append')
|
||||
```
|
||||
|
||||
In `startRecording`, after config validation, branch:
|
||||
|
||||
```ts
|
||||
if (activeSettings.mode === 'realtime') {
|
||||
await startRealtimeRecording(activeSettings)
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
Implement `startRealtimeRecording(activeSettings)`:
|
||||
|
||||
```ts
|
||||
const startRealtimeRecording = useCallback(
|
||||
async (activeSettings: SttSettings) => {
|
||||
if (typeof WebSocket === 'undefined') {
|
||||
setError('Realtime voice transcription is not supported in this browser')
|
||||
setState('error')
|
||||
return
|
||||
}
|
||||
realtimeInsertModeRef.current = activeSettings.insertMode
|
||||
const ws = new WebSocket(getSttRealtimeUrl())
|
||||
wsRef.current = ws
|
||||
ws.onmessage = event => {
|
||||
let message: SttRealtimeDelta
|
||||
try {
|
||||
message = JSON.parse(String(event.data))
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
if (message.type === 'error') {
|
||||
setError(message.error)
|
||||
setState('error')
|
||||
return
|
||||
}
|
||||
if (message.type === 'ready') return
|
||||
onTranscriptionDelta?.(message, realtimeInsertModeRef.current)
|
||||
}
|
||||
ws.onerror = () => {
|
||||
setError('Realtime voice transcription failed')
|
||||
setState('error')
|
||||
}
|
||||
const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
|
||||
streamRef.current = stream
|
||||
const mimeType = MediaRecorder.isTypeSupported?.('audio/webm') ? 'audio/webm' : undefined
|
||||
const recorder = new MediaRecorder(stream, mimeType ? { mimeType } : undefined)
|
||||
recorderRef.current = recorder
|
||||
recorder.ondataavailable = async event => {
|
||||
if (event.data.size <= 0 || ws.readyState !== WebSocket.OPEN) return
|
||||
const audio = await event.data.arrayBuffer()
|
||||
ws.send(JSON.stringify({ type: 'audio.append', audio: btoa(String.fromCharCode(...new Uint8Array(audio))) }))
|
||||
}
|
||||
recorder.start(500)
|
||||
setState('recording')
|
||||
},
|
||||
[onTranscriptionDelta],
|
||||
)
|
||||
```
|
||||
|
||||
Update `stopRecording`:
|
||||
|
||||
```ts
|
||||
if (wsRef.current?.readyState === WebSocket.OPEN) {
|
||||
wsRef.current.send(JSON.stringify({ type: 'audio.commit' }))
|
||||
wsRef.current.close()
|
||||
wsRef.current = null
|
||||
cleanupStream()
|
||||
setState('idle')
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run hook tests to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/hooks/useVoiceTranscription.test.tsx
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 5: InputBox live transcript composition
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/features/chat/InputBox.tsx`
|
||||
- Modify: `src/features/chat/input/InputToolbar.tsx`
|
||||
- Modify: `src/features/chat/input/VoiceInputButton.tsx`
|
||||
- Test: `src/features/chat/InputBox.test.tsx`
|
||||
- Test: `src/features/chat/input/VoiceInputButton.test.tsx`
|
||||
|
||||
- [ ] **Step 1: Write failing InputBox tests**
|
||||
|
||||
Update the `InputToolbar` mock in `src/features/chat/InputBox.test.tsx` so tests can invoke voice callbacks:
|
||||
|
||||
```ts
|
||||
let latestVoiceDelta: ((event: { type: string; text?: string }, insertMode: SttInsertMode) => void) | undefined
|
||||
|
||||
vi.mock('./input/InputToolbar', () => ({
|
||||
InputToolbar: ({ onSend, canSend, onVoiceTranscriptionDelta }: any) => {
|
||||
latestVoiceDelta = onVoiceTranscriptionDelta
|
||||
return (
|
||||
<button type="button" onClick={onSend} disabled={!canSend}>
|
||||
send
|
||||
</button>
|
||||
)
|
||||
},
|
||||
}))
|
||||
```
|
||||
|
||||
Add tests:
|
||||
|
||||
```ts
|
||||
it('shows realtime voice partials without duplicating previous partial text', () => {
|
||||
render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
|
||||
const textarea = screen.getByRole('textbox') as HTMLTextAreaElement
|
||||
fireEvent.change(textarea, { target: { value: 'Resumo:' } })
|
||||
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá' }, 'append'))
|
||||
expect(textarea.value).toBe('Resumo: olá')
|
||||
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá mundo' }, 'append'))
|
||||
expect(textarea.value).toBe('Resumo: olá mundo')
|
||||
})
|
||||
|
||||
it('keeps final realtime transcription editable in the composer', () => {
|
||||
render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
|
||||
const textarea = screen.getByRole('textbox') as HTMLTextAreaElement
|
||||
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: 'olá' }, 'replace'))
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.final', text: 'olá mundo' }, 'replace'))
|
||||
fireEvent.change(textarea, { target: { value: `${textarea.value}!` } })
|
||||
|
||||
expect(textarea.value).toBe('olá mundo!')
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx
|
||||
```
|
||||
|
||||
Expected: FAIL because `onVoiceTranscriptionDelta` is not wired.
|
||||
|
||||
- [ ] **Step 3: Implement composition**
|
||||
|
||||
In `VoiceInputButton.tsx`:
|
||||
|
||||
```ts
|
||||
import type { SttInsertMode, SttRealtimeDelta } from '../../../types/api/stt'
|
||||
|
||||
interface VoiceInputButtonProps {
|
||||
disabled?: boolean
|
||||
onTranscription: (text: string, insertMode: SttInsertMode) => void
|
||||
onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
|
||||
}
|
||||
```
|
||||
|
||||
Pass to hook:
|
||||
|
||||
```ts
|
||||
const { state, isAvailable, error, startRecording, stopRecording } = useVoiceTranscription({
|
||||
onTranscription,
|
||||
onTranscriptionDelta,
|
||||
})
|
||||
```
|
||||
|
||||
In `InputToolbar.tsx`, extend props and pass through:
|
||||
|
||||
```ts
|
||||
onVoiceTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
|
||||
```
|
||||
|
||||
```tsx
|
||||
<VoiceInputButton
|
||||
disabled={controlsDisabled}
|
||||
onTranscription={onVoiceTranscription}
|
||||
onTranscriptionDelta={onVoiceTranscriptionDelta}
|
||||
/>
|
||||
```
|
||||
|
||||
In `InputBox.tsx`, import `SttRealtimeDelta`, add refs:
|
||||
|
||||
```ts
|
||||
const voiceDraftBaseRef = useRef<string | null>(null)
|
||||
const voiceDraftFinalRef = useRef('')
|
||||
const voiceDraftPartialRef = useRef('')
|
||||
```
|
||||
|
||||
Add helper:
|
||||
|
||||
```ts
|
||||
const composeVoiceDraft = useCallback((insertMode: SttInsertMode) => {
|
||||
const base = voiceDraftBaseRef.current ?? ''
|
||||
const body = `${voiceDraftFinalRef.current}${voiceDraftPartialRef.current}`.trim()
|
||||
if (!body) return base
|
||||
if (insertMode === 'replace' || !base.trim()) return body
|
||||
const needsSpace = !base.endsWith(' ') && !base.endsWith('\n')
|
||||
return `${base}${needsSpace ? ' ' : ''}${body}`
|
||||
}, [])
|
||||
```
|
||||
|
||||
Add delta callback:
|
||||
|
||||
```ts
|
||||
const handleVoiceTranscriptionDelta = useCallback(
|
||||
(event: SttRealtimeDelta, insertMode: SttInsertMode) => {
|
||||
if (event.type !== 'transcript.partial' && event.type !== 'transcript.final') return
|
||||
if (voiceDraftBaseRef.current === null) {
|
||||
voiceDraftBaseRef.current = insertMode === 'replace' ? '' : text
|
||||
voiceDraftFinalRef.current = ''
|
||||
voiceDraftPartialRef.current = ''
|
||||
}
|
||||
if (event.type === 'transcript.partial') {
|
||||
voiceDraftPartialRef.current = event.text
|
||||
} else {
|
||||
voiceDraftFinalRef.current = `${voiceDraftFinalRef.current} ${event.text}`.trim()
|
||||
voiceDraftPartialRef.current = ''
|
||||
}
|
||||
setText(composeVoiceDraft(insertMode))
|
||||
requestAnimationFrame(() => {
|
||||
const textarea = textareaRef.current
|
||||
if (!textarea) return
|
||||
textarea.focus()
|
||||
textarea.setSelectionRange(textarea.value.length, textarea.value.length)
|
||||
})
|
||||
},
|
||||
[composeVoiceDraft, text],
|
||||
)
|
||||
```
|
||||
|
||||
Pass to toolbar:
|
||||
|
||||
```tsx
|
||||
onVoiceTranscriptionDelta = { handleVoiceTranscriptionDelta }
|
||||
```
|
||||
|
||||
In existing `handleVoiceTranscription`, clear voice refs before final file-mode insert:
|
||||
|
||||
```ts
|
||||
voiceDraftBaseRef.current = null
|
||||
voiceDraftFinalRef.current = ''
|
||||
voiceDraftPartialRef.current = ''
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run InputBox tests to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 6: Final verification and build
|
||||
|
||||
**Files:**
|
||||
|
||||
- Verify all touched files from Tasks 1–5.
|
||||
|
||||
- [ ] **Step 1: Format only touched files**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
node node_modules/prettier/bin/prettier.cjs --write src/types/api/stt.ts src/api/stt.ts src/api/stt.test.ts src/hooks/useVoiceTranscription.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/InputToolbar.tsx src/features/chat/input/VoiceInputButton.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-settings-store.ts extensions/piagentui-server.ts extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: Prettier completes without errors.
|
||||
|
||||
- [ ] **Step 2: Run focused tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/api/stt.test.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: all selected test files pass.
|
||||
|
||||
- [ ] **Step 3: Run typechecks**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run typecheck
|
||||
npm run typecheck:extensions
|
||||
```
|
||||
|
||||
Expected: both commands exit 0.
|
||||
|
||||
- [ ] **Step 4: Build frontend dist**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected: `tsc -b && vite build` exits 0 and updates `dist/`.
|
||||
|
||||
- [ ] **Step 5: Check whitespace**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: no output.
|
||||
|
||||
- [ ] **Step 6: Runtime smoke test**
|
||||
|
||||
After restarting PiAgentUI:
|
||||
|
||||
1. Settings → Voice → enable STT.
|
||||
2. Set `Mode` to `Realtime WebSocket`.
|
||||
3. Configure a compatible realtime STT server.
|
||||
4. Open a chat session.
|
||||
5. Click microphone.
|
||||
6. Speak a short sentence.
|
||||
7. Confirm partial text appears in the message field before stopping.
|
||||
8. Click stop.
|
||||
9. Confirm the final text remains editable and is not sent automatically.
|
||||
|
||||
Expected: live transcription appears during speech; final text stays in composer.
|
||||
Reference in new issue
Block a user