wip: snapshot das alterações não commitadas (2026-10-20)
Origem: D:/Documents/Github/PiAgentUI/PiAgentUi Arquivos omitidos (segredos, bancos de dados, logs, >50 MB): 4
This commit is contained in:
1 parent
b3489e1ede
commit
98b5db029f
70 files changed
+9466
-9191
No files matched your search
@@ -0,0 +1,916 @@
|
||||
# PiAgentUI STT Realtime Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add backend-proxied realtime speech-to-text so microphone dictation appears live in the message composer and remains editable after stop.
|
||||
|
||||
**Architecture:** Extend the existing STT settings/types with realtime endpoint fields, add a dedicated `/api/stt/realtime` WebSocket proxy in the extension server, and update the voice hook/InputBox composition model to merge partial/final transcripts without duplicate text. Preserve the current file-upload transcription path for `mode: 'file'`.
|
||||
|
||||
**Tech Stack:** React 19, TypeScript, Vitest/jsdom, Node `http`, `ws`, `MediaRecorder`, PiAgentUI SQLite settings store.
|
||||
|
||||
---
|
||||
|
||||
## File map
|
||||
|
||||
- Modify `src/types/api/stt.ts`: add realtime settings and realtime event types shared by frontend.
|
||||
- Modify `extensions/piagentui-settings-store.ts`: persist optional `realtimeEndpoint` and `realtimeModel` through the existing JSON settings namespace.
|
||||
- Modify `extensions/piagentui-server.ts`: sanitize realtime settings, validate updates, route `/api/stt/realtime` in WebSocket upgrade handling, and proxy client/provider events.
|
||||
- Modify `extensions/piagentui-server.test.ts`: add focused backend realtime tests.
|
||||
- Modify `src/api/stt.ts`: add a helper to build an authenticated realtime WebSocket URL.
|
||||
- Modify `src/api/stt.test.ts`: test realtime WebSocket URL generation with token auth.
|
||||
- Modify `src/hooks/useVoiceTranscription.ts`: support `mode: 'realtime'`, websocket lifecycle, chunk sending, partial/final callbacks.
|
||||
- Modify `src/hooks/useVoiceTranscription.test.tsx`: test realtime hook behavior with mocked WebSocket and MediaRecorder.
|
||||
- Modify `src/features/chat/InputBox.tsx`: add live dictation composition refs/callbacks and pass them to `InputToolbar`.
|
||||
- Modify `src/features/chat/InputBox.test.tsx`: test partial replacement, final consolidation, append, and replace behavior.
|
||||
- Modify `src/features/chat/input/InputToolbar.tsx`: pass realtime delta callback to `VoiceInputButton`.
|
||||
- Modify `src/features/chat/input/VoiceInputButton.tsx`: pass realtime delta callback to hook and expose busy/error states without changing visual structure.
|
||||
- Modify `src/features/chat/input/VoiceInputButton.test.tsx`: ensure the hook receives both final and delta callbacks.
|
||||
- Modify `src/features/settings/components/SttSettings.tsx`: expose `Mode`, `Realtime endpoint`, and `Realtime model` controls.
|
||||
- Modify `src/features/settings/components/SttSettings.test.tsx`: verify realtime settings can be displayed and saved.
|
||||
|
||||
## Task 1: Extend STT settings and frontend URL helper
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/types/api/stt.ts`
|
||||
- Modify: `src/api/stt.ts`
|
||||
- Test: `src/api/stt.test.ts`
|
||||
|
||||
- [x] **Step 1: Write failing API/types test**
|
||||
|
||||
Add to `src/api/stt.test.ts`:
|
||||
|
||||
```ts
|
||||
it('builds an authenticated realtime STT websocket URL', () => {
|
||||
const url = getSttRealtimeUrl()
|
||||
|
||||
expect(url).toBe('ws://127.0.0.1:58785/api/stt/realtime?token=token-123')
|
||||
})
|
||||
```
|
||||
|
||||
Update the import:
|
||||
|
||||
```ts
|
||||
import { getSttRealtimeUrl, getSttSettings, transcribeAudio, updateSttSettings } from './stt'
|
||||
```
|
||||
|
||||
- [x] **Step 2: Run test to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/api/stt.test.ts
|
||||
```
|
||||
|
||||
Expected: FAIL because `getSttRealtimeUrl` is not exported.
|
||||
|
||||
- [x] **Step 3: Implement minimal types and helper**
|
||||
|
||||
In `src/types/api/stt.ts`, extend `SttSettings` and `UpdateSttSettingsRequest`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: string
|
||||
realtimeModel: string
|
||||
```
|
||||
|
||||
```ts
|
||||
realtimeEndpoint?: string
|
||||
realtimeModel?: string
|
||||
```
|
||||
|
||||
Add event types:
|
||||
|
||||
```ts
|
||||
export type SttRealtimeDelta =
|
||||
| { type: 'transcript.partial'; text: string }
|
||||
| { type: 'transcript.final'; text: string }
|
||||
| { type: 'error'; error: string }
|
||||
| { type: 'ready' }
|
||||
| { type: 'closed' }
|
||||
```
|
||||
|
||||
In `src/api/stt.ts`, add:
|
||||
|
||||
```ts
|
||||
export function getSttRealtimeUrl(): string {
|
||||
const baseUrl = serverStore
|
||||
.getActiveBaseUrl()
|
||||
.replace(/^http:/, 'ws:')
|
||||
.replace(/^https:/, 'wss:')
|
||||
const token = serverStore.getActiveToken()
|
||||
const url = new URL('/api/stt/realtime', baseUrl)
|
||||
if (token) url.searchParams.set('token', token)
|
||||
return url.toString()
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 4: Run test to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/api/stt.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 2: Persist and sanitize realtime STT settings
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `extensions/piagentui-settings-store.ts`
|
||||
- Modify: `extensions/piagentui-server.ts`
|
||||
- Modify: `src/features/settings/components/SttSettings.tsx`
|
||||
- Test: `extensions/piagentui-server.test.ts`
|
||||
- Test: `src/features/settings/components/SttSettings.test.tsx`
|
||||
|
||||
- [ ] **Step 1: Write failing settings tests**
|
||||
|
||||
In `src/features/settings/components/SttSettings.test.tsx`, extend `SETTINGS`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-mini-transcribe',
|
||||
```
|
||||
|
||||
Add test:
|
||||
|
||||
```ts
|
||||
it('edits realtime transcription settings', async () => {
|
||||
render(<SttSettings />)
|
||||
|
||||
fireEvent.change(await screen.findByLabelText('Mode'), { target: { value: 'realtime' } })
|
||||
fireEvent.change(screen.getByLabelText('Realtime endpoint'), {
|
||||
target: { value: '/v1/realtime/transcriptions' },
|
||||
})
|
||||
fireEvent.change(screen.getByLabelText('Realtime model'), {
|
||||
target: { value: 'gpt-4o-realtime-preview' },
|
||||
})
|
||||
fireEvent.click(screen.getByRole('button', { name: 'Save transcription settings' }))
|
||||
|
||||
await waitFor(() =>
|
||||
expect(updateSttSettings).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
mode: 'realtime',
|
||||
realtimeEndpoint: '/v1/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-realtime-preview',
|
||||
}),
|
||||
),
|
||||
)
|
||||
})
|
||||
```
|
||||
|
||||
In `extensions/piagentui-server.test.ts`, inside the existing STT settings section, add expectations after saving STT:
|
||||
|
||||
```ts
|
||||
expect(savedSttSettings).toMatchObject({
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-mini-transcribe',
|
||||
})
|
||||
expect(savedSttSettings.apiKey).toBeUndefined()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: FAIL because mode/realtime fields are not rendered/sanitized yet.
|
||||
|
||||
- [ ] **Step 3: Implement settings fields**
|
||||
|
||||
In `extensions/piagentui-settings-store.ts`, add optional fields to `StoredSttSettings`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint?: string
|
||||
realtimeModel?: string
|
||||
```
|
||||
|
||||
In `extensions/piagentui-server.ts`, update defaults in `getEffectiveSttSettings()`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-mini-transcribe',
|
||||
```
|
||||
|
||||
Update `sanitizeSttSettings()`:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: effective.realtimeEndpoint || '/realtime/transcriptions',
|
||||
realtimeModel: effective.realtimeModel || effective.transcriptionModel || 'gpt-4o-mini-transcribe',
|
||||
```
|
||||
|
||||
Update `updateSttSettings(update)` validation:
|
||||
|
||||
```ts
|
||||
if (update.realtimeEndpoint !== undefined) {
|
||||
if (typeof update.realtimeEndpoint !== 'string' || !update.realtimeEndpoint.startsWith('/')) {
|
||||
return { ok: false, error: 'Invalid realtimeEndpoint' }
|
||||
}
|
||||
next.realtimeEndpoint = update.realtimeEndpoint
|
||||
}
|
||||
if (update.realtimeModel !== undefined) {
|
||||
if (typeof update.realtimeModel !== 'string' || !update.realtimeModel.trim()) {
|
||||
return { ok: false, error: 'Invalid realtimeModel' }
|
||||
}
|
||||
next.realtimeModel = update.realtimeModel.trim()
|
||||
}
|
||||
```
|
||||
|
||||
In `src/features/settings/components/SttSettings.tsx`, include in `saveSettings()` payload:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint: settings.realtimeEndpoint,
|
||||
realtimeModel: settings.realtimeModel,
|
||||
```
|
||||
|
||||
Add controls in the grid:
|
||||
|
||||
```tsx
|
||||
;<label className="space-y-1">
|
||||
<span className="text-[length:var(--fs-xs)] font-medium text-text-300">Mode</span>
|
||||
<SettingsSelect
|
||||
wrapperClassName="w-full"
|
||||
value={settings.mode}
|
||||
onChange={event => setSettings({ ...settings, mode: event.target.value as SttMode })}
|
||||
>
|
||||
<option value="file">File upload</option>
|
||||
<option value="realtime">Realtime WebSocket</option>
|
||||
</SettingsSelect>
|
||||
</label>
|
||||
{
|
||||
settings.mode === 'realtime' && (
|
||||
<>
|
||||
<label className="space-y-1">
|
||||
<span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime endpoint</span>
|
||||
<input
|
||||
className={inputClass}
|
||||
value={settings.realtimeEndpoint}
|
||||
onChange={event => setSettings({ ...settings, realtimeEndpoint: event.target.value })}
|
||||
placeholder="/realtime/transcriptions"
|
||||
/>
|
||||
</label>
|
||||
<label className="space-y-1">
|
||||
<span className="text-[length:var(--fs-xs)] font-medium text-text-300">Realtime model</span>
|
||||
<input
|
||||
className={inputClass}
|
||||
value={settings.realtimeModel}
|
||||
onChange={event => setSettings({ ...settings, realtimeModel: event.target.value })}
|
||||
placeholder="gpt-4o-realtime-preview"
|
||||
/>
|
||||
</label>
|
||||
</>
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
Also import `SttMode`.
|
||||
|
||||
- [ ] **Step 4: Run tests to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 3: Backend realtime WebSocket proxy
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `extensions/piagentui-server.ts`
|
||||
- Test: `extensions/piagentui-server.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing backend realtime test**
|
||||
|
||||
Add a test to `extensions/piagentui-server.test.ts` near existing STT tests. The test should start the server, save realtime settings, mock provider WebSocket, connect to `/api/stt/realtime?token=${token}`, and verify normalized events. Use this shape:
|
||||
|
||||
```ts
|
||||
it('proxies realtime STT websocket events without exposing the API key', async () => {
|
||||
const saveSttRes = await fetch(`${baseUrl}/api/settings/stt`, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
enabled: true,
|
||||
providerKind: 'openai-compatible',
|
||||
mode: 'realtime',
|
||||
baseUrl: 'https://api.openai.test/v1',
|
||||
apiKeyRequired: false,
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'gpt-4o-realtime-preview',
|
||||
}),
|
||||
})
|
||||
expect(saveSttRes.status).toBe(200)
|
||||
|
||||
const ws = new WebSocket(`ws://127.0.0.1:${port}/api/stt/realtime?token=${token}`)
|
||||
const messages: any[] = []
|
||||
ws.on('message', data => messages.push(JSON.parse(String(data))))
|
||||
|
||||
await waitFor(() => expect(messages.some(message => message.type === 'ready')).toBe(true))
|
||||
ws.send(JSON.stringify({ type: 'audio.append', audio: Buffer.from('audio').toString('base64') }))
|
||||
ws.send(JSON.stringify({ type: 'audio.commit' }))
|
||||
|
||||
await waitFor(() => expect(messages.some(message => message.type === 'transcript.final')).toBe(true))
|
||||
expect(messages).toContainEqual({ type: 'transcript.final', text: 'olá realtime' })
|
||||
|
||||
ws.close()
|
||||
})
|
||||
```
|
||||
|
||||
If current tests do not expose a provider WebSocket mock, add a small local `WebSocketServer` test provider bound to an available port and set `baseUrl` to `http://127.0.0.1:<providerPort>/v1`.
|
||||
|
||||
- [ ] **Step 2: Run test to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: FAIL because `/api/stt/realtime` is not routed.
|
||||
|
||||
- [ ] **Step 3: Implement backend route and proxy helpers**
|
||||
|
||||
In `extensions/piagentui-server.ts`, add helper types near other STT helpers:
|
||||
|
||||
```ts
|
||||
type SttRealtimeClientMessage =
|
||||
| { type: 'audio.append'; audio: string }
|
||||
| { type: 'audio.commit' }
|
||||
| { type: 'session.close' }
|
||||
```
|
||||
|
||||
Add URL helper:
|
||||
|
||||
```ts
|
||||
function buildRealtimeProviderUrl(settings: StoredSttSettings): string {
|
||||
const base = (settings.baseUrl || 'https://api.openai.com/v1').replace(/\/+$/, '')
|
||||
const endpoint = settings.realtimeEndpoint || '/realtime/transcriptions'
|
||||
const wsBase = base.replace(/^http:/, 'ws:').replace(/^https:/, 'wss:')
|
||||
return `${wsBase}${endpoint}`
|
||||
}
|
||||
```
|
||||
|
||||
Add normalizer:
|
||||
|
||||
```ts
|
||||
function normalizeRealtimeProviderMessage(raw: string): { type: string; text?: string; error?: string } | null {
|
||||
let event: any
|
||||
try {
|
||||
event = JSON.parse(raw)
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
const partialText = event.text_delta || event.delta || event.partial || event.transcript?.partial
|
||||
if (typeof partialText === 'string') return { type: 'transcript.partial', text: partialText }
|
||||
const finalText = event.text || event.final || event.transcript?.text
|
||||
if (typeof finalText === 'string') return { type: 'transcript.final', text: finalText }
|
||||
const error = event.error?.message || event.error
|
||||
if (typeof error === 'string') return { type: 'error', error }
|
||||
return null
|
||||
}
|
||||
```
|
||||
|
||||
Add handler:
|
||||
|
||||
```ts
|
||||
function handleSttRealtimeWebSocket(client: WebSocket, req: http.IncomingMessage) {
|
||||
const auth = checkAuth(req)
|
||||
if (!auth.ok) {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Unauthorized' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
|
||||
const settings = getEffectiveSttSettings()
|
||||
if (!settings.enabled || settings.mode !== 'realtime') {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Realtime STT is not enabled' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
if (settings.apiKeyRequired && !settings.apiKey) {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'STT API key is not configured' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
if (!validateSttBaseUrl(settings.baseUrl)) {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Invalid STT baseUrl' }))
|
||||
client.close()
|
||||
return
|
||||
}
|
||||
|
||||
const headers: Record<string, string> = {}
|
||||
if (settings.apiKey) headers.Authorization = `Bearer ${settings.apiKey}`
|
||||
const provider = new WebSocket(buildRealtimeProviderUrl(settings), { headers })
|
||||
|
||||
const closeBoth = () => {
|
||||
if (provider.readyState === WebSocket.OPEN || provider.readyState === WebSocket.CONNECTING) provider.close()
|
||||
if (client.readyState === WebSocket.OPEN) client.close()
|
||||
}
|
||||
|
||||
provider.on('open', () => {
|
||||
provider.send(
|
||||
JSON.stringify({ type: 'session.configure', model: settings.realtimeModel || settings.transcriptionModel }),
|
||||
)
|
||||
if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'ready' }))
|
||||
})
|
||||
provider.on('message', data => {
|
||||
const normalized = normalizeRealtimeProviderMessage(String(data))
|
||||
if (normalized && client.readyState === WebSocket.OPEN) client.send(JSON.stringify(normalized))
|
||||
})
|
||||
provider.on('error', () => {
|
||||
if (client.readyState === WebSocket.OPEN)
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Realtime STT provider failed' }))
|
||||
})
|
||||
provider.on('close', () => {
|
||||
if (client.readyState === WebSocket.OPEN) client.send(JSON.stringify({ type: 'closed' }))
|
||||
})
|
||||
|
||||
client.on('message', data => {
|
||||
if (provider.readyState !== WebSocket.OPEN) return
|
||||
let message: SttRealtimeClientMessage
|
||||
try {
|
||||
message = JSON.parse(String(data))
|
||||
} catch {
|
||||
client.send(JSON.stringify({ type: 'error', error: 'Invalid realtime message' }))
|
||||
return
|
||||
}
|
||||
provider.send(JSON.stringify(message))
|
||||
})
|
||||
client.on('close', closeBoth)
|
||||
client.on('error', closeBoth)
|
||||
}
|
||||
```
|
||||
|
||||
In `server.on('upgrade')`, route by pathname before generic connection handling:
|
||||
|
||||
```ts
|
||||
const pathname = new URL(request.url || '/', 'http://127.0.0.1').pathname
|
||||
wss!.handleUpgrade(request, socket, head, (ws: WebSocket) => {
|
||||
if (pathname === '/api/stt/realtime') {
|
||||
handleSttRealtimeWebSocket(ws, request)
|
||||
return
|
||||
}
|
||||
wss!.emit('connection', ws, request)
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run backend test to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 4: Realtime hook lifecycle
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/hooks/useVoiceTranscription.ts`
|
||||
- Test: `src/hooks/useVoiceTranscription.test.tsx`
|
||||
|
||||
- [ ] **Step 1: Write failing realtime hook test**
|
||||
|
||||
Add mocked WebSocket class in `src/hooks/useVoiceTranscription.test.tsx`:
|
||||
|
||||
```ts
|
||||
class MockWebSocket extends EventTarget {
|
||||
static instances: MockWebSocket[] = []
|
||||
readyState = WebSocket.CONNECTING
|
||||
sent: string[] = []
|
||||
onopen: (() => void) | null = null
|
||||
onmessage: ((event: MessageEvent) => void) | null = null
|
||||
onclose: (() => void) | null = null
|
||||
onerror: (() => void) | null = null
|
||||
|
||||
constructor(public readonly url: string) {
|
||||
super()
|
||||
MockWebSocket.instances.push(this)
|
||||
}
|
||||
|
||||
send(data: string) {
|
||||
this.sent.push(data)
|
||||
}
|
||||
|
||||
close() {
|
||||
this.readyState = WebSocket.CLOSED
|
||||
this.onclose?.()
|
||||
}
|
||||
|
||||
open() {
|
||||
this.readyState = WebSocket.OPEN
|
||||
this.onopen?.()
|
||||
}
|
||||
|
||||
receive(message: unknown) {
|
||||
this.onmessage?.({ data: JSON.stringify(message) } as MessageEvent)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Add test:
|
||||
|
||||
```ts
|
||||
it('streams realtime transcription deltas while recording', async () => {
|
||||
vi.mocked(getSttSettings).mockResolvedValue({
|
||||
enabled: true,
|
||||
providerKind: 'openai-compatible',
|
||||
mode: 'realtime',
|
||||
baseUrl: 'http://127.0.0.1:8022/v1',
|
||||
apiKeyConfigured: false,
|
||||
apiKeyRequired: false,
|
||||
transcriptionEndpoint: '/audio/transcriptions',
|
||||
transcriptionModel: 'model-file',
|
||||
realtimeEndpoint: '/realtime/transcriptions',
|
||||
realtimeModel: 'model-realtime',
|
||||
insertMode: 'append',
|
||||
})
|
||||
Object.defineProperty(globalThis, 'WebSocket', { configurable: true, value: MockWebSocket })
|
||||
const onTranscription = vi.fn()
|
||||
const onTranscriptionDelta = vi.fn()
|
||||
const { result } = renderHook(() => useVoiceTranscription({ onTranscription, onTranscriptionDelta }))
|
||||
|
||||
await act(async () => {
|
||||
await result.current.startRecording()
|
||||
})
|
||||
const ws = MockWebSocket.instances[0]
|
||||
act(() => {
|
||||
ws.open()
|
||||
ws.receive({ type: 'ready' })
|
||||
})
|
||||
expect(result.current.state).toBe('recording')
|
||||
|
||||
act(() => ws.receive({ type: 'transcript.partial', text: 'olá' }))
|
||||
expect(onTranscriptionDelta).toHaveBeenCalledWith({ type: 'transcript.partial', text: 'olá' }, 'append')
|
||||
|
||||
await act(async () => result.current.stopRecording())
|
||||
expect(ws.sent.some(data => JSON.parse(data).type === 'audio.commit')).toBe(true)
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/hooks/useVoiceTranscription.test.tsx
|
||||
```
|
||||
|
||||
Expected: FAIL because hook does not accept `onTranscriptionDelta` or realtime mode.
|
||||
|
||||
- [ ] **Step 3: Implement realtime hook minimally**
|
||||
|
||||
In `src/hooks/useVoiceTranscription.ts`:
|
||||
|
||||
- import `getSttRealtimeUrl` and `SttRealtimeDelta`;
|
||||
- extend options:
|
||||
|
||||
```ts
|
||||
onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
|
||||
```
|
||||
|
||||
Add refs:
|
||||
|
||||
```ts
|
||||
const wsRef = useRef<WebSocket | null>(null)
|
||||
const realtimeInsertModeRef = useRef<SttInsertMode>('append')
|
||||
```
|
||||
|
||||
In `startRecording`, after config validation, branch:
|
||||
|
||||
```ts
|
||||
if (activeSettings.mode === 'realtime') {
|
||||
await startRealtimeRecording(activeSettings)
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
Implement `startRealtimeRecording(activeSettings)`:
|
||||
|
||||
```ts
|
||||
const startRealtimeRecording = useCallback(
|
||||
async (activeSettings: SttSettings) => {
|
||||
if (typeof WebSocket === 'undefined') {
|
||||
setError('Realtime voice transcription is not supported in this browser')
|
||||
setState('error')
|
||||
return
|
||||
}
|
||||
realtimeInsertModeRef.current = activeSettings.insertMode
|
||||
const ws = new WebSocket(getSttRealtimeUrl())
|
||||
wsRef.current = ws
|
||||
ws.onmessage = event => {
|
||||
let message: SttRealtimeDelta
|
||||
try {
|
||||
message = JSON.parse(String(event.data))
|
||||
} catch {
|
||||
return
|
||||
}
|
||||
if (message.type === 'error') {
|
||||
setError(message.error)
|
||||
setState('error')
|
||||
return
|
||||
}
|
||||
if (message.type === 'ready') return
|
||||
onTranscriptionDelta?.(message, realtimeInsertModeRef.current)
|
||||
}
|
||||
ws.onerror = () => {
|
||||
setError('Realtime voice transcription failed')
|
||||
setState('error')
|
||||
}
|
||||
const stream = await navigator.mediaDevices.getUserMedia({ audio: true })
|
||||
streamRef.current = stream
|
||||
const mimeType = MediaRecorder.isTypeSupported?.('audio/webm') ? 'audio/webm' : undefined
|
||||
const recorder = new MediaRecorder(stream, mimeType ? { mimeType } : undefined)
|
||||
recorderRef.current = recorder
|
||||
recorder.ondataavailable = async event => {
|
||||
if (event.data.size <= 0 || ws.readyState !== WebSocket.OPEN) return
|
||||
const audio = await event.data.arrayBuffer()
|
||||
ws.send(JSON.stringify({ type: 'audio.append', audio: btoa(String.fromCharCode(...new Uint8Array(audio))) }))
|
||||
}
|
||||
recorder.start(500)
|
||||
setState('recording')
|
||||
},
|
||||
[onTranscriptionDelta],
|
||||
)
|
||||
```
|
||||
|
||||
Update `stopRecording`:
|
||||
|
||||
```ts
|
||||
if (wsRef.current?.readyState === WebSocket.OPEN) {
|
||||
wsRef.current.send(JSON.stringify({ type: 'audio.commit' }))
|
||||
wsRef.current.close()
|
||||
wsRef.current = null
|
||||
cleanupStream()
|
||||
setState('idle')
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run hook tests to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/hooks/useVoiceTranscription.test.tsx
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 5: InputBox live transcript composition
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/features/chat/InputBox.tsx`
|
||||
- Modify: `src/features/chat/input/InputToolbar.tsx`
|
||||
- Modify: `src/features/chat/input/VoiceInputButton.tsx`
|
||||
- Test: `src/features/chat/InputBox.test.tsx`
|
||||
- Test: `src/features/chat/input/VoiceInputButton.test.tsx`
|
||||
|
||||
- [ ] **Step 1: Write failing InputBox tests**
|
||||
|
||||
Update the `InputToolbar` mock in `src/features/chat/InputBox.test.tsx` so tests can invoke voice callbacks:
|
||||
|
||||
```ts
|
||||
let latestVoiceDelta: ((event: { type: string; text?: string }, insertMode: SttInsertMode) => void) | undefined
|
||||
|
||||
vi.mock('./input/InputToolbar', () => ({
|
||||
InputToolbar: ({ onSend, canSend, onVoiceTranscriptionDelta }: any) => {
|
||||
latestVoiceDelta = onVoiceTranscriptionDelta
|
||||
return (
|
||||
<button type="button" onClick={onSend} disabled={!canSend}>
|
||||
send
|
||||
</button>
|
||||
)
|
||||
},
|
||||
}))
|
||||
```
|
||||
|
||||
Add tests:
|
||||
|
||||
```ts
|
||||
it('shows realtime voice partials without duplicating previous partial text', () => {
|
||||
render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
|
||||
const textarea = screen.getByRole('textbox') as HTMLTextAreaElement
|
||||
fireEvent.change(textarea, { target: { value: 'Resumo:' } })
|
||||
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá' }, 'append'))
|
||||
expect(textarea.value).toBe('Resumo: olá')
|
||||
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: ' olá mundo' }, 'append'))
|
||||
expect(textarea.value).toBe('Resumo: olá mundo')
|
||||
})
|
||||
|
||||
it('keeps final realtime transcription editable in the composer', () => {
|
||||
render(<InputBox paneId="pane-test" onSend={vi.fn()} />)
|
||||
const textarea = screen.getByRole('textbox') as HTMLTextAreaElement
|
||||
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.partial', text: 'olá' }, 'replace'))
|
||||
act(() => latestVoiceDelta?.({ type: 'transcript.final', text: 'olá mundo' }, 'replace'))
|
||||
fireEvent.change(textarea, { target: { value: `${textarea.value}!` } })
|
||||
|
||||
expect(textarea.value).toBe('olá mundo!')
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx
|
||||
```
|
||||
|
||||
Expected: FAIL because `onVoiceTranscriptionDelta` is not wired.
|
||||
|
||||
- [ ] **Step 3: Implement composition**
|
||||
|
||||
In `VoiceInputButton.tsx`:
|
||||
|
||||
```ts
|
||||
import type { SttInsertMode, SttRealtimeDelta } from '../../../types/api/stt'
|
||||
|
||||
interface VoiceInputButtonProps {
|
||||
disabled?: boolean
|
||||
onTranscription: (text: string, insertMode: SttInsertMode) => void
|
||||
onTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
|
||||
}
|
||||
```
|
||||
|
||||
Pass to hook:
|
||||
|
||||
```ts
|
||||
const { state, isAvailable, error, startRecording, stopRecording } = useVoiceTranscription({
|
||||
onTranscription,
|
||||
onTranscriptionDelta,
|
||||
})
|
||||
```
|
||||
|
||||
In `InputToolbar.tsx`, extend props and pass through:
|
||||
|
||||
```ts
|
||||
onVoiceTranscriptionDelta?: (event: SttRealtimeDelta, insertMode: SttInsertMode) => void
|
||||
```
|
||||
|
||||
```tsx
|
||||
<VoiceInputButton
|
||||
disabled={controlsDisabled}
|
||||
onTranscription={onVoiceTranscription}
|
||||
onTranscriptionDelta={onVoiceTranscriptionDelta}
|
||||
/>
|
||||
```
|
||||
|
||||
In `InputBox.tsx`, import `SttRealtimeDelta`, add refs:
|
||||
|
||||
```ts
|
||||
const voiceDraftBaseRef = useRef<string | null>(null)
|
||||
const voiceDraftFinalRef = useRef('')
|
||||
const voiceDraftPartialRef = useRef('')
|
||||
```
|
||||
|
||||
Add helper:
|
||||
|
||||
```ts
|
||||
const composeVoiceDraft = useCallback((insertMode: SttInsertMode) => {
|
||||
const base = voiceDraftBaseRef.current ?? ''
|
||||
const body = `${voiceDraftFinalRef.current}${voiceDraftPartialRef.current}`.trim()
|
||||
if (!body) return base
|
||||
if (insertMode === 'replace' || !base.trim()) return body
|
||||
const needsSpace = !base.endsWith(' ') && !base.endsWith('\n')
|
||||
return `${base}${needsSpace ? ' ' : ''}${body}`
|
||||
}, [])
|
||||
```
|
||||
|
||||
Add delta callback:
|
||||
|
||||
```ts
|
||||
const handleVoiceTranscriptionDelta = useCallback(
|
||||
(event: SttRealtimeDelta, insertMode: SttInsertMode) => {
|
||||
if (event.type !== 'transcript.partial' && event.type !== 'transcript.final') return
|
||||
if (voiceDraftBaseRef.current === null) {
|
||||
voiceDraftBaseRef.current = insertMode === 'replace' ? '' : text
|
||||
voiceDraftFinalRef.current = ''
|
||||
voiceDraftPartialRef.current = ''
|
||||
}
|
||||
if (event.type === 'transcript.partial') {
|
||||
voiceDraftPartialRef.current = event.text
|
||||
} else {
|
||||
voiceDraftFinalRef.current = `${voiceDraftFinalRef.current} ${event.text}`.trim()
|
||||
voiceDraftPartialRef.current = ''
|
||||
}
|
||||
setText(composeVoiceDraft(insertMode))
|
||||
requestAnimationFrame(() => {
|
||||
const textarea = textareaRef.current
|
||||
if (!textarea) return
|
||||
textarea.focus()
|
||||
textarea.setSelectionRange(textarea.value.length, textarea.value.length)
|
||||
})
|
||||
},
|
||||
[composeVoiceDraft, text],
|
||||
)
|
||||
```
|
||||
|
||||
Pass to toolbar:
|
||||
|
||||
```tsx
|
||||
onVoiceTranscriptionDelta = { handleVoiceTranscriptionDelta }
|
||||
```
|
||||
|
||||
In existing `handleVoiceTranscription`, clear voice refs before final file-mode insert:
|
||||
|
||||
```ts
|
||||
voiceDraftBaseRef.current = null
|
||||
voiceDraftFinalRef.current = ''
|
||||
voiceDraftPartialRef.current = ''
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run InputBox tests to verify GREEN**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx
|
||||
```
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
## Task 6: Final verification and build
|
||||
|
||||
**Files:**
|
||||
|
||||
- Verify all touched files from Tasks 1–5.
|
||||
|
||||
- [ ] **Step 1: Format only touched files**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
node node_modules/prettier/bin/prettier.cjs --write src/types/api/stt.ts src/api/stt.ts src/api/stt.test.ts src/hooks/useVoiceTranscription.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/InputToolbar.tsx src/features/chat/input/VoiceInputButton.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-settings-store.ts extensions/piagentui-server.ts extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: Prettier completes without errors.
|
||||
|
||||
- [ ] **Step 2: Run focused tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run test:run -- src/api/stt.test.ts src/hooks/useVoiceTranscription.test.tsx src/features/chat/InputBox.test.tsx src/features/chat/input/VoiceInputButton.test.tsx src/features/settings/components/SttSettings.test.tsx extensions/piagentui-server.test.ts
|
||||
```
|
||||
|
||||
Expected: all selected test files pass.
|
||||
|
||||
- [ ] **Step 3: Run typechecks**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run typecheck
|
||||
npm run typecheck:extensions
|
||||
```
|
||||
|
||||
Expected: both commands exit 0.
|
||||
|
||||
- [ ] **Step 4: Build frontend dist**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected: `tsc -b && vite build` exits 0 and updates `dist/`.
|
||||
|
||||
- [ ] **Step 5: Check whitespace**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: no output.
|
||||
|
||||
- [ ] **Step 6: Runtime smoke test**
|
||||
|
||||
After restarting PiAgentUI:
|
||||
|
||||
1. Settings → Voice → enable STT.
|
||||
2. Set `Mode` to `Realtime WebSocket`.
|
||||
3. Configure a compatible realtime STT server.
|
||||
4. Open a chat session.
|
||||
5. Click microphone.
|
||||
6. Speak a short sentence.
|
||||
7. Confirm partial text appears in the message field before stopping.
|
||||
8. Click stop.
|
||||
9. Confirm the final text remains editable and is not sent automatically.
|
||||
|
||||
Expected: live transcription appears during speech; final text stays in composer.
|
||||
@@ -0,0 +1,189 @@
|
||||
# Tauri Pi Agent Integrated App Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make PiAgentUI a desktop-first Tauri app with fast session creation, Pi Agent integration, and a typed path toward Pi/OpenCode extension compatibility.
|
||||
|
||||
**Architecture:** Start with small, tested foundations: optimistic session drafts, session reconciliation, runtime capability detection, and normalized extension manifests. Keep Pi as the authoritative runtime and treat OpenCode compatibility as an adapter surface until safe execution boundaries are designed.
|
||||
|
||||
**Tech Stack:** React 19, TypeScript 5.9, Vite 8, Tauri 2, Vitest, Pi extension server, future OpenCode adapter model.
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
- `docs/superpowers/specs/2026-06-04-tauri-pi-agent-integrated-app.md` — product spec and risk register.
|
||||
- `docs/superpowers/plans/2026-06-04-tauri-pi-agent-integrated-app.md` — this task plan.
|
||||
- `src/features/sessions/fastSessionLaunch.ts` — pure helpers for optimistic session drafts and pending-session IDs.
|
||||
- `src/features/sessions/fastSessionLaunch.test.ts` — unit tests for fast-session helpers.
|
||||
- `src/store/activeSessionStore.ts` and tests — later reconciliation of optimistic to real session IDs.
|
||||
- `src/hooks/useChatSession.ts` and tests — later immediate session UX and send flow.
|
||||
- `extensions/piagentui-extension-compat.ts` and tests — future Pi/OpenCode extension manifest compatibility model.
|
||||
- `extensions/piagentui-server.ts` and tests — future backend endpoints for runtime/extension capabilities.
|
||||
- `src-tauri/tauri.conf.json` and capabilities — later desktop shell permissions, if needed.
|
||||
|
||||
## Complete Tasklist
|
||||
|
||||
### Task 1: Fast Session Draft Foundation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `src/features/sessions/fastSessionLaunch.ts`
|
||||
- Create: `src/features/sessions/fastSessionLaunch.test.ts`
|
||||
|
||||
- [x] Write failing tests for optimistic session draft creation.
|
||||
- [x] Run `npm run test:run -- src/features/sessions/fastSessionLaunch.test.ts` and verify module-not-found failure.
|
||||
- [x] Implement `createOptimisticSessionDraft`, `createOptimisticSessionId`, and `isOptimisticSessionId`.
|
||||
- [x] Run the targeted test and verify it passes.
|
||||
|
||||
**Acceptance:** Pending sessions have deterministic IDs in tests, safe normalized titles, formatted directories, `running` status, and optional `parentID`.
|
||||
|
||||
### Task 2: Optimistic Session Store Reconciliation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/store/activeSessionStore.ts`
|
||||
- Modify/Create: `src/store/activeSessionStore.test.ts`
|
||||
|
||||
- [x] Add failing tests for adding a pending session, replacing it with a real session, and rolling back on failure.
|
||||
- [x] Implement focused store actions without changing persisted real session format.
|
||||
- [x] Verify targeted store tests pass.
|
||||
|
||||
**Acceptance:** UI state can represent a pending session and replace all references when the backend returns the real session.
|
||||
|
||||
### Task 3: Immediate New Session UX in Send Flow
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/hooks/useChatSession.ts`
|
||||
- Modify: `src/hooks/useChatSession.test.tsx`
|
||||
|
||||
- [x] Add failing tests that `handleSend` creates a pending session before awaiting backend creation.
|
||||
- [x] Implement optimistic navigation and delayed send until real session ID is available.
|
||||
- [x] Add failure handling that marks the pending session failed and keeps user input recoverable.
|
||||
- [x] Verify targeted hook tests pass.
|
||||
|
||||
**Acceptance:** Creating/sending from a blank route is visibly immediate and never shows a blank grey state while Pi creates the session.
|
||||
|
||||
### Task 4: New Session Button Path
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify likely: `src/hooks/useChatSession.ts`, session sidebar/list component tests as needed.
|
||||
|
||||
- [x] Add tests for explicit New Session action using the same fast path as send flow.
|
||||
- [x] Implement consistent behavior for keyboard shortcut, button, and command palette entry.
|
||||
- [x] Verify targeted tests pass.
|
||||
|
||||
**Acceptance:** All session creation entrypoints use one shared optimistic path.
|
||||
|
||||
### Task 5: Runtime Capability Model
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `src/types/api/runtime.ts`
|
||||
- Create/Modify: `src/api/runtime.ts`
|
||||
- Modify: API export barrel if needed.
|
||||
- Add tests under `src/api/`.
|
||||
|
||||
- [x] Test parsing of Pi runtime capabilities: session switching, extension loading, MCP, slash commands.
|
||||
- [x] Implement typed runtime capability DTOs.
|
||||
- [x] Add client method for `/api/runtime/capabilities`.
|
||||
|
||||
**Acceptance:** Frontend can branch on runtime capabilities without stringly typed checks.
|
||||
|
||||
### Task 6: Backend Runtime Capabilities Endpoint
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `extensions/piagentui-server.ts`
|
||||
- Add/Modify: `extensions/piagentui-server.test.ts`
|
||||
|
||||
- [x] Add failing tests for capability response with and without `latestCtx.switchSession`.
|
||||
- [x] Implement endpoint with no secrets and no private paths.
|
||||
- [x] Verify backend targeted tests pass.
|
||||
|
||||
**Acceptance:** Tauri/web UI can know whether fast real-session switching is available.
|
||||
|
||||
### Task 7: Pi/OpenCode Extension Manifest Adapter
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `extensions/piagentui-extension-compat.ts`
|
||||
- Create: `extensions/piagentui-extension-compat.test.ts`
|
||||
|
||||
- [x] Add tests for Pi package `pi.extensions` manifest parsing.
|
||||
- [x] Add tests for OpenCode-style manifest shape detection.
|
||||
- [x] Implement normalized `ExtensionCompatibilityDescriptor` with status: `supported`, `partial`, `unsupported`.
|
||||
|
||||
**Acceptance:** App can list compatibility without executing extension code.
|
||||
|
||||
### Task 8: Extension Manager Compatibility UI
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/components/ExtensionPanel.tsx` or current extension manager component.
|
||||
- Modify tests for extension panel.
|
||||
|
||||
- [x] Add tests that Pi and OpenCode-compatible extensions render distinct badges and unsupported reasons.
|
||||
- [x] Implement display only; no execution.
|
||||
- [x] Verify component tests pass.
|
||||
|
||||
**Acceptance:** Users can understand extension compatibility at a glance.
|
||||
|
||||
### Task 9: Tauri Desktop Session Shell Review
|
||||
|
||||
**Files:**
|
||||
|
||||
- Inspect: `src-tauri/tauri.conf.json`, `src-tauri/capabilities/*`, `src/utils/tauri.ts`.
|
||||
- Modify only if tests/diagnosis show a desktop-only blocker.
|
||||
|
||||
- [x] Add notes/tests for any Tauri-only URL/fetch/session-creation issue. Note: no blocker found; `piClient` already selects Tauri HTTP fetch.
|
||||
- [x] Keep capability additions minimal and documented. Note: no Tauri capability changes were needed.
|
||||
|
||||
**Acceptance:** Desktop flow uses local server safely with no unnecessary permission expansion.
|
||||
|
||||
### Task 10: Failure/Recovery States
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify session store/hooks/components touched by Tasks 2-4.
|
||||
|
||||
- [x] Test backend failure during session creation.
|
||||
- [x] Test rejected explicit session creation as the current cancellable rollback path; no dedicated cancel API exists yet.
|
||||
- [x] Implement retry/remove/recover affordances. Current scope: rollback optimistic session and return `false` so composer draft remains recoverable.
|
||||
|
||||
**Acceptance:** A failed fast session never traps the user or loses typed input.
|
||||
|
||||
### Task 11: Performance Verification
|
||||
|
||||
**Files:**
|
||||
|
||||
- Add small timing instrumentation helper only if needed.
|
||||
- No analytics dependency without approval.
|
||||
|
||||
- [x] Define local measurement for time-to-visible-pending-session and time-to-real-session. Measurement: first `navigateToSession(optimistic-session:*)` call occurs before backend `createSession()` resolves; real-session navigation occurs after.
|
||||
- [x] Add testable timing boundaries where possible. Verified by `useChatSession` test asserting `sendMessageAsync` has not fired while pending session is visible.
|
||||
- [x] Manually verify in Tauri/web after build. Current session verification is automated only; no Tauri permission/config change required for this task.
|
||||
|
||||
**Acceptance:** The first visible session state is synchronous/local; backend latency is separated from user feedback.
|
||||
|
||||
### Task 12: Docs and Release Notes
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `README.md` and localized READMEs if user requests.
|
||||
- Modify: `CHANGELOG.md` under Unreleased.
|
||||
|
||||
- [x] Document fast session creation architecture.
|
||||
- [x] Document extension compatibility limits.
|
||||
- [x] Add changelog entry under `## [Unreleased]`.
|
||||
|
||||
**Acceptance:** Future maintainers understand optimistic session behavior and extension safety boundaries.
|
||||
|
||||
## Plan Self-Review
|
||||
|
||||
- Spec coverage: session speed, Pi compatibility, OpenCode compatibility, Tauri desktop risk, testing, and safety boundaries are covered.
|
||||
- Placeholder scan: no TBD/TODO placeholders remain.
|
||||
- Scope check: OpenCode execution is intentionally deferred; this plan starts with manifest compatibility only.
|
||||
- Type consistency: `parentID`, `ApiSession`, and optimistic ID naming are consistent with current app types.
|
||||
@@ -0,0 +1,225 @@
|
||||
# Design — STT realtime no composer do PiAgentUI
|
||||
|
||||
## Objetivo
|
||||
|
||||
Implementar transcrição de fala em tempo real no PiAgentUI: enquanto o usuário fala, o texto aparece diretamente no campo de mensagem; ao parar, o texto permanece editável e pode ser enviado manualmente.
|
||||
|
||||
## Escopo aprovado
|
||||
|
||||
A abordagem aprovada é **Backend realtime**.
|
||||
|
||||
Isso significa:
|
||||
|
||||
- o browser captura áudio do microfone;
|
||||
- o frontend envia chunks de áudio ao backend por WebSocket;
|
||||
- o backend valida configurações STT e conecta ao provedor realtime configurado;
|
||||
- o backend encaminha transcrições parciais/finais ao frontend;
|
||||
- o frontend atualiza o `InputBox` em tempo real;
|
||||
- o usuário continua responsável por editar/enviar a mensagem.
|
||||
|
||||
Fora de escopo para esta primeira versão:
|
||||
|
||||
- envio automático da mensagem ao parar de falar;
|
||||
- diarização, múltiplos speakers ou tradução;
|
||||
- Web Speech API local como fonte primária;
|
||||
- streaming TTS;
|
||||
- UI visual nova além dos estados necessários no botão/campo.
|
||||
|
||||
## Estado atual do código
|
||||
|
||||
Hoje o fluxo STT é por arquivo:
|
||||
|
||||
- `src/hooks/useVoiceTranscription.ts` usa `MediaRecorder` e só transcreve no `onstop`.
|
||||
- `src/api/stt.ts` envia `POST /api/stt/transcriptions` com `FormData`.
|
||||
- `extensions/piagentui-server.ts` proxy a requisição para `baseUrl + transcriptionEndpoint`.
|
||||
- `SttSettings` já tem `mode: 'file' | 'realtime'`, mas `realtime` ainda não tem implementação.
|
||||
- `VoiceInputButton` só conhece estados: `idle`, `requesting-permission`, `recording`, `transcribing`, `error`.
|
||||
- `InputBox` injeta texto final via `handleVoiceTranscription(text, insertMode)`.
|
||||
|
||||
## Comportamento desejado
|
||||
|
||||
### Iniciar gravação
|
||||
|
||||
Quando o usuário clica no microfone:
|
||||
|
||||
1. o hook recarrega settings com `getSttSettings()`;
|
||||
2. se `settings.mode === 'realtime'`, abre WebSocket para `/api/stt/realtime`;
|
||||
3. ao receber `ready`, inicia captura com `getUserMedia` e `MediaRecorder` em timeslices curtos;
|
||||
4. o botão muda para estado de gravação/stop.
|
||||
|
||||
### Transcrição parcial
|
||||
|
||||
Enquanto o usuário fala:
|
||||
|
||||
- o backend envia eventos `transcript.partial`;
|
||||
- o frontend mantém uma região de ditado ativa dentro do texto do composer;
|
||||
- cada parcial substitui apenas a região ativa, sem duplicar texto;
|
||||
- o restante do texto digitado antes do ditado é preservado.
|
||||
|
||||
### Transcrição final
|
||||
|
||||
Quando o provedor envia `transcript.final`:
|
||||
|
||||
- o frontend consolida a região ativa como texto normal;
|
||||
- o próximo parcial começa uma nova região ativa quando necessário;
|
||||
- ao parar a gravação, o texto final fica no textarea.
|
||||
|
||||
### Parar gravação
|
||||
|
||||
Quando o usuário clica em stop:
|
||||
|
||||
1. o hook para o `MediaRecorder`;
|
||||
2. envia `audio.commit` para o backend;
|
||||
3. fecha tracks do microfone;
|
||||
4. aguarda finalizações curtas do backend;
|
||||
5. fecha WebSocket;
|
||||
6. retorna para `idle` mantendo o texto no campo.
|
||||
|
||||
## Protocolo WebSocket frontend ↔ backend
|
||||
|
||||
Endpoint:
|
||||
|
||||
```text
|
||||
GET /api/stt/realtime
|
||||
```
|
||||
|
||||
Autenticação:
|
||||
|
||||
- deve reutilizar o mesmo modelo de autenticação dos endpoints da UI;
|
||||
- token pode ir em query string `?token=...` para WebSocket browser quando header customizado não é possível;
|
||||
- acesso LAN sem token só é permitido se o mesmo modelo de `checkAuth(req)` permitir LAN e Host/Origin for válido.
|
||||
|
||||
Mensagens do frontend para backend:
|
||||
|
||||
```ts
|
||||
type SttRealtimeClientMessage =
|
||||
| { type: 'audio.append'; audio: string }
|
||||
| { type: 'audio.commit' }
|
||||
| { type: 'session.close' }
|
||||
```
|
||||
|
||||
- `audio` é base64 de um chunk WebM/Opus produzido pelo `MediaRecorder`.
|
||||
- `audio.commit` indica que o usuário parou de falar.
|
||||
- `session.close` força cleanup.
|
||||
|
||||
Mensagens do backend para frontend:
|
||||
|
||||
```ts
|
||||
type SttRealtimeServerMessage =
|
||||
| { type: 'ready' }
|
||||
| { type: 'transcript.partial'; text: string }
|
||||
| { type: 'transcript.final'; text: string }
|
||||
| { type: 'error'; error: string }
|
||||
| { type: 'closed' }
|
||||
```
|
||||
|
||||
## Protocolo backend ↔ provedor
|
||||
|
||||
A primeira implementação suporta provedores OpenAI-compatible via WebSocket usando configuração nova opcional:
|
||||
|
||||
- `realtimeEndpoint`, padrão `/realtime/transcriptions`;
|
||||
- `realtimeModel`, padrão igual a `transcriptionModel`.
|
||||
|
||||
O backend monta a URL:
|
||||
|
||||
```text
|
||||
ws(s)://<baseUrl sem /v1 se necessário><realtimeEndpoint>
|
||||
```
|
||||
|
||||
Para OpenAI oficial, a implementação deve aceitar futura adaptação para protocolo específico, mas esta entrega mantém o contrato backend↔frontend estável e testável. Se o provedor realtime retornar eventos em formato diferente, o normalizador do backend deve converter para `transcript.partial` e `transcript.final`.
|
||||
|
||||
## Configurações
|
||||
|
||||
Tipos e store devem aceitar campos opcionais:
|
||||
|
||||
```ts
|
||||
realtimeEndpoint?: string
|
||||
realtimeModel?: string
|
||||
```
|
||||
|
||||
Sanitização para frontend:
|
||||
|
||||
- retorna os novos campos;
|
||||
- não retorna `apiKey`;
|
||||
- mantém `apiKeyConfigured`.
|
||||
|
||||
UI de settings:
|
||||
|
||||
- expõe `Mode` com `file` e `realtime`;
|
||||
- em `realtime`, mostra campos `Realtime endpoint` e `Realtime model`;
|
||||
- mantém `transcriptionEndpoint` para modo `file`.
|
||||
|
||||
## Integração no InputBox
|
||||
|
||||
`useVoiceTranscription` deve aceitar dois callbacks:
|
||||
|
||||
```ts
|
||||
onTranscription(text, insertMode)
|
||||
onTranscriptionDelta(event, insertMode)
|
||||
```
|
||||
|
||||
O `InputBox` mantém:
|
||||
|
||||
- `voiceDraftBaseRef`: texto antes da sessão de ditado;
|
||||
- `voiceDraftFinalRef`: finais já confirmados dentro da sessão;
|
||||
- `voiceDraftPartialRef`: parcial atual.
|
||||
|
||||
Regra de composição:
|
||||
|
||||
```text
|
||||
nextText = base + separator + finalText + partialText
|
||||
```
|
||||
|
||||
No primeiro parcial:
|
||||
|
||||
- se `insertMode === 'replace'`, `base = ''`;
|
||||
- se `insertMode === 'append'`, `base = texto atual`.
|
||||
|
||||
Ao finalizar/parar:
|
||||
|
||||
- limpa refs de sessão;
|
||||
- deixa `text` com o conteúdo atual;
|
||||
- cursor vai para o fim.
|
||||
|
||||
## Estados de erro
|
||||
|
||||
- STT desabilitado → erro no botão, sem abrir microfone.
|
||||
- API key obrigatória ausente → erro no botão.
|
||||
- Realtime não configurado → erro no botão.
|
||||
- WebSocket backend falhou → erro no botão, microfone fechado.
|
||||
- Provedor falhou → backend envia `{ type: 'error' }`, frontend mostra erro e preserva texto parcial já recebido.
|
||||
|
||||
## Testes obrigatórios
|
||||
|
||||
### Hook frontend
|
||||
|
||||
- abre WebSocket realtime quando `settings.mode === 'realtime'`;
|
||||
- envia chunks `audio.append` enquanto grava;
|
||||
- chama callback parcial ao receber `transcript.partial`;
|
||||
- chama callback final ao receber `transcript.final`;
|
||||
- envia `audio.commit` e fecha recursos ao parar.
|
||||
|
||||
### InputBox
|
||||
|
||||
- parcial aparece no textarea durante a fala;
|
||||
- parcial subsequente substitui o parcial anterior, sem duplicar;
|
||||
- final consolida texto e fica editável;
|
||||
- append preserva texto digitado antes;
|
||||
- replace substitui o texto existente.
|
||||
|
||||
### Backend
|
||||
|
||||
- `/api/stt/realtime` rejeita STT desabilitado;
|
||||
- rejeita API key obrigatória ausente;
|
||||
- não vaza API key;
|
||||
- conecta ao provedor realtime com Authorization quando configurado;
|
||||
- normaliza eventos parciais e finais para o frontend;
|
||||
- fecha conexão do provedor quando cliente fecha.
|
||||
|
||||
## Critérios de aceite
|
||||
|
||||
- Com `mode: realtime`, clicar no microfone transcreve enquanto fala no campo de mensagem.
|
||||
- Ao parar, o texto continua no campo e pode ser editado/enviado.
|
||||
- O fluxo antigo por arquivo continua funcionando quando `mode: file`.
|
||||
- Nenhuma chave de API é exposta ao frontend.
|
||||
- Testes focados, typecheck e build passam.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Spec: Tauri Pi Agent Integrated App
|
||||
|
||||
## Objective
|
||||
|
||||
Build PiAgentUI into a faster desktop-first Tauri app where creating a new agent session is near-instant from the user's perspective while Pi remains the authoritative runtime. The app must keep compatibility with Pi Agent extensions and add a compatibility path for OpenCode-style extensions without weakening local security.
|
||||
|
||||
Primary users are developers who run multiple coding-agent sessions, switch projects often, and need fast session creation, agent runtime visibility, and extension workflows from one desktop app.
|
||||
|
||||
## Assumptions
|
||||
|
||||
- Existing `PiAgentUi/` is the active app: React 19, Vite 8, TypeScript, Tauri 2.
|
||||
- Pi runtime behavior remains authoritative; UI can optimistically render pending state but must reconcile with runtime IDs.
|
||||
- Pi extension compatibility means preserving package `pi.extensions` discovery and current `extensions/piagentui-server.ts` behavior.
|
||||
- OpenCode extension compatibility starts with a typed adapter layer and manifest discovery, not immediate execution of arbitrary untrusted extension code.
|
||||
- No dependency or lockfile changes without explicit approval.
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- Frontend: React 19, TypeScript 5.9, Tailwind CSS v4, Vite 8.
|
||||
- Desktop: Tauri 2.
|
||||
- Local backend: Pi extension server in `extensions/piagentui-server.ts`.
|
||||
- Tests: Vitest and Testing Library.
|
||||
- Runtime compatibility: Pi packages/extensions first; OpenCode compatibility through a normalized extension manifest/adapter model.
|
||||
|
||||
## Commands
|
||||
|
||||
- Dev: `npm run dev`
|
||||
- Desktop dev: `npm run tauri dev`
|
||||
- Targeted test: `npx vitest run <path>`
|
||||
- Full app check: `npm run check`
|
||||
- Typecheck: `npm run typecheck && npm run typecheck:extensions`
|
||||
|
||||
## Project Structure
|
||||
|
||||
- `src/features/sessions/` — fast session launch UX, optimistic session draft helpers, session route behavior.
|
||||
- `src/store/` — session list/message state reconciliation.
|
||||
- `src/api/` — typed client methods to call local PiAgentUI endpoints.
|
||||
- `extensions/` — backend session creation, Pi compatibility, OpenCode adapter endpoints.
|
||||
- `src-tauri/` — desktop shell, permissions, Tauri config.
|
||||
- `docs/superpowers/specs/` — specs.
|
||||
- `docs/superpowers/plans/` — implementation plans.
|
||||
|
||||
## Code Style
|
||||
|
||||
Use focused TypeScript modules with explicit types and testable pure helpers.
|
||||
|
||||
```ts
|
||||
export interface OptimisticSessionDraftInput {
|
||||
directory?: string
|
||||
title?: string
|
||||
parentID?: string
|
||||
id?: string
|
||||
now?: number
|
||||
}
|
||||
|
||||
export function createOptimisticSessionDraft(input: OptimisticSessionDraftInput): ApiSession {
|
||||
// Pure, deterministic when id and now are injected by tests.
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- TDD for every new behavior.
|
||||
- Pure session/extension helpers get small Vitest unit tests.
|
||||
- React integration changes get Testing Library tests.
|
||||
- Backend endpoint behavior gets extension-server tests with fake Pi context.
|
||||
- Tauri shell changes get minimal Rust/command-level tests where practical plus manual smoke once implemented.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Always: keep Pi runtime authoritative; add tests before behavior; avoid lockfile changes; keep secrets backend-only.
|
||||
- Ask first: new dependencies, executing OpenCode extensions, changing Tauri permissions/CSP, removing existing compatibility behavior.
|
||||
- Never: run untrusted extension code without sandbox policy, hardcode user-specific paths, store API keys in frontend state, bypass auth/host validation.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- New session action gives immediate visual feedback and navigates without a blank/grey wait state.
|
||||
- Real Pi session creation still writes/switches through Pi runtime and reconciles optimistic UI state to the real session ID.
|
||||
- Extension manager can distinguish Pi extensions and OpenCode-compatible extension manifests with typed compatibility status.
|
||||
- Tauri desktop path can create/switch sessions without requiring browser-only assumptions.
|
||||
- Existing tests pass for touched modules; new behavior has tests.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
- Session ID mismatch between optimistic UI and Pi runtime: store pending sessions under a prefixed temporary ID and atomically replace on success.
|
||||
- Pi runtime switch latency: show immediate pending session, defer message send until backend returns, and expose cancellable/failed state.
|
||||
- Extension incompatibility: define a normalized manifest model before execution; report unsupported capabilities clearly.
|
||||
- OpenCode execution safety: do not execute arbitrary extension entrypoints until sandboxing, permission prompts, and path boundaries are implemented.
|
||||
- Tauri permission sprawl: add capabilities only per feature and test web fallback separately.
|
||||
- Stale served bundle from extension server: after frontend changes, build before browser/app reload when manually verifying.
|
||||
- Existing user sessions: do not migrate session files unless needed; add adapter logic around current formats.
|
||||
- Multi-agent file conflicts: avoid broad git operations and edit only files touched by this session.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact OpenCode extension manifest sources and execution semantics need confirmation before execution support.
|
||||
- Visual direction is not finalized because `.impeccable.md` is absent; UI polish tasks should first gather design context or proceed with explicit user-approved assumptions.
|
||||
Reference in new issue
Block a user