Realtime voice and planning services for a caregiving prototype

Separate planning and voice runtimes

The planning backend assembles goals, tasks, and memory over HTTP. A separate phone service handles network delay, transcoding, model response time, and playback for each audio chunk.

System architecture

Bounded queues and gated interruption

Incoming Twilio μ-law is decoded and resampled from 8 kHz to 16 kHz PCM. Gemini’s 24 kHz PCM response is downsampled and encoded back to μ-law. Input and output use separate bounded asyncio queues: the response queue holds roughly 1.25 seconds of audio; the input queue holds about three seconds. Bounding both prevents a slow downstream consumer from turning a live call into delayed playback.

Barge-in is gated in two ways. A cooldown ignores audio immediately after text-to-speech begins, and WebRTC VAD must observe sustained voiced frames before interruption is allowed. The production constructor sets a 250 ms minimum, which is long enough to reject many clicks and bursts without making a real interruption feel ignored.

def ok_to_barge(self, chunk: bytes) -> bool:
    if int(time.time() * 1000) - self._tts_since < self.cooldown_ms:
        return False
    if len(chunk) < self.frame_bytes:
        return False

    try:
        voiced = self.vad.is_speech(chunk[:self.frame_bytes], self.sr)
    except Exception:
        return False

    self._active = self._active + 1 if voiced else 0
    return self._active >= self.min_frames

The adapter also tracks connection state, reconnect attempts, audio bytes, received chunks, interruptions, completed turns, call duration, and average send bitrate.

State and tool boundaries

The agent called tools to read and update profile information, generate hierarchical care goals, create short tasks, record completion, build daily plans, save moods, and produce recaps. The FastAPI layer returned function-call traces so the mobile client and logs could distinguish a model message from a state-changing operation.

Cross-session continuity used Vertex AI Memory Bank, while Firestore stored product truth: user/session mappings, goals, tasks, category order, daily selections, mood entries, and recaps. This separation avoided asking semantic retrieval to behave like a transactional database.

Deployment and smoke checks

The backend shipped as containers through Cloud Build, Artifact Registry, and Cloud Run. API Gateway validates Firebase secure tokens before forwarding requests. Firestore emulator and mock-LLM smoke paths made it possible to exercise basic routes without consuming live model traffic.

CareCorgi was accepted into the Google for Startups Cloud Program and received approximately $20,000 per month in Google Cloud credits.

Failure handling

• Unbounded audio buffers add playback latency. Input and output queues have fixed capacity and explicit backpressure behavior.
• Single-frame VAD triggers on hiss or echo. A cooldown and sustained-speech threshold gate interruption.
• Transactional state belongs in Firestore. Memory Bank handles semantic retrieval.
• Mobile and backend schemas can drift. Pydantic request and response models reject malformed calls.
• Long-lived calls retain tasks after disconnects. Per-call ownership and stale-session cleanup release them.

My scope

I was the founding engineer responsible for the Python agent backend and phone service: ADK integration, memory and planning services, FastAPI contracts, Firestore persistence, function tools, Twilio/Gemini Live bridge, audio conversion, VAD-gated barge-in, metrics, deployment assets, and mobile/backend integration. The React Native mobile application was primarily built by another team member.