Private AI phone receptionist
A phone receptionist for a real estate agency. It answers every call, says it's an AI, answers listing questions from the agency's own data, qualifies the caller, books a showing and transfers to a person when it should. Every call is recorded, scored and reviewed, so the agent gets measurably better.
- search_listings
- book_showing
- capture_lead
- transfer_to_agent
- Tools
- 9
- Scored on
- 6 axes
- Barge-in
- 2 words
- Target reply
- ≤ 1 s
Target reply is the pilot gate for median voice-to-voice latency, not a measured result.
The challenge
Fast enough to feel human. Private enough to trust.
A phone agent has about a second to reply before a caller starts talking over it — and it has to cope gracefully when they do.
Callers share names, numbers and addresses, so the system had to work without sending call audio or transcripts to a third-party cloud.
What it does on a call
Natural turn-taking
Noise suppression, voice detection and a turn-taking model that listens to the audio, not just silence. “Mm-hm” doesn't interrupt; “wait, what price?” does — and the agent keeps only what it actually said in its memory.
Grounded answers
Nine tools: search listings, get a listing, check and book showing slots, capture a lead, send an SMS, transfer, set recording consent and end the call. Prices, availability and features come only from tool results.
Compliance built in
AI disclosure and a recording notice in the first sentence, Fair Housing rules in the system prompt, and no legal, tax or mortgage advice — checked on every call by the reviewer.
A full record of every call
Stereo recordings, timestamped turns and interruptions, every tool call with its arguments and timing, latency breakdowns and the lead — with caller numbers encrypted at rest.
Review and regression
A reviewer model scores six dimensions and flags hallucinations. Low scores, transfers, drops and a 10% random sample go to people, whose corrections become a regression suite.
Private by default, hybrid by choice
Speech-to-text, the language model and the voice run locally. An optional gateway can use Claude for reasoning, with names, numbers and addresses replaced by placeholders before any text leaves.
Pilot gates
Agreed before the first live call, and published so the results can be judged against them.
Speech recognition
Word error rate of 10% or less on phone audio.
Task completion
85% of calls or more end with the caller's goal met.
Hallucination
3% of calls or fewer contain an ungrounded claim.
Naturalness
3.8 out of 5 or better from reviewers.
Response time
A median voice-to-voice reply of one second or less.
Built with
- Pipecat
- NVIDIA Riva
- vLLM · gpt-oss-20b
- Kokoro TTS
- Silero VAD
- Smart Turn
- Postgres + pgvector
- Twilio / Telnyx
- Claude (optional)
Where it stands
Code complete and unit-tested across the agent, the post-call worker, the privacy gateway and the review app.
Next: the first run on DGX Spark hardware with a live number, then a two-week pilot measured against the gates above.