AI 101 — What is an AI voice agent?

An AI voice agent is software that holds a spoken conversation on its own: it hears the caller, works out what they want, does something about it — looks up the order, books the slot, takes the payment — and answers out loud, with no human on the line.

That last clause is what separates it from the things you already know. A voice assistant answers one command at a time and forgets you. A phone tree plays a recording and waits for a keypress. A voice agent takes a goal — "move my Thursday flight to Friday, aisle seat" — and works through the turns until the goal is met or it hands you to a person.

Why it matters right now

Salesforce has agreed to buy Cresta, the contact-center AI company that started at Stanford, and Salesforce's own announcement confirms the deal and the customer list — Marriott, Comcast, Alaska Airlines — without naming a price. The Information put the price at $2.7 billion. Two point seven billion dollars for a company whose main product is a voice on a phone line is the clearest signal yet that this stopped being a demo category.

The technology underneath changed shape in the same window. In August, OpenAI's engineers described rebuilding their voice system around a model that listens and speaks at the same time, removing the separate turn detector from the audio path so that reasoning and tool calls happen off to the side without freezing the conversation. Google answered a month later with Gemini 3.8 Live, claiming the top spot on speech quality at roughly a sixth of the price, as we covered in Gemini 3.8 Live tops voice benchmarks at a sixth of GPT-Live's price.

Then the reality check. On the agentic banking benchmark Google cited in that same launch, the best voice model completed about a third of the tasks. A third. The listening and the talking are solved enough to sell. The doing is not.

The mental model

Four jobs run inside a voice agent, and only three of them are obvious.

Hear. Speech-to-text turns the caller's audio into words, streaming partial results while they are still talking. Think. A language model reads the transcript, decides what to do, and calls tools — a calendar, a billing API, a knowledge base. Speak. Text-to-speech turns the reply back into audio, sentence by sentence, so the first words play while the rest is still being generated.

Decide when the caller is finished. This is the job with no equivalent in a text chatbot, and it is where most of the disappointment lives. The agent has to distinguish a pause for thought from the end of a turn, within a few hundred milliseconds, or it either interrupts people or sounds stunned.

The number everyone quotes is the gap between the caller's last syllable and the agent's first sound. A well-built cascade — separate speech-to-text, model, and text-to-speech stages — lands around 750 to 950 milliseconds in published measurements. Codebridge ran 499 recorded calls across four architecture classes and found the fastest stack answering in 707 milliseconds, with honest production numbers stretching from 758 milliseconds to 2.4 seconds once endpointing was included. Their summary of what actually decides the score: latency is a system property, not a model property.

The interpreter at the front desk

Think of a hotel front desk where the clerk doesn't speak your language, so three people handle you in relay. One writes down what you said. A second reads the note and decides what to do. A third reads the decision back to you out loud. Each of them is quick and competent, and the whole thing still feels like talking through a wall — because it is. The note also drops everything the words didn't carry: your hesitation, your tone, the fact that you were being sarcastic about the "upgrade."

Speech-to-speech models collapse the relay into one person who hears and answers directly, which is genuinely better for warmth. It is not automatically faster: on Coval's independent leaderboard of live-call voice-to-voice latency, the best models still take roughly 0.8 to 1.8 seconds to answer, measured from the end of your speech to the first sound of theirs. One person doing all three jobs still has to think.

And there is the waiter problem. A waiter who clears your plate the moment you stop chewing has misread the pause. Voice agents do this constantly, because silence, grammar and intonation are all just proxies for "this person is done." In Codebridge's tests, a 680-millisecond pause in the middle of one sentence — "I'd like to book… an appointment for Friday at three" — defeated every endpointing technology on the market: zero clean passes in more than 65 attempts, across OpenAI, Google, Deepgram, ElevenLabs and LiveKit.

Common misconceptions

"It's a chatbot with a microphone." Text gives the user a send button. Speech gives the agent a guess. Everything hard about the format comes from that missing button — the interruptions, the false endings, the fact that a caller can talk over an answer and expects it to stop instantly.

"Speech-to-speech beats a cascade." Not on the numbers, and not on operations. A cascade keeps a text transcript in the middle, which is what lets a team log calls, redact card numbers, swap one vendor when a model is deprecated, and test the reasoning step separately. Speech-to-speech wins on tone and simplicity; a cascade wins when the agent has to actually do things.

"It's a smarter phone tree." A phone tree routes. An agent acts. That is the entire value and the entire risk: the same system that can issue a refund can issue the wrong one, which is why the tool permissions — not the voice — are what a serious deployment spends its time on. Our explainer on what tool calling does covers how the model reaches those systems, and what an AI agent is covers the loop around them.

"One good demo means it's ready." Measure the slow turn, not the average one. A call is dozens of exchanges, and the caller remembers the one where the agent talked over them.

Where to learn more

If you are building one, start with the boring parts: a turn-detection rule you can audit, a tool list short enough to explain to a lawyer, and a hard handoff to a human after two failed attempts. Then call your own agent from a real phone, in a noisy room, and listen for the pause.

Related reading: What is an agent harness? explains the loop that turns a model into something that can act — the voice is just its mouth — and What is voice cloning? covers the other half of the phone call: how a specific human's voice gets reproduced, and why regulators moved on it before they moved on agents.

Would you stay on the line for an AI agent, or hang up and wait for a human? Write to us via our contact page.