Why AI voice agents need different telephony

Most voice networks were tuned for human ears and human patience. A person hears a clipped syllable and reconstructs it from context, waits through a half second of dead air without comment, and presses a keypad digit twice if the first one did not register. An AI voice agent does none of that. It mistranscribes, it talks over the caller, and it silently misses the digit. The failures are not louder than they were before, they are quieter, which makes them harder to find.

The latency budget is shared, and the network spends first

When a person finishes speaking, the silence before your agent replies is the sum of several things, and the caller experiences it as one thing. Audio has to travel from the caller to your media endpoint. The recognizer has to decide the caller is done and emit a final transcript. The model has to produce a response. Speech synthesis has to produce the first audio frame. That audio has to travel back.

Set yourself a target for the whole thing, and then treat it as a budget you are allowed to spend once. If you decide a turn should land inside roughly one second, an illustrative allocation might look like a couple hundred milliseconds for endpointing and final transcription, a few hundred for the model, a couple hundred for time-to-first-audio from synthesis, and whatever is left for the network. That remainder is usually smaller than people assume, and it is the part you do not control from inside your application.

Two things eat it. The first is transit and processing delay: media relays, transcoding steps, and any hop that buffers before forwarding. Every transcode between codecs adds delay and, if the transcode is between lossy codecs, degrades the signal the recognizer is working from. The second is the jitter buffer, which is where most of the surprise hides.

Jitter buffers tuned for recognition, not for hold music

RTP packets do not arrive at even intervals. A jitter buffer absorbs that variation by holding packets briefly and releasing them on a steady clock. The size of the buffer is a trade: a larger buffer hides more variance and adds more delay, a smaller buffer adds less delay and lets more gaps through.

For a human conversation, and especially for hold music or an announcement, the traditional answer is to buffer generously. Nobody complains about an extra eighty milliseconds on a hold prompt, and everybody complains about a gap. For a machine listener the calculation inverts in one respect and holds in another, which is what makes it interesting.

Recognizers do care about continuity. Packet loss inside a word produces exactly the kind of artifact that turns a confident transcript into a wrong one, and concealment algorithms that sound acceptable to a person can invent phonemes. So you cannot simply shrink the buffer to zero. What you can do is stop paying for variance that is not there. A route with consistently low jitter does not need a buffer sized for a route with bad jitter, and a static buffer sized for the worst path on the network taxes every call on every good path.

The practical requirements are: adaptive buffering that tracks the actual jitter of the path rather than a fixed worst case, loss concealment that is conservative rather than creative, and a preference for paths that do not need much buffering in the first place. That last one is a routing decision, not a media decision, which is why the two are worth owning together.

DTMF that actually arrives

Keypad digits are the most common silent failure in voice automation, because there is more than one way to carry them and the ways do not always survive a hop.

• In-band audio. The tones are simply part of the audio stream. This survives nothing well: compression can mangle the tones, and a recognizer will happily try to transcribe them.
• RFC 2833 and RFC 4733 telephone-events. Digits travel as named events inside the RTP stream. This is the common default and it is what you should expect to work.
• SIP INFO. Digits travel out of band as signaling messages. Some endpoints and some carriers prefer it, and some will send it when you expected RFC 2833.

The failure mode is that a digit is sent one way and expected another, so nothing arrives and nothing errors. Your agent asks for the last four of a policy number, the caller types it, and the agent waits. From inside the application it looks like the caller did not respond.

Handle both RFC 2833 and SIP INFO at the edge, normalize them into one event stream before your application sees them, and test per carrier rather than per platform. A digit path that works on one route can fail on another because of what happens in the middle. If you are building an agent that collects account numbers, dates of birth, or payment digits, this is worth a dedicated test call on every route you use, and worth rechecking when a route changes.

Symmetric RTP and why turn detection depends on it

Symmetric RTP means media flows back to the same address and port it arrived from, rather than to whatever address was advertised in the session description. In a world of NAT, firewalls, and session border controllers, this is what keeps audio flowing in both directions instead of one.

The connection to turn-taking is less obvious. Barge-in, the ability for a caller to interrupt your agent mid-sentence, requires your system to be receiving inbound audio while it is transmitting outbound audio, and to be able to tell the difference between the caller speaking and its own speech returning as echo. If the media path is asymmetric, or if it changes mid-call because a relay reconnected, the timing relationship between what you sent and what you are hearing is no longer stable. Echo cancellation degrades. Voice activity detection starts firing on your own output. The agent either talks over the caller or stops mid-word for no reason.

What you want from the network is boring and specific: media paths that stay put for the duration of the call, symmetric RTP handling by default, consistent packetization, and no unnecessary relay in the middle that might renegotiate. Turn detection is hard enough as a modeling problem without the network changing the rules underneath it.

Concurrency is a different shape for agent fleets

Human call centers ramp. Agents log in over a half hour, take breaks, and go home. Agent fleets do not. A campaign starts and several hundred sessions come up inside a minute, each one holding a media path and a model session for the duration.

That changes what capacity means. Channel counts sized on average concurrency will fail on the first burst. Trunk limits that were comfortable for a sixty-seat floor are not comfortable for a fleet that dials in parallel. And because every session carries active media rather than sitting in a queue, the load is sustained rather than bursty once it arrives.

Three things worth settling before you scale: how many simultaneous sessions your account is provisioned for and how that limit is raised, whether that number is a hard cap or a soft target, and what happens at the ceiling. Rejection with a clear cause code is workable. Silent degradation of audio quality across every active call is not.

What to measure once you are live

Agent quality dashboards tend to measure the model. These four measure the thing underneath it, and they are the ones that explain a bad week when the prompt did not change.

Post-dial delay

The time between sending the call and hearing ringback. It is a routing metric, not an application metric, and it varies by carrier and destination. High post-dial delay costs you on every attempt, including the ones nobody answers, and on outbound campaigns it shows up as callers picking up into silence. Track it per route, not as an average.

Answer-seizure ratio

The share of call attempts that result in an answered call. A drop is usually a route problem or a reputation problem rather than a script problem, and it is the earliest signal that a carrier has started treating your traffic differently. Watch it per destination and per originating number.

Round-trip audio latency

Not signaling latency: audio. Measure the real path by injecting a tone or a marker and detecting it on the return leg. This is the number that tells you how much of your turn budget the network is actually consuming, and it is the one most teams never instrument, which is why they end up tuning the model to compensate for a media path.

Barge-in accuracy

Two error rates, tracked separately. False positives are the agent stopping when nobody interrupted, usually echo or noise being read as speech. False negatives are the agent continuing to talk over a caller who did interrupt. They have different causes and different fixes, and a single combined score hides both. If either moves without a change to your endpointing configuration, look at the media path before you look at the model.

The part that is easy to miss

None of this is exotic. Adaptive jitter buffering, RFC 2833 handling, symmetric RTP, and honest concurrency limits are ordinary carrier engineering. What changed is the tolerance of the thing on the other end of the audio. A network that was comfortably good enough for a floor of people can be quietly bad for a fleet of agents, and the symptom will present as a model problem: worse transcripts, awkward interruptions, callers who hang up early.

Before you swap the recognizer, measure the path. Then ask your provider what the buffer is doing, which DTMF methods they support end to end, whether media paths stay stable for the life of a call, and what your concurrency ceiling actually is. Those four answers explain most of the gap between an agent that demos well and an agent that works at volume.