The StateSet Voice Engine for Customer Experience and Commerce Operations

The phone call is still the highest-intent customer channel in commerce. Customers call when the order is late, the subscription needs to change, the return is urgent, or the answer cannot wait for another ticket. The problem was never that voice lacked value. The problem was that voice was disconnected from operations.

We built the StateSet Voice Engine to close that gap. It is a realtime voice system for customer experience and commerce operations: a caller can speak naturally, the agent can reason through the request, use approved tools, update systems, escalate to a human, and leave behind a durable operational record.

Why Voice Needed a New Runtime

Legacy phone systems were built around routing. IVRs moved callers through menus. Contact centers moved calls into queues. Help desks converted conversations into tickets. None of that architecture was designed to complete the work while the customer was still on the line.

Commerce operations require a different model. A single call may touch order status, fulfillment context, subscription state, return policy, delivery exceptions, account verification, SMS follow-up, and live escalation. That is not a phone tree. It is an orchestration problem.

Inside the StateSet Voice Engine

The engine is built as a Rust service using Tokio and Axum. It bridges Twilio Media Streams with realtime AI models over WebSockets, supports OpenAI and OpenAI-compatible realtime providers, and can switch into an ElevenLabs-powered audio path when teams want dedicated speech output and transcription services.

Each call resolves to a tenant, an agent configuration, and a set of approved functions. Those functions are not decorative. They are the contract between conversation and action: look up order status, build fulfillment context, send tracking SMS, modify Loop subscriptions, capture receptionist intake, create callback tasks, request appointments, send follow-up messages, and escalate to a human when the situation requires judgment.

Twilio call
  -> tenant and agent resolution
  -> realtime voice session
  -> tool orchestration
  -> commerce system action
  -> transcript, summary, outbox, and audit trail

Realtime by Default

Customer calls are latency-sensitive. The system keeps a live bidirectional media bridge open instead of waiting for bounded request-response turns. Audio streams in, the agent produces speech and tool calls, and the server handles barge-in cancellation so customers can interrupt naturally.

Commerce-Aware Tools

The voice agent is grounded in the operational vocabulary of commerce: orders, exchanges, refunds, subscriptions, fulfillment, sales inquiries, receptionist workflows, callback queues, and handoff notes. That lets teams define voice behavior as operational policy rather than generic chatbot copy.

Durable Operations

A call is not complete when the audio ends. The engine persists call logs, voice sessions, review fields, outbox events, outbound attempts, operator cleanup tasks, and webhook delivery state. If a downstream system is temporarily unavailable, the work is retried durably instead of disappearing into a transcript.

Customer Experience Meets Operator Control

The goal is not to hide humans. It is to use humans where they matter. The engine supports supervisor actions such as monitor, whisper, barge, escalate, and end. It can create after-hours follow-up tickets, send structured handoff notes, and route live bridge workflows from tenant-level configuration instead of hard-coded call center rules.

What Makes This Different from a Voice Bot

A voice bot answers questions. A voice engine completes work. That distinction changes the architecture.

• It resolves tenants and agents dynamically. A phone number, token, or signed media-stream bootstrap can map the call to the right business, route, model, prompt, and tool set.
• It separates read and write behavior. Low-risk lookups can happen quickly, while write actions, external messages, bookings, and escalations require confirmation.
• It treats exact values carefully. Order IDs, tracking numbers, phone numbers, and emails are captured conservatively and confirmed before account-specific action.
• It is observable. Latency, provider outcomes, retries, transcripts, logs, and session state are all first-class operational data.

Designed for the Messy Middle of Commerce

The hardest calls are rarely simple FAQs. They are the messy middle: a delayed shipment with a subscription renewal coming up, a customer asking for a human after business hours, a caller who wants to leave a message but also needs a tracking link, a receptionist workflow that needs structured intake before follow-up.

The Voice Engine is built for that middle layer. It can answer directly when the answer is clear, search knowledge when facts need grounding, use tools when action is required, and escalate when policy or empathy demands a person.

The New Operating Model

Voice used to be measured by containment or average handle time. Those metrics are incomplete. The better question is whether the customer's intent became an operational outcome: a subscription changed, a callback scheduled, a ticket enriched, a fulfillment issue triaged, a lead captured, a human connected with context.

That is the operating model we are building toward: every conversation becomes structured operational state. Every operational state can trigger an agentic workflow. Every workflow is observable, retryable, and governed.

Where We Go Next

Realtime voice models are improving quickly. The next generation can reason longer, handle interruptions more naturally, transcribe and translate live audio, and call tools with more precision. The StateSet Voice Engine is designed to absorb those model improvements without giving up the operational controls commerce teams need in production.

The future of customer experience is not a better phone tree. It is a voice-native operations layer where customers speak, agents reason, systems act, and humans stay in control of the moments that matter.