Multilingual Voice AI Agent

Next-Gen Voice AI
Banking Support Agent

A production-grade, low-latency, multilingual voice banking edge system in Go. Dynamic English/Hindi Turn Supervisor, speculative cache probing to bypass LLM latency, WebRTC media gateway, and Cassandra audit logs.

Stack: Go 1.25 · LiveKit · Qdrant Database: MongoDB · Redis · Cassandra Edge Gateway: WebRTC (UDP)
Dharmendra Yadav
Lead Engineer & Architect
< 200ms
Response Latency
80%+
LLM Prefill Saved
100%
Double-Charge Protected
Cassandra
Compliance Ledger
Overview

Autonomous voice-first support edge

Built in Go, v2 replaces slow serial text execution with a highly parallelized edge framework designed for the strict safety, localization, and sub-second latency constraints of modern banking apps.

Saga Transaction Safeguards Safety

Implements client-side generated unique_ref_no to prevent duplicate debit actions on network retry loops. Automatically reconciles banking states with payment_ref_no returned on success.

MongoDB Idempotency Atomicity

Speculative Probing Performance

Runs local deflector LLM prefill (Ollama) in parallel with Qdrant vector database cache probes. Halted early on similarity scores ≥ 0.96 to bypass GPU inference queues entirely.

Qdrant Ollama Parallelism
System Architecture

Symmetric Multi-Replica Topology

The application uses Nginx L4 load balancing to route connections across three stateless Go orchestrator server replicas. Session states and confirmation tokens are synchronized in a shared Redis cache layer.

Voice AI Support Agent v2 Web Dashboard
Voice AI Support Agent v2 control panel — exhibiting live waveforms, balance inquiries, and transaction registries.
Speculative Cache Probing

Parallel Execution Pipeline

Bypassing GPU Prefill Latency

While the user is speaking, the edge gateway splits the speech recognition transcripts into two parallel execution branches:

  • Branch A (Speculative Prefill): Warm-ups the local deflector LLM (Ollama) to perform prefill computation on incremental tokens.
  • Branch B (Cache Probe): Checks Qdrant vector database collections for exact semantic matches against historical FAQ records or parameterized transaction triggers.
  • Halt Point: If Qdrant returns a score exceeding the EXTREME threshold (≥ 0.96), Branch A is immediately aborted, saving CPU/GPU clock cycles.
Branch A Speculative LLM Prefill (Ollama)
Halt Signal (≥ 0.96)
Branch B Qdrant Semantic Cache Probe
Observability & Telemetry

Durable Compliance Audit Logs

Every turn decision point is recorded asynchronously. Logging runs on top of Redis Streams (our local Kafka stand-in) and writes to Apache Cassandra for time-series persistence.

Telemetry Metrics Analyser

Our custom Go metrics analyzer outputs compliance reports based on live event streams:

  • Cache Hit Rate: Ratio of questions answered via cache/FAQ records vs. LLM deflection.
  • Wasted-Prefill Ratio: Prefill tokens wasted on turns that ended as cache hits.
  • Early-Halt Precision: Confirmed extreme-confidence cache hits verified by final transcript reconciles.
$ ./metrics.sh
# Runs CLI metrics parser on top of Redis Stream logs
Voice AI Support Agent v2 Metrics CLI Output
Observability CLI output terminal displaying calculated cache statistics and prefill savings.
Getting Started

Setup & Deployment

Run the integrated launcher script. It checks system prerequisites, downloads required models, compiles Go binaries, and starts the containerized database services.

# 1. Clone the repository and execute bootstrapper
$ ./start-app-v2.sh

# 2. View the dashboard panel in your browser
🔗 http://localhost:9090

# 3. View the system performance telemetry report
$ ./metrics.sh

# 4. Tear down the stack and clean data volumes
$ ./teardown.sh