Build a Real-Time Voice Agent with LiveKit and IPFLY: Powering Voice AI with Global Web Data

Voice agents are transforming customer support, travel, finance, and retail industries, offering hands-free, conversational experiences that feel like human interactions. But to be truly useful, voice agents need instant and relevant web data – like flight statuses, stock prices, or product inventories – to avoid giving outdated or generic responses. LiveKit provides the infrastructure for low-latency voice communication, and IPFLY’s premium proxy solutions, spanning over 90 million global IPs across 190+ countries with static/dynamic residential and data center proxies, solves a critical bottleneck: unrestricted access to real-time web data.

This guide walks you through building a real-time voice agent using LiveKit (for voice streaming), OpenAI Whisper (for speech-to-text), TTS (for text-to-speech), and IPFLY (for web data extraction). You’ll learn how to integrate IPFLY to scrape real-time data, bypass geo-restrictions, and ensure your voice agent delivers accurate and context-rich responses – whether users ask about local weather, global stock market trends, or product availability.

Introduction to Voice Agents, LiveKit, and IPFLY

What is a Voice Agent?

A voice agent (or voice bot) uses Natural Language Processing (NLP) and speech recognition technologies to interact with users through voice. Unlike chatbots, they operate in real time, requiring immediate access to live data to answer questions such as:

• “What’s the current price of Bitcoin?”
• “Is my flight to Paris delayed?”
• “Do you have the new wireless headphones in stock?”

Why Choose LiveKit?

LiveKit is the backbone for real-time voice agents, offering scalable, low-latency WebRTC voice streaming, room management, and audio processing. These features are crucial for smooth, lag-free conversations.

• Low Latency: Ensures voice streams are processed in milliseconds, vital for natural dialogue.
• Scalability: Supports thousands of concurrent voice sessions, ideal for enterprise deployments.
• Flexibility: Integrates with NLP tools (Whisper, ChatGPT), TTS engines, and custom data sources.
• Reliability: Built for production environments with built-in fault tolerance and global edge nodes.

Why Choose IPFLY?

LiveKit enables voice streaming, but voice agents need real-time web data to function. IPFLY solves the biggest data access challenges:

• Geo-Restrictions: Access region-specific data (e.g., UK train schedules, Japanese retail prices) using IPFLY’s pool of IPs across 190+ countries.
• Anti-Bot Blocks: Dynamic residential proxies mimic real users, avoiding blocks on airline, e-commerce, or cryptocurrency websites like Amazon, Delta, or CoinGecko.
• Speed: Data center proxies provide low-latency data retrieval for time-sensitive queries like stock prices or sports scores.
• Consistency: 99.9% uptime and multi-layer IP filtering ensure your agent never fails to fetch data during critical conversations.

Together, LiveKit and IPFLY create a powerful technology stack. LiveKit handles voice interactions, while IPFLY powers it with real-time web data, making responses more practical and informative.

Prerequisites

Before building your voice agent, ensure you have:

• Python 3.10+ (for backend logic)
• A LiveKit account (free tier available; sign up here)
• LiveKit Server SDK (for room management) and Client SDK (for voice streaming)
• OpenAI API Key (for Whisper API and TTS; get it here)
• An IPFLY account (with API key, proxy endpoint, and access to dynamic residential proxies)
• Basic familiarity with WebRTC, Python, and REST APIs

Install the necessary dependencies:

pip install livekit-server-sdk livekit-client openai requests python-dotenv

Step-by-Step Guide: Building a Real-Time Voice Agent with LiveKit and IPFLY

We’ll build a travel-focused voice agent capable of:

1. Streaming voice via LiveKit (user asks a query like, “Is Delta flight DL123 delayed?”)
2. Converting speech to text using OpenAI Whisper
3. Scraping real-time flight data from the Delta website using IPFLY proxies
4. Generating a natural response using OpenAI TTS
5. Streaming the response back to the user via LiveKit

Step 1: Set Up Your LiveKit Project

1. Log into your LiveKit account and create a new project (e.g., “TravelVoiceAgent”).
2. Go to “Project Settings” → “API Keys” and generate a Server API Key and Secret (store these securely – they’ll authenticate your backend with LiveKit).
3. Note your LiveKit Server URL (e.g., wss://project-xyz.livekit.cloud).

Step 2: Configure IPFLY Proxies for Real-Time Data Extraction

IPFLY will power your agent’s flight data scraping. Here’s how to set it up:

1. Log into your IPFLY account and obtain:
   • Your proxy endpoint (e.g., http://[USERNAME]:[PASSWORD]@proxy.ipfly.com:8080)
   • Your API key (for proxy management)
2. For travel data (e.g., Delta flight status), use dynamic residential proxies – they mimic real users, avoiding blocks on airline websites.

Create a .env file to store your credentials securely:

LIVEKIT_API_KEY=""
LIVEKIT_API_SECRET=""
LIVEKIT_SERVER_URL="wss://.livekit.cloud"
OPENAI_API_KEY=""
IPFLY_PROXY_ENDPOINT="http://[USERNAME]:[PASSWORD]@proxy.ipfly.com:8080"
IPFLY_API_KEY=""

Step 3: Build the Backend (LiveKit + Whisper + IPFLY)

Create a voice_agent_backend.py file to handle voice streaming, speech-to-text, data extraction, and text-to-speech.

Step 3.1: Initialize Dependencies and Load Environment Variables

import os
import json
import requests
from dotenv import load_dotenv
from bs4 import BeautifulSoup
from livekit import RoomServiceClient, AccessToken
from livekit.rtc import RoomEvent, ParticipantEvent
from openai import OpenAI

# Load environment variables
load_dotenv()

# Initialize clients
livekit_client = RoomServiceClient(
    url=os.getenv("LIVEKIT_SERVER_URL"),
    api_key=os.getenv("LIVEKIT_API_KEY"),
    api_secret=os.getenv("LIVEKIT_API_SECRET")
)
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
IPFLY_PROXY = {"http": os.getenv("IPFLY_PROXY_ENDPOINT"), "https": os.getenv("IPFLY_PROXY_ENDPOINT")}

Step 3.2: Create IPFLY Data Extraction Tool (Flight Status Scraper)

Add a function to scrape live flight status using IPFLY proxies:

def get_flight_status(airline: str, flight_number: str) -> str:
    """Scrape live flight status using IPFLY proxies (Delta example)."""
    # Delta flight status URL (customize for other airlines)
    url = f"https://www.delta.com/en-us/flights/status?flightNumber={flight_number}&date=today"
    
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
    
    try:
        # Send request with IPFLY proxy to avoid blocks
        response = requests.get(
            url,
            proxies=IPFLY_PROXY,
            headers=headers,
            timeout=15  # Low timeout for real-time voice agent
        )
        response.raise_for_status()
        
        # Parse flight status (customize selector for target airline)
        soup = BeautifulSoup(response.text, "html.parser")
        status_element = soup.find("div", class_="flight-status-value")
        departure_time = soup.find("div", class_="departure-time").get_text(strip=True) if soup.find("div", class_="departure-time") else "N/A"
        arrival_time = soup.find("div", class_="arrival-time").get_text(strip=True) if soup.find("div", class_="arrival-time") else "N/A"
        
        if status_element:
            status = status_element.get_text(strip=True)
            return f"Flight {airline} {flight_number} status: {status}. Departure: {departure_time}. Arrival: {arrival_time}."
        else:
            return f"Could not retrieve status for flight {airline} {flight_number}."
    except Exception as e:
        return f"Error fetching flight status: {str(e)}"

Step 3.3: Add Speech-to-Text (Whisper) and Text-to-Speech (TTS) Functionality

Add functions to convert user speech to text and agent responses to speech:

def voice_to_text(audio_data: bytes) -> str:
    """Convert audio from LiveKit to text using OpenAI Whisper."""
    with open("temp_audio.wav", "wb") as f:
        f.write(audio_data)
    
    response = openai_client.audio.transcriptions.create(
        model="whisper-1",
        file=open("temp_audio.wav", "rb"),
        language="en"
    )
    os.remove("temp_audio.wav")
    return response.text

def text_to_speech(text: str) -> bytes:
    """Convert agent response text to speech using OpenAI TTS."""
    response = openai_client.audio.speech.create(
        model="tts-1",
        voice="alloy",
        input=text
    )
    return response.content

Step 3.4: Define LiveKit Room Logic

Add logic to handle LiveKit rooms, audio streams, and agent responses:

async def handle_room(room):
    """Handle LiveKit room events (user join, audio stream, etc.)."""
    print(f"Room {room.name} created. Waiting for users...")
    
    @room.on(ParticipantEvent.TRACK_PUBLISHED)
    async def on_track_published(participant, track):
        """Process audio track from user."""
        print(f"User {participant.identity} published audio track.")
        
        # Subscribe to the audio track
        await track.subscribe()
        
        # Collect audio data (stream in chunks for real-time processing)
        audio_chunks = []
        
        @track.on("data")
        def on_audio_data(data):
            audio_chunks.append(data)
            
            # Process after ~3 seconds of audio (adjust for longer/shorter queries)
            if len(audio_chunks) > 30:  # ~3s of 100ms chunks
                process_audio(participant, b"".join(audio_chunks))
                audio_chunks.clear()

    async def process_audio(participant, audio_data):
        """Convert audio to text, retrieve data, generate response."""
        try:
            # Step 1: Voice to text
            user_query = voice_to_text(audio_data)
            print(f"User query: {user_query}")
            
            # Step 2: Extract intent (simplified NLP for travel agent)
            if "flight" in user_query.lower() and ("status" in user_query.lower() or "delayed" in user_query.lower()):
                # Extract flight details (simplified—use NLP library for production)
                airline = "Delta"  # Customize with NLP extraction (e.g., "United flight 456" → airline=United)
                flight_number = user_query.split()[-1]  # Assume last word is flight number
                
                # Step 3: Retrieve live flight status with IPFLY
                agent_response = get_flight_status(airline, flight_number)
            else:
                agent_response = "I can help with flight status queries. Please ask: 'What's the status of Delta flight 123?'"
            
            # Step 4: Text to speech
            audio_response = text_to_speech(agent_response)
            
            # Step 5: Stream response back to user via LiveKit
            await publish_audio(room, participant, audio_response)
        except Exception as e:
            error_response = "Sorry, I couldn't process your request. Please try again."
            audio_error = text_to_speech(error_response)
            await publish_audio(room, participant, audio_error)
            print(f"Error processing audio: {str(e)}")

    async def publish_audio(room, participant, audio_data):
        """Publish agent's audio response to the user."""
        # Create audio track (LiveKit expects PCM 16kHz, 16-bit, mono)
        audio_track = await room.create_local_audio_track(
            name="agent-response",
            source=audio_data
        )
        await room.local_participant.publish_track(audio_track)
        
        # Send track to specific user (or broadcast to all)
        await room.local_participant.send_data(
            data=audio_data,
            destination_identities=[participant.identity]
        )

    # Create a LiveKit room and start handling events
    async def start_voice_agent():
        room = await livekit_client.create_room(name="travel-voice-agent-room")
        await handle_room(room)
        
        # Keep room alive (run in production with a server like Uvicorn)
        while True:
            await asyncio.sleep(1)

    if __name__ == "__main__":
        import asyncio
        asyncio.run(start_voice_agent())

Step 4: Build the Frontend (LiveKit Client)

Create a simple HTML/JavaScript frontend (index.html) that allows users to join the voice room and interact with the agent:

    
Travel Voice Agent
    
Ask about flight status (e.g., "What's the status of Delta flight 123?")
    
    
    
Status: Disconnected

    

Step 5: Add Token Generation (Backend)

To ensure the security of your LiveKit rooms, add a token generation endpoint (use FastAPI or Flask for production). Here’s a simple FastAPI example (token_server.py):

from fastapi import FastAPI
from livekit import AccessToken
import os
from dotenv import load_dotenv

load_dotenv()

app = FastAPI()

@app.post("/generate-token")
def generate_token():
    token = AccessToken(
        api_key=os.getenv("LIVEKIT_API_KEY"),
        api_secret=os.getenv("LIVEKIT_API_SECRET"),
        identity="user-123",  # Replace with dynamic user identity
        room_name="travel-voice-agent-room"
    )
    token.add_grant("join", room="travel-voice-agent-room")
    return token.to_jwt()

# Run with: uvicorn token_server:app --reload

Step 6: Test Your Voice Agent

1. Start the token server:

uvicorn token_server:app --reload

1. Start the voice agent backend:

python voice_agent_backend.py

1. Open index.html in a browser, click “Join Voice Room,” and ask, “What’s the status of Delta flight 123?”

The agent will:

• Convert your speech to text using Whisper
• Scrape real-time flight status from the Delta website using IPFLY proxies
• Generate a natural audio response using TTS
• Stream the response back to you via LiveKit

Key Advantages of IPFLY for Voice Agents

IPFLY’s proxies are critical for the success of voice agents. Here’s how they boost performance:

1. Real-Time Data Retrieval: Data center proxies provide low-latency responses (critical for voice conversations where delays over 1 second feel unnatural).
2. Bypassing Anti-Bot Mechanisms: Dynamic residential proxies mimic real users, avoiding blocks on airline, e-commerce, or financial websites.
3. Global Coverage: Access region-specific data using IPFLY’s pool of IPs across 190+ countries (e.g., European flight statuses, Asian stock prices).
4. Consistency: 99.9% uptime ensures your agent never fails to fetch data during conversations.
5. Protocol Support: Supports HTTP/HTTPS/SOCKS5, integrating seamlessly with LiveKit and scraping tools.

Use Cases for LiveKit + IPFLY Voice Agents

1. Travel Voice Assistant

• Scrape real-time flight statuses, hotel availability, and local attraction hours.
• Access country-specific travel data (e.g., German train schedules, Japanese Shinkansen information) using IPFLY’s regional IPs.

2. Finance Voice Bot

• Retrieve real-time stock prices, cryptocurrency values, and market trends.
• Use static residential proxies for consistent access to financial websites like Bloomberg or Yahoo Finance.

3. Retail Voice Agent

• Check product inventory, pricing, and store hours.
• Scrape competitor pricing to offer price matching (IPFLY’s dynamic proxies avoid blocks on retail websites).

4. Customer Support Voice Bot

• Pull real-time order tracking data from e-commerce platforms.
• Access regional support policies (e.g., Canadian vs. UK refund policies) using IPFLY’s global IPs.

Production Optimization Tips

1. Choose the Right IPFLY Proxy Type

• Time-Sensitive Queries (cryptocurrency prices, flight statuses): Use data center proxies for speed.
• Strict Websites (airlines, banks): Use dynamic residential proxies for anonymity.
• Repeat Queries (store hours): Use static residential proxies for consistency.

2. Add NLP for Intent Recognition

Replace simplified intent extraction with tools like spaCy or OpenAI GPT to handle complex queries (e.g., “Is my 3 PM United flight to Chicago delayed?”).

3. Cache Frequent Queries

Cache data for frequently requested information (e.g., popular flight routes) for 5–10 minutes to reduce proxy usage and latency.

4. Use LiveKit Cloud for Scaling

For enterprise deployments, use LiveKit Cloud to handle thousands of concurrent voice sessions. IPFLY’s unlimited concurrency can scale with you.

5. Monitor Proxy Performance

Use IPFLY’s dashboard to track success rates, latency, and IP usage. Optimize proxy types based on performance data.

Conclusion

Building a real-time voice agent that delivers value requires two key components: low-latency voice streaming (LiveKit) and unrestricted access to real-time web data (IPFLY). With LiveKit handling the voice infrastructure, and IPFLY solving data access challenges, you can create voice agents that feel natural, are practically useful, and are reliably informed.

IPFLY’s 90+ million global IPs, anti-block technology, and 99.9% uptime ensure your voice agent always has the data it needs – whether users ask about flight statuses, stock prices, or product availability. Combine this with LiveKit’s scalability and Whisper’s accurate speech-to-text capabilities, and you have an enterprise-grade voice AI solution that stands out from the sea of static voice bots.

Ready to build your own voice agent? Start with IPFLY’s free trial, LiveKit’s free tier, and the code from this guide – unlock the power of real-time web data for voice AI today!