Build Real-Time Voice Agents with LiveKit & IPFLY: Powering Voice AI with Global Web Data

Voice agents are revolutionizing industries like customer support, travel, finance, and retail by providing hands-free, conversational experiences that feel remarkably human. To truly excel, these voice agents require real-time, relevant web data. This includes up-to-the-minute information such as flight statuses, stock prices, and product availability. Without access to live data, voice agents risk providing outdated or generic responses, diminishing their utility and user satisfaction. LiveKit offers the essential infrastructure for low-latency voice communication, while IPFLY’s premium proxy solutions address the crucial bottleneck of unrestricted access to live web data, boasting over 90 million global IPs across more than 190 countries, with static/dynamic residential and data center proxies.

Build Real-Time Voice Agents with LiveKit & IPFLY – Power Voice AI with Global Web Data

This comprehensive guide will walk you through the process of building a real-time voice agent using LiveKit for seamless voice streaming, OpenAI Whisper for accurate voice-to-text conversion, TTS (Text-to-Speech) for natural-sounding responses, and IPFLY for robust web data retrieval. You’ll discover how to integrate IPFLY to efficiently scrape live data, effectively bypass geo-restrictions, and ensure your voice agent consistently delivers accurate, context-rich responses. Whether users are inquiring about local weather conditions, tracking global stock trends, or checking the availability of specific products, your voice agent will be equipped to provide timely and relevant information.

Introduction to Voice Agents, LiveKit & IPFLY’s Role

Voice agents, also known as voice bots, utilize natural language processing (NLP) and sophisticated voice recognition technology to interact with users through spoken language. Unlike traditional chatbots that rely on text-based interactions, voice agents operate in real-time, necessitating instant access to live data to accurately answer dynamic questions such as:

“What’s the current price of Bitcoin?”
“Is my flight to Paris delayed?”
“Do you have the new wireless headphones in stock?”

LiveKit serves as the fundamental backbone for real-time voice agents, providing scalable, low-latency WebRTC-based voice streaming, efficient room management, and advanced audio processing capabilities. These features are critical for maintaining smooth, lag-free conversations. However, LiveKit alone cannot directly access web data; this is where IPFLY plays a pivotal role.

IPFLY’s robust proxy infrastructure is specifically designed to meet the demanding needs of real-time voice AI:

Dynamic Residential Proxies: These proxies rotate with each request, mimicking the behavior of real users and effectively avoiding blocks on data-rich websites such as airline portals and e-commerce platforms.
Static Residential Proxies: These proxies offer consistent access to trusted sources, including financial APIs and government weather sites, ensuring reliable data retrieval.
Data Center Proxies: These proxies deliver high-speed, low-latency data retrieval for time-sensitive queries such as stock prices and sports scores, ensuring immediate responses.
Extensive Country Coverage (190+ countries): This unlocks access to region-specific data, including local store hours and regional flight information, enabling the creation of truly global voice agents.
Exceptional Uptime (99.9%): This ensures your voice agent consistently retrieves live data during critical conversations, providing uninterrupted service.

Together, LiveKit and IPFLY create a powerful synergy: LiveKit manages the complexities of voice interaction, while IPFLY provides the real-time web data that makes those interactions meaningful and useful.

What Are Voice Agents & Why LiveKit + IPFLY Is a Game-Changer

What Are Voice Agents?

Voice agents are AI-powered tools that process spoken language, retrieve relevant data, and respond using natural-sounding speech. They are deployed across a wide range of industries to enhance efficiency and customer experience:

Customer Support: Voice agents can answer frequently asked questions (FAQs), track orders, and troubleshoot issues, reducing the reliance on human agents and improving response times.
Travel: Voice agents can check flight statuses, book hotels, and provide information on local attractions, offering convenient and personalized travel assistance.
Finance: Voice agents can provide real-time stock prices, account balances, and investment insights, empowering users to make informed financial decisions.
Retail: Voice agents can check product availability, compare prices, and process voice orders, streamlining the shopping experience.

The key differentiator for successful voice agents is real-time data access. A voice agent that cannot provide live flight information or current stock prices feels outdated and unreliable, leading users to abandon it in favor of human support. The ability to deliver accurate and up-to-date information is paramount for user satisfaction and adoption.

Why LiveKit?

LiveKit is an open-source, scalable WebRTC framework designed for real-time audio and video applications. Its robust features make it an ideal choice for building sophisticated voice agents:

Low Latency: LiveKit ensures voice streams are processed in milliseconds, which is critical for creating natural and engaging conversations.
Scalability: LiveKit supports thousands of concurrent voice sessions, making it suitable for enterprise-level deployments with high user volumes.
Flexibility: LiveKit integrates seamlessly with NLP tools like Whisper and ChatGPT, TTS engines, and custom data sources, providing unparalleled adaptability.
Reliability: LiveKit is built for production environments, featuring built-in fault tolerance and a global network of edge nodes for optimal performance and stability.

Why IPFLY?

While LiveKit facilitates voice streaming, voice agents require live web data to function effectively. IPFLY addresses the major challenges associated with data access:

Geo-Restrictions: IPFLY’s extensive pool of over 90 million IPs across 190+ countries enables access to region-specific data, such as UK train schedules and Japanese retail prices, overcoming geographical limitations.
Anti-Scraping Blocks: Dynamic residential proxies mimic real user behavior, effectively avoiding detection and blocks on websites like Amazon, Delta, and CoinGecko, ensuring continuous data access.
Speed: Data center proxies deliver low-latency data retrieval for time-sensitive queries like sports scores and cryptocurrency prices, providing immediate and accurate information.
Consistency: IPFLY’s exceptional 99.9% uptime and multi-layer IP filtering ensure your agent never fails to retrieve data mid-conversation, maintaining a reliable and consistent user experience.

Without IPFLY, your voice agent would be limited to static data, rendering it largely ineffective for real-time use cases. IPFLY’s ability to provide dynamic and unrestricted data access is crucial for creating truly intelligent and responsive voice agents.

Prerequisites

Before you begin building your voice agent, ensure you have the following prerequisites in place:

Python 3.10+ (for backend logic and scripting).
A LiveKit account (a free tier is available; sign up here).
LiveKit Server SDK (for room management) and Client SDK (for voice streaming).
OpenAI API key (for Whisper API and TTS services; obtain one here).
An IPFLY account with an active API key, proxy endpoint, and access to dynamic residential proxies.
Basic familiarity with WebRTC technology, Python programming, and REST APIs.

Install the necessary dependencies using pip:

pip install livekit-server-sdk livekit-client openai requests python-dotenv

Step-by-Step Guide: Build a Real-Time Voice Agent with LiveKit & IPFLY

In this guide, we will build a travel-focused voice agent that demonstrates the integration of LiveKit and IPFLY for real-time data retrieval. This agent will:

Stream voice via LiveKit, capturing user queries such as “Is my Delta flight DL123 delayed?”
Convert the captured voice to text using the OpenAI Whisper API.
Utilize IPFLY proxies to scrape live flight data from Delta’s website, bypassing potential geo-restrictions and anti-scraping measures.
Generate a natural-sounding response using OpenAI TTS (Text-to-Speech).
Stream the generated response back to the user via LiveKit, completing the interactive voice loop.

Step 1: Configure LiveKit Project

Log into your LiveKit account and create a new project (e.g., “TravelVoiceAgent”). This project will serve as the container for your voice agent application.
Navigate to “Project Settings” → “API Keys” and generate a Server API Key and Secret. Store these credentials securely, as they will be used to authenticate your backend with LiveKit.
Note your LiveKit server URL (e.g., wss://project-xyz.livekit.cloud). This URL is essential for connecting to the LiveKit service.

Step 2: Set Up IPFLY Proxies for Live Data Retrieval

IPFLY will be the engine powering your voice agent’s ability to retrieve up-to-date flight data. Here’s how to configure it:

Log into your IPFLY account and retrieve the following credentials:
1. Proxy endpoint (e.g., http://[USERNAME]:[PASSWORD]@proxy.ipfly.com:8080). This endpoint is the gateway through which your requests will be routed.
2. API key (for proxy management). This key allows you to manage your IPFLY proxies programmatically.
For accessing travel data (e.g., Delta flight status), it’s crucial to use dynamic residential proxies. These proxies mimic the behavior of real users, significantly reducing the risk of being blocked by airline websites, which often employ sophisticated anti-scraping measures.

Create a .env file to securely store your credentials:

LIVEKIT_API_KEY=""
LIVEKIT_API_SECRET=""
LIVEKIT_SERVER_URL="wss://.livekit.cloud"
OPENAI_API_KEY=""
IPFLY_PROXY_ENDPOINT="http://[USERNAME]:[PASSWORD]@proxy.ipfly.com:8080"
IPFLY_API_KEY=""

Step 3: Build the Backend (LiveKit + Whisper + IPFLY)

Create a file named voice_agent_backend.py to manage voice streaming, voice-to-text conversion, data retrieval using IPFLY, and text-to-speech generation.

Step 3.1: Initialize Dependencies & Load Environment Variables

import os
import json
import requests
from dotenv import load_dotenv
from bs4 import BeautifulSoup
from livekit import RoomServiceClient, AccessToken
from livekit.rtc import RoomEvent, ParticipantEvent
from openai import OpenAI

# Load environment variables
load_dotenv()

# Initialize clients
livekit_client = RoomServiceClient(
    url=os.getenv("LIVEKIT_SERVER_URL"),
    api_key=os.getenv("LIVEKIT_API_KEY"),
    api_secret=os.getenv("LIVEKIT_API_SECRET"))

openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

IPFLY_PROXY = {"http": os.getenv("IPFLY_PROXY_ENDPOINT"), "https": os.getenv("IPFLY_PROXY_ENDPOINT")}

Step 3.2: Create IPFLY Data Retrieval Tool (Flight Status Scraper)

Implement a function to scrape live flight status information using IPFLY proxies:

def get_flight_status(airline: str, flight_number: str) -> str:
    """Scrape live flight status using IPFLY proxies (Delta example)."""
    # Delta flight status URL (customize for other airlines)
    url = f"https://www.delta.com/en-us/flights/status?flightNumber={flight_number}&date=today"
    
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
    try:
        # Send request with IPFLY proxy to avoid blocks
        response = requests.get(
            url,
            proxies=IPFLY_PROXY,
            headers=headers,
            timeout=15  # Low timeout for real-time voice agent
        )
        response.raise_for_status()

        # Parse flight status (customize selector for target airline)
        soup = BeautifulSoup(response.text, "html.parser")
        status_element = soup.find("div", class_="flight-status-value")
        departure_time = soup.find("div", class_="departure-time").get_text(strip=True) if soup.find("div", class_="departure-time") else "N/A"
        arrival_time = soup.find("div", class_="arrival-time").get_text(strip=True) if soup.find("div", class_="arrival-time") else "N/A"

        if status_element:
            status = status_element.get_text(strip=True)
            return f"Flight {airline} {flight_number} status: {status}. Departure: {departure_time}. Arrival: {arrival_time}."
        else:
            return f"Could not retrieve status for flight {airline} {flight_number}."
    except Exception as e:
        return f"Error fetching flight status: {str(e)}"

Step 3.3: Add Voice-to-Text (Whisper) & Text-to-Speech (TTS)

Implement functions to convert user speech to text and agent responses to speech:

def voice_to_text(audio_data: bytes) -> str:
    """Convert audio from LiveKit to text using OpenAI Whisper."""
    with open("temp_audio.wav", "wb") as f:
        f.write(audio_data)
    
    response = openai_client.audio.transcriptions.create(
        model="whisper-1",
        file=open("temp_audio.wav", "rb"),
        language="en"
    )
    os.remove("temp_audio.wav")
    return response.text

def text_to_speech(text: str) -> bytes:
    """Convert agent response text to speech using OpenAI TTS."""
    response = openai_client.audio.speech.create(
        model="tts-1",
        voice="alloy",
        input=text
    )
    return response.content

Step 3.4: Define LiveKit Room Logic

Implement logic to handle LiveKit rooms, audio streaming, and agent responses:

async def handle_room(room):
    """Handle LiveKit room events (user join, audio stream, etc.)."""
    print(f"Room {room.name} created. Waiting for users...")

    @room.on(ParticipantEvent.TRACK_PUBLISHED)
    async def on_track_published(participant, track):
        """Process audio track from user."""
        print(f"User {participant.identity} published audio track.")
        # Subscribe to the audio track
        await track.subscribe()

        # Collect audio data (stream in chunks for real-time processing)
        audio_chunks = []

        @track.on("data")
        def on_audio_data(data):
            audio_chunks.append(data)

        # Process after ~3 seconds of audio (adjust for longer/shorter queries)
        if len(audio_chunks) > 30:  # ~3s of 100ms chunks
            process_audio(participant, b"".join(audio_chunks))
            audio_chunks.clear()

    async def process_audio(participant, audio_data):
        """Convert audio to text, retrieve data, generate response."""
        try:
            # Step 1: Voice to text
            user_query = voice_to_text(audio_data)
            print(f"User query: {user_query}")

            # Step 2: Extract intent (simplified NLP for travel agent)
            if "flight" in user_query.lower() and ("status" in user_query.lower() or "delayed" in user_query.lower()):
                # Extract flight details (simplified—use NLP library for production)
                airline = "Delta"  # Customize with NLP extraction (e.g., "United flight 456" → airline=United)
                flight_number = user_query.split()[-1]  # Assume last word is flight number

                # Step 3: Retrieve live flight status with IPFLY
                agent_response = get_flight_status(airline, flight_number)
            else:
                agent_response = "I can help with flight status queries. Please ask: 'What's the status of Delta flight 123?'"

            # Step 4: Text to speech
            audio_response = text_to_speech(agent_response)

            # Step 5: Stream response back to user via LiveKit
            await publish_audio(room, participant, audio_response)

        except Exception as e:
            error_response = "Sorry, I couldn't process your request. Please try again."
            audio_error = text_to_speech(error_response)
            await publish_audio(room, participant, audio_error)
            print(f"Error processing audio: {str(e)}")

    async def publish_audio(room, participant, audio_data):
        """Publish agent's audio response to the user."""
        # Create audio track (LiveKit expects PCM 16kHz, 16-bit, mono)
        audio_track = await room.create_local_audio_track(
            name="agent-response",
            source=audio_data
        )
        await room.local_participant.publish_track(audio_track)

        # Send track to specific user (or broadcast to all)
        await room.local_participant.send_data(
            data=audio_data,
            destination_identities=[participant.identity]
        )

# Create a LiveKit room and start handling events
async def start_voice_agent():
    room = await livekit_client.create_room(name="travel-voice-agent-room")
    await handle_room(room)

    # Keep room alive (run in production with a server like Uvicorn)
    while True:
        await asyncio.sleep(1)

if __name__ == "__main__":
    import asyncio
    asyncio.run(start_voice_agent())

Step 4: Build the Frontend (LiveKit Client)

Create a simple HTML/JavaScript frontend (index.html) to allow users to join the voice room and interact with the agent:






    Travel Voice Agent
    Ask about flight status (e.g., "What's the status of Delta flight 123?")
    
    
    Status: Disconnected

Step 5: Add Token Generation (Backend)

To secure your LiveKit room, add a token generation endpoint (use FastAPI or Flask for production). Here’s a simple FastAPI example (token_server.py):

from fastapi import FastAPI
from livekit import AccessToken
import os
from dotenv import load_dotenv

load_dotenv()
app = FastAPI()

@app.post("/generate-token")
def generate_token():
    token = AccessToken(
        api_key=os.getenv("LIVEKIT_API_KEY"),
        api_secret=os.getenv("LIVEKIT_API_SECRET"),
        identity="user-123",  # Replace with dynamic user identity
        room_name="travel-voice-agent-room"
    )
    token.add_grant("join", room="travel-voice-agent-room")
    return token.to_jwt()

# Run with: uvicorn token_server:app --reload

Step 6: Test the Voice Agent

Start the token server:
```
uvicorn token_server:app --reload
```
Start the voice agent backend:
```
python voice_agent_backend.py
```
Open index.html in a browser, click “Join Voice Room,” and ask: “What’s the status of Delta flight 123?”

The agent will:

Convert your speech to text using Whisper.
Use IPFLY proxies to scrape live flight status from Delta’s website.
Generate a natural audio response using TTS.
Stream the response back to you via LiveKit.

Key IPFLY Benefits for Voice Agents

IPFLY’s proxies are vital to the voice agent’s success. Here’s how they enhance performance:

Real-Time Data Retrieval: Data center proxies deliver low-latency responses (critical for voice conversations, where delays >1s feel unnatural).
Anti-Block Bypass: Dynamic residential proxies mimic real users to avoid blocks on airline, e-commerce, or financial sites.
Global Coverage: Access region-specific data (e.g., European flight statuses, Asian stock prices) with IPFLY’s 190+ country IP pool.
Consistency: 99.9% uptime ensures your agent never fails to retrieve data mid-conversation.
Protocol Support: Works with HTTP/HTTPS/SOCKS5—seamless integration with LiveKit and scraping tools.

Use Cases for LiveKit + IPFLY Voice Agents

1. Travel Voice Assistants

Scrape live flight statuses, hotel availability, and local attraction hours.

Use IPFLY’s regional IPs to access country-specific travel data (e.g., train schedules in Germany, bullet train info in Japan).

2. Financial Voice Bots

Retrieve real-time stock prices, crypto values, and market trends.

Use static residential proxies for consistent access to financial sites like Bloomberg or Yahoo Finance.

3. Retail Voice Agents

Check product availability, prices, and store hours.

Scrape competitor pricing to offer price matches (IPFLY’s dynamic proxies avoid blocks on retail sites).

4. Customer Support Voice Bots

Pull live order tracking data from e-commerce platforms.

Access regional support policies (e.g., refund rules in Canada vs. the UK) with IPFLY’s global IPs.

Optimization Tips for Production

Choose the Right IPFLY Proxy Type:
- Time-sensitive queries (crypto prices, flight status): Use data center proxies for speed.
- Strict sites (airlines, banks): Use dynamic residential proxies for anonymity.
- Recurring queries (store hours): Use static residential proxies for consistency.
Add NLP for Intent Recognition:
Replace the simplified intent extraction with tools like spaCy or OpenAI GPT to handle complex queries (e.g., “Is my 3 PM United flight to Chicago delayed?”).
Cache Frequent Queries:
Reduce proxy usage and latency by caching frequently requested data (e.g., popular flight routes) for 5–10 minutes.
Scale with LiveKit Cloud:
For enterprise deployments, use LiveKit Cloud to handle thousands of concurrent voice sessions—IPFLY’s unlimited concurrency scales with you.
Monitor Proxy Performance:
Use IPFLY’s dashboard to track success rates, latency, and IP usage—optimize proxy types based on performance data.

Building real-time voice agents that deliver substantial value requires two essential elements: low-latency voice streaming capabilities provided by LiveKit and unrestricted access to live web data, facilitated by IPFLY. By leveraging LiveKit for managing voice infrastructure and IPFLY for overcoming data access hurdles, you can create voice agents that are not only natural and intuitive but also exceptionally helpful and reliable.

IPFLY’s extensive network of over 90 million global IPs, coupled with its robust anti-block technology and impressive 99.9% uptime, ensures that your voice agent consistently has access to the data it needs, regardless of user inquiries about flight statuses, stock prices, or product availability. When combined with LiveKit’s scalability and Whisper’s accurate voice-to-text conversion, you have a sophisticated, enterprise-grade voice AI solution that distinguishes itself from generic, static voice bots.

Are you ready to embark on building your own voice agent? Begin with IPFLY’s free trial, LiveKit’s complimentary tier, and the code provided in this guide to unlock the transformative power of real-time web data for voice AI applications.