Scrape Google Trends with GitHub Tools: Implementation Guide

Introduction: Why Google Trends Data Matters and Why GitHub Is the Starting Point

Google Trends occupies a unique place in today’s data ecosystem. It is one of the few broadly available sources that reveals what millions of people search for in near real time, with filters for geography, time range, and search property. For teams involved in market research, SEO, content strategy, product development, and competitive intelligence, the signals in Google Trends can be extremely valuable. A rising query in a particular market can signal an emerging need months before conventional reports catch up, while declining interest can reveal shifting audience attention that requires a strategic response.

The difficulty is accessing that information at scale. Google Trends does not publish absolute search counts; instead it returns a normalized interest index from 0 to 100, scaled to the peak of the selected query, region, and time range. That normalization means each value is a relative comparison rather than an absolute volume, and it creates specific technical challenges for anyone building an automated extraction pipeline.

GitHub has become the natural starting point for engineers and data scientists who want to automate Google Trends extraction. The platform hosts a wide variety of open-source projects, from the long-standing pytrends library to newer alternatives that address its limitations. This article explains how to scrape Google Trends using tools available on GitHub, covering the data characteristics, implementation patterns, and infrastructure considerations that separate a prototype from a production-grade pipeline.

Understanding What Google Trends Actually Returns

Before opening a code repository or writing a line of Python, it’s essential to understand the shape of the data you will collect. That understanding drives every architectural choice that follows.

The Normalized Interest Index Explained

Google Trends does not provide raw search counts. Each point in an interest-over-time series represents the relative share of searches within the selected geography and time window, indexed so the peak equals 100. Every other point is reported as a percentage of that peak.

This normalization has important consequences for scraping. Separate API calls for the same keyword in different regions each receive their own 0–100 scale. A score of 100 for “electric vehicles” in Norway and a score of 100 for the same term in Australia are not equivalent volumes. To compare terms directly, submit them in the same request so Google normalizes them on a shared scale.

Available Data Views and Their Extraction Methods

The Google Trends interface exposes several distinct data views, each tied to a different JSON endpoint. Principal views include interest over time, interest by region, related queries, and related topics. Each view uses a widget token and a particular endpoint structure.

The real-time trending searches view is handled differently, using a separate endpoint that returns currently trending topics by geographic location. This view is valuable for time-sensitive use cases like news monitoring or agile marketing, but it requires more frequent polling and increases request volume.

The Anti-JSON Prefix and Token Lifetimes

One common quirk is that Google Trends responses often include anti-JSON characters at the start of the payload. These prefixes must be removed before parsing with standard JSON libraries; otherwise parsing errors occur. Widget tokens also expire quickly and cannot be cached indefinitely. A typical workflow calls an explore endpoint to obtain fresh tokens and immediately uses each token to retrieve the associated series data.

The pytrends Library: GitHub’s Long-Standing Standard

For nearly a decade, pytrends has been the most widely used Python library for programmatic access to Google Trends. Hosted on GitHub and distributed via PyPI, it offers a pseudo-API that hides the complexity of Google’s internal endpoints.

Installation and Basic Configuration

Installing pytrends is straightforward with pip. The library depends on Requests, lxml, and pandas, common components in data science environments. It supports Python 3.3 and newer.

A connection object is created with language and timezone parameters. For example, US English with a Central Standard Time offset uses hl=’en-US’ and tz=360. Google’s timezone convention—positive values for westward offsets—can confuse developers accustomed to standard timezone notation, so pay attention to that detail.

Key API Methods for Data Extraction

pytrends exposes methods that map to Google Trends data views. Interest over time returns a pandas DataFrame indexed by date with columns for each keyword. Interest by region provides geographic breakdowns at country, region, and city levels. Related queries and related topics return structures containing top and rising lists with associated interest values.

Additional methods support trending searches, real-time search trends with finer time granularity, and suggestions for autocomplete expansion. These tools are useful for building and expanding keyword sets.

Proxy Configuration Within pytrends

pytrends supports proxy configuration within the connection object by accepting a list of HTTPS proxy URLs. Timeout settings can specify separate connect and read intervals, and retry logic with exponential backoff can be configured to handle transient failures. Note that only HTTPS proxies are supported by pytrends’ internal request handling; HTTP proxies will not work with this configuration.

Limitations and Maintenance Status

pytrends was archived in April 2025 and has since become unreliable against Google’s current endpoints. Users report 429 rate limit errors, empty DataFrames, and silent failures. This maintenance gap has led to forks, alternative libraries, and managed services that handle token management and request rotation for you.

Modern GitHub Alternatives to pytrends

The open-source ecosystem has responded with several maintained alternatives that address pytrends’ shortcomings.

trendspyg: A Maintained Python Library and CLI

trendspyg provides a modern replacement offering both a Python library and a command-line interface. It supports trending-now queries, interest over time, related queries, and regional breakdowns. The project focuses on session management and token refresh improvements.

Managed API Alternatives and Client Libraries

Several managed services supply client libraries that wrap their Google Trends scraping APIs. These libraries take care of session rotation, token management, and rate-limit backoff, presenting a simple Python interface for retrieval. The trade-off is moving from self-hosted scraping to a managed model with per-request pricing, but it removes the maintenance burden of following Google’s endpoint changes.

Custom Playwright-Based Scrapers

For teams that need full control, GitHub hosts Playwright-based scrapers that automate a browser to render the Trends UI and extract data from the DOM. Browser automation is more resource-intensive than JSON-based approaches but can be more resilient to endpoint changes and can access views not exposed through JSON endpoints.

A typical Playwright scraper launches a browser context, navigates to the trends page, waits for full load, extracts content using selectors or XPath, and writes results to CSV or a database.

Why Proxy Infrastructure Determines Scraping Success

No matter which GitHub repository or extraction method you choose, the collection pipeline’s reliability depends heavily on the quality and reputation of the IP addresses used for requests. IP reputation is often the first signal a service uses to accept or reject traffic.

IP Reputation and Trust

When a script sends an HTTPS request to Google Trends, the server evaluates the source IP address almost immediately. Industry research shows a large share of anti-bot decisions are made based on IP reputation before header or cookie inspection. Requests from datacenter IP ranges are often flagged because those ranges are commonly associated with automated traffic. No amount of header or fingerprint spoofing can fully hide the origin when an IP is known to be from a datacenter.

Residential and Static Residential Proxy Considerations

Residential IPs—addresses assigned by ISPs to consumers—carry the implicit trust of normal browsing activity and can significantly reduce rate-limiting and blocking. Static residential (also called ISP) proxies combine residential identity with stable, low-latency routing, which can be useful when session persistence matters. Datacenter proxies, meanwhile, offer high throughput and low cost for large-volume historical extraction tasks where endpoint sensitivity to IP reputation is lower.

Building a Practical Google Trends Scraping Pipeline

The following outlines a practical, production-minded pipeline architecture that combines GitHub tools with appropriate proxy infrastructure.

Phase One: Keyword Set Construction

Start with a seed list of keywords: product categories, competitor names, and problem-oriented queries. Use suggestion endpoints or autocomplete data to expand the seed set. Size your keyword batches against observed rate limits, and distribute requests across multiple IPs to increase throughput.

Phase Two: Session Management and Token Refresh

Every scraping session requires fresh widget tokens from the explore endpoint. Tokens expire quickly, so implement logic to fetch new tokens when a session starts or when token-related errors occur. Static residential proxies can help by preserving a consistent exit IP, which simplifies session state across requests.

Phase Three: Data Extraction and Error Handling

Iterate over keyword groups, submit payloads, and parse the returned JSON. Handle common failure modes:

Configure retry backoff to avoid overwhelming endpoints. Progressive delays that grow exponentially help resolve transient errors while preserving throughput.

Phase Four: Data Storage and Normalization

Store the raw normalized values together with metadata describing query parameters, geography, timeframe, and extraction time. Because each series is scaled to its own peak, direct comparisons between separate runs are invalid unless the terms were grouped in the same request. For cross-term comparison, include up to five keywords per query so Google normalizes them on a single scale.

Code Example: Configuring pytrends with HTTPS Proxies

from pytrends.request import TrendReq

pytrends = TrendReq(
    hl='en-US',
    tz=360,
    timeout=(10, 25),
    proxies=['https://user:[email protected]:port'],
    retries=3,
    backoff_factor=0.1
)

Only HTTPS proxy URLs are accepted by pytrends. The proxy URL can include credentials, or authentication can be configured separately depending on your provider.

Code Example: Extracting Interest Over Time with Error Handling

import pandas as pd
from pytrends.request import TrendReq
import time

def extract_interest_over_time(keywords, geo='US', timeframe='today 12-m'):
    try:
        pytrends.build_payload(keywords, cat=0, timeframe=timeframe, geo=geo)
        data = pytrends.interest_over_time()
        if data.empty:
            raise ValueError('Empty response returned')
        return data
    except Exception as e:
        print(f'Extraction failed: {e}')
        time.sleep(5)
        return None

This pattern checks for empty responses—a common sign of rate limiting or token expiry—and uses a basic retry delay. In production, extend retry logic to include proxy rotation and more sophisticated backoff strategies.

Case Study: Improving Success Rates with Residential Infrastructure

In one example, a firm collecting weekly trends across multiple countries initially routed requests through a single datacenter IP and saw a low success rate and frequent 429 errors. After moving to a geographically distributed residential proxy configuration and aligning exit IP locations with query geographies, the firm drastically improved success rates and reduced extraction time. Using static residential proxies for parts of the workflow that require session persistence also reduced token churn and improved data consistency.

Compliance and Responsible Data Collection

Google Trends publishes aggregated, anonymized interest data intended for public use. Collecting these public aggregates for analysis generally aligns with that intent, but responsible practices still matter. Respect rate limits, Google’s terms of service, and reasonable request pacing. Proxy infrastructure should be used to ensure reliable access to public data rather than to circumvent access controls.

Conclusion: From GitHub Repository to Production Pipeline

Moving from a GitHub example to a reliable, scalable Google Trends pipeline involves more than choosing a library. It requires understanding normalized data, building robust token and session management, and depending on reputable proxy infrastructure. While pytrends is a useful learning tool, its archival has pushed many teams toward maintained alternatives and custom implementations that offer better long-term reliability.

Choose proxy types and extraction strategies that match your workload: dynamic residential IPs for broad geographic distribution, static residential proxies for session-consistent workflows, and datacenter proxies when throughput is the primary concern. Combine a maintained extraction library or a custom Playwright implementation with disciplined session, token, and error handling to produce dependable trend datasets for analysis.

Next Steps

Evaluate available open-source tools, test extraction workflows against representative keyword sets and geographies, and plan proxy strategy early. Implement robust logging, monitoring, and retry logic, and store raw series alongside metadata to preserve provenance. With the right combination of tooling and infrastructure, Google Trends can become a dependable input to your analytics pipeline.

• Dynamic Residential Proxies – Use for geographically distributed, rotating exit addresses to reduce rate limiting and improve success rates for high-frequency requests.
• Static Residential Proxies – Use where session consistency matters and you need stable, ISP-registered exit addresses for multi-endpoint extraction.
• Datacenter Proxies – Use for high-volume historical extraction when endpoint sensitivity to IP reputation is lower and throughput is the priority.