Scrape Google Trends Data Using a GitHub Tool

Introduction: Why Google Trends Data Matters and Why GitHub Is the Starting Point

Google Trends holds a unique position in modern data work. It is a publicly accessible source that reveals near-real-time search behavior for millions of users, segmented by geography, time range, and search attributes. For teams working in market research, SEO, content strategy, product development, or competitive intelligence, the signals in Google Trends can be invaluable. A rising search interest for a term in a specific market can signal an emerging consumer need months before traditional market reports notice it, while declining interest can indicate shifting attention that requires strategic adjustment.

The challenge is how to collect this data at scale. Google Trends does not expose absolute search counts; it returns a normalized interest index ranging from 0 to 100, scaled relative to the peak for the queried keywords, regions, and date range. This normalization means every retrieved value is a relative measure rather than an absolute count, which creates specific technical considerations when building automated data pipelines.

For developers and data engineers who want to automate Google Trends data extraction, GitHub has become the de facto starting point. The platform hosts a rich ecosystem of open-source repositories—from the longstanding pytrends library to modern alternatives that address its limitations. This article explains how to use GitHub-based tools to fetch Google Trends data, covering technical fundamentals, implementation strategies, and infrastructure requirements that separate prototype scripts from production-grade pipelines.

Understanding What Google Trends Actually Returns

Before browsing any code repositories or writing a single line of Python, you must understand the nature of the data you will collect. This distinction will shape every subsequent architectural decision.

Interpreting the Normalized Interest Index

Google Trends does not provide raw search volume. Each data point in a time series represents the proportion of total searches for the selected region and window, scaled so that the maximum within that query equals 100. A value of 100 corresponds to the highest observed popularity for the query parameters; every other point is expressed as a percentage of that peak.

This normalization has deep implications for data collection. Two independent API calls for the same keyword in different regions will yield separate 0–100 scales. “Electric cars” may score 100 in Norway and 100 in Australia, but those scores are not equivalent in absolute volume. To compare terms fairly, include them in the same request so Google normalizes them to a shared scale.

Available Data Views and How to Extract Them

Google Trends exposes several data views through its interface, each backed by a different JSON endpoint. Key views include interest over time, interest by region, related queries, and related topics. Each view uses distinct widget tokens and endpoint structures.

The real-time trending search view uses a separate endpoint that returns currently trending topics by geography. This view is valuable for news monitoring or time-sensitive marketing use cases, but it requires more frequent polling and therefore increases request volume.

Anti-JSON Prefix Issue

An implementation detail that often surprises developers is Google Trends responses sometimes include an anti-JSON prefix that must be stripped before standard JSON parsers can decode the payload. Failure to handle this correctly causes parsing errors that can be hard to debug if the raw response body is not inspected.

Widget tokens also have short lifetimes and cannot be reused across long periods or separate sessions. Typical extraction workflows first call an explore endpoint to obtain fresh tokens and immediately pass those tokens to the corresponding data endpoints to retrieve the time series.

pytrends: The Long-Standing GitHub Standard for Google Trends

For nearly a decade, pytrends has been the most widely used Python library for programmatic access to Google Trends. Hosted on GitHub and distributed via PyPI, it provides a pseudo-API that abstracts many internal Google endpoints.

Installation and Basic Configuration

Installing pytrends is a simple pip command. The library depends on requests, lxml, and pandas—common components in data science environments—and supports Python 3.3 and above. When instantiating a connection object, specify language and timezone parameters (for example, hl=’en-US’ and tz=360 for US English and a specific offset). Note that Google’s timezone convention uses positive values for westward offsets, which can confuse developers accustomed to standard timezone notation.

Key API Methods for Data Extraction

pytrends provides methods that correspond to Google Trends views. The interest_over_time method returns a pandas DataFrame indexed by date with a column for each keyword. Interest_by_region yields geographic breakdowns at country, region, and city levels. related_queries and related_topics return dictionaries with lists of top and rising queries and their associated interest values.

hottrends can fetch current trending topics for a region, while realtime_trending_searches provides hourly-updated, fine-grained trend data. The suggestions method supplies autocomplete suggestions to expand a seed keyword set.

Proxy Configuration in pytrends

pytrends supports proxy configuration directly on the connection object, accepting URLs in the https://ip:port format. Timeouts accept a tuple for connection and read timeouts, and the library supports retry logic with exponential backoff to handle transient failures. The backoff factor controls the delay growth between retries.

One important limitation is that pytrends only accepts HTTPS proxies in its proxy configuration. HTTP proxies are not compatible with the library’s request handling. This behavior stems from pytrends internals rather than any particular proxy provider.

Limitations and Maintenance Status

As of mid-2025, the original pytrends repository was archived and may not reliably work with Google’s current endpoints. Developers commonly encounter 429 rate-limit errors, empty DataFrames, or silent failures. This maintenance gap has driven the development of forks, independent libraries, and hosted scraping services that abstract token management and request rotation.

Modern GitHub Alternatives to pytrends

In response to pytrends’ decline, the GitHub ecosystem now offers multiple alternatives that address its technical debt and reliability problems.

trendspyg: A Maintained Python Library and CLI

trendspyg is a modern alternative that provides a Python library and command-line interface for accessing Google Trends. It supports current trending queries, interest over time, related queries, and region-based interest. The project actively addresses session management and token refresh issues that affected the original pytrends.

Hosted API Clients and Managed Services

Some hosted services deliver client libraries that wrap their Google Trends scraping APIs. These tools manage session rotation, token handling, and rate-limit backoff, exposing a simple Python interface for retrieval. The trade-off is moving from a self-hosted scraping model to a hosted one—introducing per-request costs and external dependency while offloading ongoing maintenance.

Playwright-Based Custom Crawlers

For teams that need complete control, many GitHub examples use Playwright to build custom crawlers. These approaches launch headless or headed browsers, navigate the Google Trends UI, and extract data from the rendered DOM. Although more resource-intensive than API-based methods, browser automation is resilient to endpoint changes and can capture data views not exposed via JSON endpoints.

A typical Playwright-based extractor defines an async function that starts a browser context, navigates to the trends page, waits for full load, retrieves page content, and parses it using XPath or CSS selectors. Extracted data is then written to CSV or a database for analysis.

Why Proxy Infrastructure Determines Crawling Success

No matter which GitHub repository or extraction method you choose, the reliability of your data pipeline ultimately depends on the quality of the IP addresses routing your requests. This is foundational to successful crawling.

Evaluating IP Reputation

When your crawler sends an HTTPS request to Google Trends, the server evaluates the source IP’s trustworthiness in milliseconds. Research shows a large portion of anti-bot decisions are made primarily on IP reputation before considering headers, cookies, or browser fingerprints. Publicly routable IPs are continuously monitored and scored by a global ecosystem of commercial and open-source threat intelligence services, including spam and bot blocklists and commercial reputation databases.

Requests from data center IP ranges are frequently flagged because those addresses are commonly associated with automated traffic rather than individual users. No amount of browser fingerprinting or user-agent spoofing fully conceals that a connection originates from a known server pool.

Residential Infrastructure and the Trust Layer

Residential IP networks composed of ISP-assigned addresses provide implicit trust because they resemble real consumer browsing activity. For Google Trends extraction, routing requests through a large pool of residential IPs can increase success rates and reduce the frequency of rate-limit responses like HTTP 429 or empty payloads. Distributing requests across many trusted IPs reduces single-point throttling problems that plague data-center-only approaches.

Static Residential (ISP) Proxies for Session Consistency

Dynamic residential proxies excel at load distribution, but some workflows require session persistence across multiple API calls. Static residential or ISP proxies combine residential IP reputation with the performance characteristics of data-center transit: the IP is registered to an ISP (showing as a residential broadband user) while traffic is carried over reliable backbone infrastructure. This combination benefits workflows that need token stability and consistent sessions for multi-step extraction tasks.

When Data Center Proxies Make Sense

Not every Google Trends task requires residential-level IP trust. Large-scale historical extraction for big keyword sets can generate very high request volumes where throughput and cost-efficiency matter more than per-request reputation. Data center proxies offer high throughput, low latency, static IPs, and large parallelism—cost-effective for high-intensity parallel tasks where the target endpoint is less sensitive to IP reputation.

Building a Practical Google Trends Extraction Pipeline

The following outlines a pragmatic architecture for a Google Trends pipeline using GitHub tools and robust proxy infrastructure.

Phase 1: Building the Keyword Set

Start with a seed list of keywords: product categories, competitor brands, and problem-oriented queries. Use suggestions/autocomplete endpoints to expand each seed into a broader keyword set. Adjust the total keyword count based on your chosen extraction method’s rate limits. Google Trends enforces per-IP rate limits; distributing requests across multiple IPs (each maintaining its own session) scales throughput proportionally.

Phase 2: Session Management and Token Refresh

Each scraping session requires fresh widget tokens from the explore endpoint. Implement token-refresh logic during session initialization and when token-related errors occur. Using static residential proxies simplifies session handling by keeping a consistent exit IP across requests, reducing token churn and reauthentication frequency.

Phase 3: Extraction and Error Handling

The extraction layer iterates over the keyword set, submits terms to the appropriate endpoints, and parses JSON responses. Error handling must cover several failure modes:

Proxy configurations should support retry logic and backoff. For example, a backoff factor of 0.1 yields incremental delays like 0.0s, 0.2s, 0.4s between retries—enough to recover from transient issues without overloading the endpoint.

Phase 4: Storage and Normalization

Google Trends returns normalized series, so handle cross-query comparisons carefully. Store the raw normalized values along with metadata describing query parameters, geography, timeframe, and extraction timestamp. For cross-term comparisons, group terms into shared requests (up to five keywords per request) so Google normalizes them on the same scale; independent extractions will not preserve relative relationships between terms.

Code Example: Configuring pytrends with an HTTPS Residential Proxy

The following snippet demonstrates creating a pytrends connection object configured to use an HTTPS residential proxy. Note that pytrends accepts only HTTPS proxy URLs.

from pytrends.request import TrendReq

pytrends = TrendReq(
    hl='en-US',
    tz=360,
    timeout=(10, 25),
    proxies=['https://user:[email protected]:port'],
    retries=3,
    backoff_factor=0.1
)

Proxy URLs require an IP or hostname followed by a colon and port. Authentication credentials can be embedded in the URL or supplied via separate headers depending on your account configuration.

Code Example: Extracting Interest Over Time with Basic Error Handling

import pandas as pd
from pytrends.request import TrendReq
import time

def extract_interest_over_time(keywords, geo='US', timeframe='today 12-m'):
    try:
        pytrends.build_payload(keywords, cat=0, timeframe=timeframe, geo=geo)
        data = pytrends.interest_over_time()
        if data.empty:
            raise ValueError('Empty response returned')
        return data
    except Exception as e:
        print(f'Extraction failed: {e}')
        time.sleep(5)
        return None

This pattern wraps extraction in error handling that detects empty responses—a common sign of rate limits or expired tokens—and implements a basic retry delay. In production, extend retries by rotating proxies to distribute requests across multiple exit IPs.

Case Study: Market Research at Scale with Residential Infrastructure

One financial services team needed weekly Google Trends data for keyword groups across fifteen countries. Their initial implementation used data center IPs and achieved only a 34% success rate, suffering frequent 429 responses and empty payloads. Full extraction took over six hours and produced inconsistent data due to incomplete series and failed requests.

After moving the pipeline to a large residential proxy pool and routing requests through country-specific exit IPs, success rate rose to 99.5% and extraction time dropped to under forty minutes. Geographic alignment of exit IPs with query parameters produced more consistent region-specific responses. The team later adopted static residential proxies for workflows requiring session persistence, enabling more reliable multi-endpoint extraction of related queries and topics.

Compliance and Responsible Data Collection

Automated collection of aggregated public search trends differs from scraping protected content or personal data. Google Trends publishes aggregated, anonymized interest data intended for public use. Extracting this data for analysis aligns with the platform’s intended use, but responsible practices still apply: respect rate limits, adhere to terms of service, and implement reasonable request pacing.

Proxy infrastructure should be used to ensure reliable access to public data, not to evade access controls. Use of residential and ISP-class proxies should follow the provider’s acceptable use policies and legal guidelines for your jurisdiction.

Conclusion: From GitHub Repo to Production Pipeline

Finding a Google Trends scraping repository on GitHub is only the beginning. Building a reliable, scalable data pipeline requires understanding the normalized nature of Trends data, implementing robust session and token management, and—crucially—deploying trustworthy proxy infrastructure. The original pytrends library remains a useful learning tool, but active projects and managed APIs offer more sustainable paths for production needs.

Ready to Build a Reliable Google Trends Pipeline?

Choose the extraction approach that matches your scale and sensitivity to IP reputation: dynamic residential proxies for distributed load, static residential (ISP) proxies for session consistency, or data center proxies for very high-volume historical extractions where throughput and cost matter most. Combine a maintained open-source client or a Playwright-based crawler with a robust proxy strategy to transform fragile prototypes into dependable data assets.

• Dynamic residential proxies — Large pools of ISP-assigned addresses that support automatic rotation and geographic distribution, suitable for high-frequency, data-intensive tasks.
• Static residential (ISP) proxies — ISP-registered residential IPs that provide stable, long-lived sessions to preserve token and session consistency for multi-step workflows.
• Data center proxies — High-throughput, low-latency proxies optimized for large-scale parallel extraction when IP reputation is less critical.
• Design your proxy and extraction stack to meet your specific Google Trends use case: geographic precision, session persistence, or high-volume historical processing.