Avoiding Thundering Herd: GitHub Outage Insights

TLDR; An autoscaling blind spot during a traffic surge caused GitHub’s sidecar proxies to crash and knock out multiple services. Recovery stretched to nearly eight hours after a VS Code bug unleashed a 10x Copilot “thundering herd” against the struggling servers.

On a pleasant Monday evening, August 17th, I looked over at my wife sitting at her desk, visibly frantic and stressed.

She was trying to debug and refactor a messy, legacy codebase. Anyone who has worked with older software knows that sinking feeling—thousands of lines of undocumented logic, fragile dependencies, and a looming deadline.

I asked her what was wrong.

“Copilot is down,” she said, frustrated. “I checked my subscription—I have plenty of credits, but it’s completely dead. Everyone on my team are unable to use Copilot.”

Working at Google, since we use Gemini Powered Internal Tools meant I wasn’t directly affected by GitHub’s downtime. But sitting right next to her, watching her entire engineering team hit a wall because an AI assistant stopped working, gave me a striking realization: we have quietly become far more dependent on AI coding assistants than we realize.

Just a couple of years ago, AI felt like a neat convenience. Today, it has become baseline muscle memory for navigating legacy code, drafting unit tests, and bridging gaps to accelerate feature delivery and bug fixing. When it disappeared, engineering productivity has been impacted heavily.

A few hours later, GitHub officially confirmed a massive global outage lasting 7 hours and 47 minutes. Little did my wife know, her own code editor was inadvertently participating in a global outage.

Here is what actually happened behind the scenes, I have tried to break it down simply with real architectural lessons every engineer should know.

The Trigger: Autoscaling Blindspot

When a platform as massive as GitHub goes down, developers usually suspect a bad deployment, an expired SSL certificate, or a botched DNS configuration.

According to GitHub’s official post-mortem, this outage was not caused by a code deployment or a configuration update. It was a pure capacity failure triggered by record-breaking platform traffic.

To understand why the platform failed to scale during the traffic surge, we have to look inside a standard Kubernetes Pod. In GitHub’s architecture, a single Pod doesn’t just hold the application code; it uses a Sidecar Pattern. This means an Istio proxy (Envoy) sits right alongside the primary application container to act as a front door, handling all incoming and outgoing network traffic.

When the record-breaking traffic hit the Central US data center, the Envoy sidecars took the immediate brunt of the load. They had to manage thousands of concurrent connections and parse requests. Very quickly, these proxies hit their maximum processing ceilings and began bottlenecking.

Here is where the automation failed: The Horizontal Pod Autoscaler (HPA) was looking at the wrong container.

The HPA—the system responsible for spinning up new Pods when things get busy—was configured to monitor only the CPU and health of the main application container.

Because the sidecar proxy was overwhelmed and dropping connections, the main application container was actually starved of traffic. It sat largely idle, waiting for requests that the sidecar couldn’t pass through, which kept its CPU utilization artificially low. The HPA saw the healthy application metrics, concluded that the Pod had plenty of capacity, and refused to provision new resources.

The Amplifier: VS Code “Thundering Herd”

The capacity crunch was serious, but what dragged the incident out for almost eight hours was a massive “thundering herd”.

When a distributed service begins timing out, connected clients often attempt to retry. If those retries are immediate and uncoordinated, they amplify the failure—turning recovery logic into a self-inflicted denial-of-service attack.

Two factors turned GitHub’s localized bottleneck into a global storm:

1. Optimistic Gateway Retries: Internal routing components aggressively retried failing backend calls, keeping internal queues saturated.
2. The VS Code Latent Retry Bug: A previously undiscovered bug in the GitHub Copilot extension for VS Code IDE was triggered by delayed responses from the authentication endpoint. Instead of backing off, the client hammered the Copilot Token Service with rapid, unthrottled token requests.

Traffic to the Copilot Token Service skyrocketed from a normal 7,000–9,000 requests per second to over 100,000 RPS—a staggering 10x traffic spike that prevented authentication servers from recovering.

Timeline of the Incident

The Developer’s Reality: What 20% and 50% Error Rates Actually Mean

Abstract percentages on a status dashboard don’t capture the developer frustration on the ground:

• A 20% API Error Rate: One out of every five interactions fails. Pushing commits via git push fails intermittently, review comments on Pull Requests are lost mid-submission, and CI/CD status webhooks fail to trigger.
• A 50% Repository Download Error Rate: Catastrophic for automated pipelines. Modern CI/CD systems fetch raw manifests, Docker base layers, and repository archives. If a project requires multiple dependency fetches or submodule pulls, a 50% per-request failure rate makes an end-to-end build virtually impossible to complete.

Emergency Mitigation: How SREs sailed through the Storm

Bringing a distributed system back online during such a complex downtime requires deliberate, phased interventions:

• Traffic Rerouting: Engineers drained incoming requests from the saturated Central US data center and rerouted traffic to healthy infrastructure in Northern Virginia.
• Pausing Saturated Nodes: Operators temporarily paused the four overwhelmed HAProxy nodes, cutting off incoming connection loops to allow backend databases and microservices to clear their queues.
• Quelling the Token Storm: To break the 100k RPS storm targeting the Copilot Token Service, engineers deployed an emergency Pull Request (PR) to dial down internal gateway retries. Crucially, they configured edge load balancers to immediately drop unauthenticated Copilot token requests with a fast 403 Forbidden response, protecting upstream authentication backends until traffic could be safely phased back in.

Architectural Lessons Learnt

Beware the Single-Container Autoscaling Trap

If you run a service mesh (Istio, Linkerd, Envoy), never configure your Horizontal Pod Autoscaler based solely on your application container’s CPU. Monitor composite pod utilization, proxy memory, and active TCP connection pools.

Always Implement Exponential Backoff with Full Jitter

Client applications, background daemons, and IDE extensions must never retry failures immediately. Use exponential backoff with randomized jitter to prevent synchronized reconnection waves.

import time
import random

def call_with_jittered_backoff(api_call, max_retries=5, base_delay=1.0, max_delay=30.0):
    for attempt in range(max_retries):
        try:
            return api_call()
        except Exception as err:
            if attempt == max_retries - 1:
                raise err
            
            # Calculate exponential backoff capped at max_delay
            backoff = min(max_delay, base_delay * (2 ** attempt))
            
            # Full Jitter: randomize evenly between 0 and backoff
            sleep_time = random.uniform(0, backoff)
            time.sleep(sleep_time)

Decouple Auth from other services

When authentication gateways share intermediate load balancers with high-volume web and download traffic, an edge traffic spike can lock everyone out of the platform. Core identity and token verification endpoints should always live on isolated, dedicated compute paths.

If you are interested, feel free to browse through my other content at gshiv.com or connect with me in LinkedIn.