GitHub CTO apologises to developers after one of the biggest outages

GitHub's Chief Technology Officer, Vlad Fedorov, issued a public apology to developers following one of the platform's most severe outages, which lasted seven hours and 47 minutes on August 17. The outage disrupted core services including authentication, Actions, APIs, pull requests, issues, and Copilot, leaving many users unable to access essential tools for their workflows. According to a blog post published on August 20, Fedorov acknowledged that the issue stemmed from a fundamental scalability challenge rather than human error. He stated that nobody pushed faulty code or misconfigured systems, and the problem arose because the platform simply ran out of space to handle unprecedented traffic levels. The root cause of the outage was traced to an autoscaling policy that monitored the wrong metric. Specifically, the policy focused on the host service rather than the Istio sidecar, which reached its concurrency limit. This led to saturation of load balancers within GitHub's Central US data centre. A latent retry bug in Visual Studio Code further exacerbated the situation by amplifying traffic to a single internal endpoint by approximately 10 times. As a result, the Copilot Token Service remained offline significantly longer than other services. Engineers eventually disabled gateway retries and started returning HTTP 403 errors for incoming Copilot token requests. Additionally, scraping attacks on codeload endpoints complicated the recovery process. At the height of the outage, GitHub's status page recorded error rates nearing 20% for web and API traffic, and approximately 50% for archive and raw repository downloads. Most services resumed normal operation by 1636 UTC, with Actions recovering by 1803 UTC and the Copilot Token Service finally coming back online at 2102 UTC. The prolonged downtime had a tangible impact on developers attempting to deploy or manage code, particularly during a period of heightened activity on the platform. Fedorov highlighted that GitHub's rapid growth has become a major factor in its reliability challenges. Monthly commit counts have surged from 1.4 billion in April to over 2.9 billion currently. To accommodate this demand, the company has deployed more than 3 million CPU cores and 120 petabytes of high-speed storage, pushing the limits of existing data centres. Azure now supports about 58% of the platform's workload, up from 12% in May, handling nearly half of all Git operations. In response, GitHub outlined several planned fixes, including implementing consistent retry limits, retry budgets, and variable timeouts across service interactions. It also plans to review lower-priority CPU and memory alerts and introduce a new architecture designed to scale read capacity proportionally with the number of concurrent readers. These changes will initially target the largest monorepositories. The timing of these developments coincides with another incident affecting GitHub earlier in August, when an Actions service disruption occurred on August 6. During this period, Cursor launched an early beta version of its Origin Code Hosting product, capitalizing on GitHub's unavailability. CloudBees CEO Moritz Plassnig expressed concerns on LinkedIn, suggesting that GitHub might no longer be the preferred option for some developers. His comments reflect broader shifts in the industry, with the code hosting landscape becoming increasingly fragmented as alternative platforms gain traction.