Gmail outage blamed on capacity miscalculation

From CNET News.com: Gmail was down from about 12:30 p.m. PDT Tuesday to about 2:30 p.m. PDT, affecting millions of Gmail customers who depend on the service for everything from fantasy football roster updates to business-critical information. The problem was caused by a classic cascade in which servers became overwhelmed with traffic in rapid succession.

According to Google, the problem began when it took several Gmail servers offline for maintenance, a routine procedure that normally is transparent to users. However, the twist this time around was that Google had made some changes to the routers that direct Gmail traffic to servers in hopes of improving reliability, and those changes backfired.

"As we now know, we had slightly underestimated the load which some recent changes (ironically, some designed to improve service availability) placed on the request routers -- servers which direct web queries to the appropriate Gmail server for response," Google said in a post to its Gmail blog late Tuesday.

"At about 12:30 p.m. Pacific a few of the request routers became overloaded and in effect told the rest of the system 'stop sending us traffic, we're too slow!' This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded," wrote Ben Treynor, vice president of engineering and site reliability czar.

View: Article @ Source Site