In light of the recent and unexpected outage across different online services in Xbox, the platform’s Chief Technology Officer, Scott Van Vliet has posted a message on Twitter/X, explaining the cause of the service outage that has prevented players from logging in, accessing their Libraries and even playing some of their installed titles. And yes, the outage has finally ended after lasting for 15-16 hours and the services were finally restored roughly around 5:30pm ET so Xbox is back to being fully functional right now.
In the lengthy message, Vliet has assured all players on the platform that they’re currently at work to make Xbox better so problems like what occurred yesterday wouldn’t happen again, while also outlining some of it’s causes along with what they did to fix those:
“What happened. Late last night, a licensing service that sits outside of XBOX, but which XBOX depends upon, began failing. This caused some sign-in scenarios to fail, and it caused many scenarios that require an entitlement check to also fail, such as listing out your Full Library and launching games that you own. It also affected several of our publishing and store partners who depend on those same systems which is why some of you saw issues in specific games and not others.
What we did. Our on-call teams caught the first signals overnight through automated monitoring and declared a major incident. Once we isolated the failing infrastructure, we shifted traffic onto the healthy parts of the system while the team continued to work towards a root cause. Service came back unevenly across regions which is why some of you recovered hours before others.”
The Exec also confirmed that measures are being taken to make sure problems such as these don’t appear again on the platform:
“We’re running a full post-incident review. I care less about the one-line root cause and more about the real questions: why a failure in one service was able to take down this much, why recovery took as long as it did, and what we change so a single point of failure can’t ruin your night again. That means hardening the dependencies underneath sign in and game launch, improving how we detect and contain this class of failure, and being faster and clearer with you when something breaks.”
Xbox is also going through a massive restructuring process that has resulted in mass layoffs and has seen a total of 1600 staff losing their jobs two weeks back and 3200 employees scheduled to be let go by the end of the current fiscal year.
