Jim Salter
Yesterday, packages.microsoft.com — the repository from which Microsoft serves software installers for Linux distributions, including CentOS, Debian, Fedora, OpenSUSE, and more — went down hard and remained offline for about 18 hours. The outage affected users trying to install .NET Core, Microsoft Teams, Microsoft SQL Server for Linux (yes, that’s a thing), and more, as well as Azure’s own devops pipelines.
We first became aware of the problem Wednesday night when we saw 404 errors in the output of apt update on an Ubuntu workstation with Microsoft Teams installed. The outage is a little better documented in this .NET Core issue report on Github, with many users from around the world sharing their experiences and theories.
The short version is that the entire repository cluster that serves all Linux packages for Microsoft was completely down for about 18 hours, throwing a series of HTTP 404 messages (content not found) and 500 messages (Internal Server Error) for every URL. issued. Microsoft engineer Rahul Bhandari confirmed the outage about five hours after it was initially reported, with a cryptic comment about the infrastructure team “running into some space issues”.
Eighteen hours after the issue was detailed, Bhandari said the mirrors were available again, albeit with a temporary drop in performance, likely due to cold caches. In this update, Bhandari said the original cause of the outage was “a network downtime”. [apt repositories] during some feature migration work that resulted in those packages becoming unavailable on the mirrors.”
We are still awaiting a comprehensive incident report as Bhandari’s status updates provide clues but no real explanations. The good news: we can confirm that packages.microsoft.com is indeed up and running again and is displaying packages as it should.