Getty Images
In a major outage yesterday, Facebook, along with its siblings, WhatsApp and Instagram, became unavailable for hours. Real-time website status tracker DownDetector received more than 14 million reports from users who were unable to use the social media giant’s apps and services.
But aside from the obvious inconvenience for those unable to access these services, yesterday’s outage has had a financial impact not only on Facebook, but also on many small businesses that rely on the platform.
Downtime is estimated to cost Facebook more than $60 million
Facebook’s 2020 revenue was $86 billion. Experts used this number to estimate the average loss the company suffered yesterday at $163,565 for every minute of the outage. Over a six-hour period, this equates to approximately $60 million in lost revenue. Another Fortune report set the loss at $100 million, stating that “for many companies, a drop in revenue of $100 million over time would be a financial event of major concern. For Facebook, it’s a drop in revenue (for now). the bucket investors are likely to shake off.”
More than the loss of revenue, the event also impacted Facebook’s stock, which fell 4.9 percent Monday, translating into $47.3 billion in lost market cap.
Julian Dunn, director of product marketing at PagerDuty, helps companies troubleshoot outages, telling Ars: “Outages [at companies] like Facebook and Instagram mean a lot of money for companies. Some companies are estimated to lose nearly $5 million for every hour their website is down. While several hours of outages are relatively rare, even brief interruptions — 15 minutes or half an hour — have an inordinate impact, as impatient consumers are only too happy to leave a down-site and go elsewhere. Plus, it has a huge effect on the IT and developer teams that keep the systems running at the sites we visit every day.”
And the losses don’t stop there. Some small businesses and firms had the equivalent of a “snow day” yesterday. Boutiques and stores that rely largely on social media platforms to communicate with customers, schedule appointments and take payments lacked the resources to run their business.
“Configuration changes” accused of implosion
Facebook has apologized for the inconvenience caused by the incident. “To all the people and businesses around the world who depend on us, we apologize for the inconvenience caused by today’s outage on our platforms. We’ve worked as hard as we can to restore access and our systems are now back up and running. up-to-date The underlying cause of this outage also affected many of the internal tools and systems we use in our day-to-day operations, complicating our efforts to quickly diagnose and fix the problem,” said Santosh Janardhan, VP for Infrastructure at Facebook.
In addition, New York Times tech reporter Sheera Frenkel had: reported that some Facebook employees were unable to enter the office buildings because badge access systems were also unavailable due to the outage.
While cybersecurity experts initially focused everyone’s attention on Facebook’s missing DNS records, the likely cause of the disruption was later attributed to a misconfiguration of the BGP. An in-depth analysis by Celso Martinho and Tom Strickx of Cloudflare explains how the engineers identified Facebook’s BGP pathways that had been removed from the web:
Routes were revoked, Facebook’s DNS servers went offline and a minute after the problem occurred, Cloudflare engineers were sitting in a room wondering why [our DNS service,] 1.1.1.1 could not resolve facebook.com and worrying that it was somehow a bug with our systems. With this [BGP route] recordings, Facebook and its sites had effectively disconnected from the Internet.
But what appeared to external observers to be BGP and DNS issues was actually the result of a configuration change that affected the entire internal backbone.
In a post-mortem update posted yesterday, Facebook’s Janardhan stated that “configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that disrupted this communication. This disruption of network traffic had a cascading effect on the way our data centers communicate, stop our services.”
Facebook’s services were restored at about 7 p.m. ET yesterday. The company has explicitly pointed out that the root cause of this outage was a faulty configuration change and that there is currently no indication that user data has been compromised. Regardless, the incident is a testament to the dominance of social media and messaging platforms on various aspects of our lives and commerce, making their availability no longer optional.