Woodkirk Stone
Just like Facebook’s Antigone Davis was live on CNBC Defending the company against a whistleblower’s accusations and its handling of research data suggesting Instagram is harmful to teens, its entire service network suddenly went offline.
The outage began just before noon ET and lasted almost six hours before being resolved. This is the worst outage for Facebook since a 2019 incident took its site offline for more than 24 hours, as downtime has hit small businesses and the creators who depend on them hardest. services for their income.
After failing all tests for most of the day, a test of the ISP’s DNS servers via DNSchecker.org showed that most of them managed to find a route to Facebook.com at 5:30 p.m. ET. A few minutes later, we were able to start using Facebook and Instagram normally; However, it may take a while for DNS fixes to reach everyone.
On Twitter, Facebook communications manager Andy Stone said, “We are aware that some people have difficulty accessing our applications and products. We are working to get things back to normal as quickly as possible and apologize for any inconvenience. Mike Schroepfer, who will be stepping down as CTO next year, tweeted, “We are having network issues and teams are working as fast as possible to debug and restore as quickly as possible. “
Inside Facebook, the outage shattered nearly every internal system that employees use to communicate and work. Several employees said The edge they resorted to chatting through their work-provided Outlook email accounts, although employees could not receive emails from external addresses. Employees who were logged in to work tools like Google Docs and Zoom before the outage can still use them, but any employee who needs to log in with their work email has been blocked.
Facebook engineers have been dispatched to the company’s U.S. data centers to try to fix the issue, according to two people familiar with the situation. That meant the outage, already Facebook’s biggest in years, could be extended further.
* Sincere * apologies to all who are currently affected by the outages in Facebook services. We are having network issues and teams are working as fast as possible to debug and restore as quickly as possible
– Mike Schroepfer (@schrep) October 4, 2021
A look at Stocking detector (or your Twitter feed) reveals that the problems were widespread. While it’s not clear exactly why the platforms were inaccessible to so many people, their DNS records show that, like last week’s Slack outage, the problem is apparently DNS (it’s still DNS).
Dane Knecht, Senior Vice President of Cloudflare Remarks that Facebook’s border gateway protocol carries – BGP helps networks choose the best path to carry Internet traffic – have suddenly been “taken off the Internet”. While some have speculated on hackers or an internal protest against last night’s whistleblower report, there is still no information to suggest anything malicious is to blame.
Instagram.com was flashing an error message from the 5xx server, while the Facebook site simply told us that something had gone wrong. The issue also affected its virtual reality arm, Oculus. Users can load games that they have already installed and the browser will work, but not social features or installing new games.
Update October 4 at 3:37 p.m. ET: Added additional information about the failure.
Update October 4, 4:15 p.m. ET: Added statement from Facebook CTO Mike Schroepfer, as well as internal updates from Facebook.
Update October 4, 5 p.m. ET: The noted outage is still ongoing, information on the 2019 outage added.
Update October 4 at 5:35 p.m. ET: DNS updates suggest Facebook is getting closer to a solution.
Update October 4 at 6:08 p.m. ET: Facebook.com is back online.