I saw this happen live because my fiber endpoint (not router because I have my own ONT) went offline for exactly 3 minutes at 4am EST so I realized they were pushing updates lol.
Fiber and internet network went fine but I guess cellular kicked the bucket.
If I had a nickel for every time something that went perfectly in testing messed up in prod, I’d have a lot of them.
I thought it was solar winds
/s
Not to defend them or minimize the corporate stupidity, but it sounded like there were less than 100k people affected out of tens of millions (100m?) accounts. I get that it was a big deal for those affected, but a 0.1% outage doesn’t seem “major”.
Yeah, but this is lemmy so… the outrage is very real. Even for most of the people complaining about it that don’t even have AT&T.
I was impacted and it sucked. Having no cell service for 8-9 hours is not fun. Can’t make or receive calls or texts, every app that requires or uses an internet connection (like Waze) was impacted. Whole Waze worked with directions using offline maps and GPS, you don’t fet stuff like traffic conditions and rerouting.
But when you only have a cell phone and limited wifi resources at the office, it’s a major pain in the butt. And I didn’t report so that 70k could’ve been a conservative number of people that reported.
I do know that FirstNet was impacted. The tablets in our fire apparatus couldn’t connect which is kind of a pain in the neck because we use that to navigate, locate fire hydrants and view their flow capabilities and whether they’re out of service, store maintenance phone numbers, view building blueprints and material safety data sheets, view responding apparatus and locations, identify helicopter landing sites, etc.
Like the job will still get done but it does throw a wrench in our ability to coordinate larger responses.
I think the reported numbers are coming from downdetector.com, which relies on self reporting and people being aware that the website exists. I imagine many more customers were affected. Also, anything the prevents emergency services communication, which occurred during this outage, should be considered a major outage imo
My understanding is that emergency services are either 2G or a mesh infrastructure (perhaps both? I am still learning tech.
No there is dedicated LTE and 5G bans for First Responders. Normal users can use it, but when First Responders connect to it they deprioritze everyone else on the band.
Not to downplay your point, because you are correct, but the outage did not affect anyones ability to contact emergency services, so that is a huge plus in the whole disaster. Any cell phone that pings off a cell tower can reach 911, even if there is no service activated on the phone. It’s important that people are aware of that fact in case they are in a situation where they can’t pay their bill, but still have an emergency.
It literally affected emergency services’ ability to contact each other in multiple areas of the country.
I did not have SOS service on my phone for about 6 hours yesterday. So you are incorrect in that all people were able to contact emergency services. ATT, Upper Midwest
If it’s that big of an upgrade, and your primary customers are North American based, why the fuck do you decide first thing in the morning of a weekday is the time to roll that out? Grab a fresh pot of coffee and start that shit at 10:00 p.m.
Normally, the odd times are because someone can’t work the weekend, which is crazy to me since I am someone who accepts all the worst go live times…
Although the more I think about it, would there really have been a fantastic day or time? Even weekends suck because it affects everything so even then it would have sucked plus if it did bring down companies those people are now doing work on the weekends as well. Idk if I disagree with a late time smack in the middle of the week date for this the more I think about it.
