What we know about the global MS outage

[Opti]

www.nytimes.com/2024/07/19/technology/microsoft-crowdstrike-outage-what-happened.html


What We Know About the Global Microsoft Outage

Airlines to banks to retailers were affected in many countries. Businesses are struggling to recover.


July 19, 2024

Across the world, critical businesses and services including airlines, hospitals, train networks and TV stations, were disrupted on Friday by a global tech outage affecting Microsoft users.


What happened?

A series of outages rippled across the globe as information displays, login systems and broadcasting networks went dark.



The problem affecting the majority of services was caused by a flawed update by CrowdStrike, an American cybersecurity firm, whose systems are intended to protect users from hackers. Microsoft said on Friday that it was aware of an issue affecting machines running “CrowdStrike Falcon.”



This happened two days ago on a Friday. It is currently Sunday. The business day started on Saturday 7/20/24 will end Monday 7/22/24 COB. 

[Tiny]

I'm not sure what to say about this. My employer was effected by late on Thursday night (before midnight) when the pacific Rim offices/ servers started throwing messages. by 1am - all the offices/servers were in some sort of trouble. by 3am things had stabilized but clean up and getting everything going again didn't really happen until hours later.

It was weird how it effected different devices differently. I didn't get the blue screen of death - but my work laptop would "reboot" - I had the black screen of death. And after many reboots/restarts and lots of "press these keys" to achieve X kinds of things - my laptop revived and was fine - I wasn't functional until 11am. A couple of my coworkers didn't have any issues (which was amazing a fortuitous as someone could deal with the servers that were down) - one coworker spent the nearly the entire day without access.


I'm guessing the effected businesses without disaster recovery plans - will be working on those in 2024. The ones who had disaster recovery plans will be reviewing and improving them along with adding a kind of "disaster" to recover from. (It was a lot like 9/11 at work - but with a bit more of a control "we know what to do" chaos to it. My employer has a disaster recovery plan that's pretty detailed. Everyone knew what to do or what to expect in terms of communication and leadership)

[Opti]

TY for your insights Tiny. I am wondering what is going to happen in the next few weeks or even months as a result of this.

[Opti]

I think this outage has caused me to get old phone messages. My friend is currently using an Android phone and used to use an Iphone. Messages pop up from the past, and I have a feeling this might continue for days.

Prior to this there was already an issue. Messaging is very inconsistent, so I call and leave a VM since texting is so unreliable.
Hopefully phone company AI is not autocorrecting my messages that I leave.

[Tiny]

Jul 22, 2024 1:56:53 GMT -5 Opti said:

TY for your insights Tiny. I am wondering what is going to happen in the next few weeks or even months as a result of this.

Probably not a whole lot is going to happen.  FWIW:  my social circle which includes very few techies or people working in corporate America (as most of those my age have retired or FIREd ) even mentioned the outage.   No one asked me if I was effected by it (work wise).  It's like it didn't happen. 

I'm guessing by now all the Big Player companies (like cell carriers, banks, hospitals, airlines, etc) with disaster recovery plans (which should include having routine backups) are up and running.  (I'm guessing they all scrambled on friday/saturday to get the Crowdstrike update backed out of their equipment/devices and their equipment/devices up and running and then got all their databases restored/resynced etc.   I doubt they lost current   "data" or "transactions".     

Not sure how long it will take smaller businesses to resolve their issues.   I expect there will be a renewed push to create "disaster recovery" plans and there will be a lot of upper manangement very very very unhappy with their underpaid/underfunded IT "departments"  and/or their inexperienced IT people.      And I don't mean any of that in a bad way.   

[busymom]

Delta airlines is still not functioning here.  I wonder how much longer it'll take them to get up & running again?  

[Opti]

Jul 22, 2024 18:16:54 GMT -5 busymom said:

Delta airlines is still not functioning here.  I wonder how much longer it'll take them to get up & running again?  

IDK. When it happens, but I'd guess anytime between end of the day to two more weeks depending. The attack on car dealership
software affected more things than I would have thought. Just one data point, I think, but no ordering parts, no creating invoices, no creating work orders etc. 

I have a feeling there was lots done via paper and even more data entry to catch up once the system seemed safe to be used again.
Is it Delta at the initial check in, Delta check in issues etc. at the gate?

[busymom]

Jul 23, 2024 10:17:03 GMT -5 Opti said:

Jul 22, 2024 18:16:54 GMT -5 busymom said:

Delta airlines is still not functioning here.  I wonder how much longer it'll take them to get up & running again?  

IDK. When it happens, but I'd guess anytime between end of the day to two more weeks depending. The attack on car dealership
software affected more things than I would have thought. Just one data point, I think, but no ordering parts, no creating invoices, no creating work orders etc. 

I have a feeling there was lots done via paper and even more data entry to catch up once the system seemed safe to be used again.
Is it Delta at the initial check in, Delta check in issues etc. at the gate?

    I don't know specifics, other than they're cancelling flights like crazy, so this really threw a wrench into their operating system.  

[bean29]

From what I understand, what is screwed up for Delta is Crew Scheduling.  The planes can't take off if they don't have a crew in the right place at the right time.  They have to deal with Union and FAA rules that say they crew members can only work X# of hours before they have a mandatory break.  I tight schedule can fall apart if a delay puts a connecting crew on a time out and then they need to bring in another crew.  Ed Bastain said they were 90% booked.  not only do displaced passengers need open seats, crew members need seats on planes to get to the right place at the right time.  Getting a crew member to a certain airport to work a flight that needs a crew member replacement might require them to bump a passenger with a confirmed ticket.  They are allowed to do that, but they try to get someone to volunteer. 

ETA Flight attendants don't have a Union, but they do have agreed upon guidelines or rules.  The Pilots do have a union.