Computer Outage

[Lee B]

The word wide computer outage caused by a faulty update from Crowd Strike, a third party security service used by Windows

has taken down the online payroll service that my largest client uses

and the online accounting program that I and my monthly write up clients use.

Hope they get it fixed by Monday morning.?

[Abby Normal]

And this is just one of the reasons why I've always avoided "cloud" software.

[BulldogTom]

What a wake up call this is for businesses.   A software company that you don't even subscribe to took down the software companies you do use.   Lots of questions about where we are going from here will be asked.   I wonder if the US government or state governments were affected?

Tom
Longview, TX

[Lee B]

Well my online accounting program is working again.

I have had problems logging in to this program several times in past years due to problems with their use of a third party log in service

Three or four years ago this program was down for 3 days due to problems with Amazon Web Services who hosted the program.

We as end users have no idea how many third party services are being used.

 

[Lee B]

21 minutes ago, Abby Normal said:

And this is just one of the reasons why I've always avoided "cloud" software.

It's kinda difficult to avoid cloud software.

Your use of your browser is cloud based.

You're on the cloud when you post to this website.

Do you paper file all of your tax returns?

E-filing is cloud based.

Do you really think ATX or Drake has no exposure to the cloud?

[Medlin Software, Dennis]

As a developer:

It is always best, but sometimes difficult, to ask about all items used to create/manage your important processes. For instance, we use one third party vendor for certain things, but the software functions even if there were to be an issue with the third party (by design). Not one single customer has ever asked! I also choose vendors who are like me, personal, and not usually the biggest player with the biggest target on them.

Nothing wrong with remote storage of data as long as there are options for local use as well.

Hardware and software not at your location is another story. One must consider outage and even instant end of life for all such things, and how to recover. A good example is charge card processing. I keep three methods somewhat ready. My main, which has actually never been down for any significant time in 40 years, and two others which I can get going within a day.

None if this is new, and if shocking, it is a personal failure to plan. No different than having more than one bank account and more than one charge card, because they do have issues (such as this issue) from time to time.

Note for the OP, the software (CrowdStrike's Falcon) is not used "by" Windows, it is software which was made for Windows machines, installed and used by the person/company (not Microsoft).

One must always consider issues, such as asking if their tax preparer/accountant/doctor/etc. is using a current version of Windows, kept current, with limited access.

[Medlin Software, Dennis]

This very forum is "cloud", and publicly stored and searchable by anyone.

[Lee B]

2 minutes ago, Medlin Software, Dennis said:

 

One must always consider issues, such as asking if their tax preparer/accountant/doctor/etc. is using a current version of Windows, kept current, with limited access.

 If I went to my bank and asked my bank manager, she wouldn't even know

[Medlin Software, Dennis]

1 minute ago, Lee B said:

If I went to my bank and asked my bank manager, she wouldn't even know

Banks are likely directly regulated and would not get out of compliance often. I have customers, and I think have read similar here, who insist using Windows 7, 8, and very soon, Windows 10, is safe (and they will NOT be upgrading).

[Medlin Software, Dennis]

43 minutes ago, Lee B said:

Three or four years ago this program was down for 3 days due to problems with Amazon Web Services who hosted the program.

We as end users have no idea how many third party services are being used.

Ask. If the information is not available/shared freely, something is wrong. If so inclined, you can use monitoring via your router/firewall/logging system, to see what an application does or tries to do online.

[Abby Normal]

58 minutes ago, Medlin Software, Dennis said:

This very forum is "cloud", and publicly stored and searchable by anyone.

It would pretty damn useless for me to have a forum just on my computer that no one else could access.

[Abby Normal]

1 hour ago, Lee B said:

E-filing is cloud based.

Efiling is sent over the internet but I can still work on tax returns locally while the internet is being restored. If my tax software was cloud, I'd have to find something else to do.

[Abby Normal]

1 hour ago, Lee B said:

You're on the cloud when you post to this website.

You could say that the entire internet is a cloud, but that would be abusing the term and making it meaningless. My browser is still local software.

[Medlin Software, Dennis]

2 minutes ago, Abby Normal said:

It would pretty damn useless for me to have a forum just on my computer that no one else could access.

You can and could setup your own server for hosting a forum. While it would be accessed through the internet or wayback, through a direct connection such as sending and receiving modems.

I suppose the semantics are defining cloud versus non-local hardware, as well as who controls the management of the hardware and software.

"Cloud" seems to have morphed into some sort of magical storage hardware "in the sky" for which most have no idea what or where it is. For me, I demand to know where the hardware is, as I want it in the US, and want to know where in the US so I have storage in more than one area for disaster reasons, as well as for compliance reasons.

[BulldogTom]

Just read a story that I could not believe.   Southwest Airlines is not affected because they still use Windows 3.1 !!!!

I don't know if I really ever want to fly that airline again, but right now it is the only one flying.

Tom
Longview, TX

[Lee B]

12 hours ago, BulldogTom said:

Just read a story that I could not believe.   Southwest Airlines is not affected because they still use Windows 3.1 !!!!

I don't know if I really ever want to fly that airline again, but right now it is the only one flying.

Tom
Longview, TX

According to local news reports, Alaska Airlines was flying in and out of Portland with no delays.

[TexTaxToo]

AI generated summary of yesterday's events:

[Abby Normal]

21 hours ago, TexTaxToo said:

AI generated summary of yesterday's events:

AI is blaming the new guys.

[Medlin Software, Dennis]

What gets me is this product is used by high end computer users, with what should be experienced people maintaining the devices. There was no auto restore/revert process in place? Failing that, no image or similar they could have brought back in short order? Or secondary devices ready to failover? So if not, which seems to be the case, why use the "security" software in the first place, as they clearly are not really protecting themselves from failures? Not having alternate hardware system in place is the REAL failure and is one that strikes all at some point from the new computer user to the highest experts. This is a case of belt AND suspenders, and only a few actually take the time to strap on both every single time.

I doubt SWA is using W3.1 as many opine, but they software could be from that era, such as 16 but software (for instance). There is some truth to being outdated as the script kiddies have no idea how to hack software made before they were born.

[Lee B]

I have read multiple articles saying that SWA has been using a mixture of Windows 3.1 and Windows 95.

I have read airline industry articles over the last 3 or 4 years criticizing SWA for not updating their IT infrastructure.

The most amazing thing to me is reading that some IT departments are having to deal with this faulty update one computer at a time.

Read a very interesting article saying that most of our IT systems have multiple "single points of failure" ,

explaining that similar outages will happen again in the near future.

 

[Medlin Software, Dennis]

Even if someone is using 3.1 or 95, it would not bother me depending on the circumstance. There are still plenty of COBOL uses, which predate my lifetime. What is missing is the acceptance everything can fail, and redundancy is not an option. I saw a headline Delta still has not fully recovered, and the politicians are investigating why.

I still try to point out all who use a computer need to be prepared for it to disappear/poof at any moment, and the same for their office. Once someone can say they can reasonably quickly start over with no location or hardware from the past, and proved it by real testing, then and only then do they have a good recovery plan.

[Lee B]

Our CEOs and Board of Directors are trained at the elite business schools to consider IT and IT security as a controllable expense since it doesn't produce any revenue.

As a result many IT functions are subcontracted out too often to the lowest bidder.

[Medlin Software, Dennis]

"Our CEOs and Board of Directors are trained at the elite business schools to consider IT and IT security as a controllable expense since it doesn't produce any revenue."

Skipping how it can cost revenue (at best) and end the entity (at worst)? Skippy teaching Skippy. There needs to be some old war horses kept in the education and reality loop. People who have dealt with total data loss and total machine failure, at the same moment. School of Hard Knocks (or for original Raider fans, the University of Mars) wins again! I am a firm believer that IT "school" can never be timely, it can give a foundation, but hard knocks is a MUST to be current.

"As a result many IT functions are subcontracted out too often to the lowest bidder."

"Lowest bidder" always makes the hairs on my neck tingle. Yet, I drive over overpasses built by a company who usually is the lowest bidder, but they figured out how to pad by getting a bonus for early completion.

[Lee B]

"Delta Air Lines has canceled hundreds of flights for the third day in a row, as the airline is struggling to recover from Friday’s global software outage — even as other U.S. airlines are bouncing back and returning to normal operations."

 CEO Ed Bastion:

"In the public letter, Bastian wrote that several Delta applications run on Microsoft Windows, resulting in many tools being impacted in the outage. More specifically, one of the airline’s crew tracking tools was deemed inoperable and “unable to effectively process the unprecedented number of changes triggered by the system shutdown,” Bastian wrote."

[Lee B]

Now here is a quote that emphasizes the "single point of failure" problems of our current  world wide IT infrastructure:

"The reality is despite CrowdStrike's epic failure last week, there are few alternatives to CrowdStrike and the switching costs are high," Gene Munster, managing partner at Deepwater Asset Management, which is known for its tech investments, wrote on social media."