Heart Surgery Stalled For Five Minutes Thanks To Errant Anti-Virus Scan
from the death-by-horrible-IT-support dept
If you’ve ever had the pleasure of simply asking one medical outfit to transfer your records to another company or organization, you’ve probably become aware of the sorry state of medical IT. Billions are spent on medical hardware and software, yet this is a sector for which the fax machine remains the pinnacle of innovation and a cornerstone of daily business life. Meanwhile, getting systems to actually communicate with each other appears to be a bridge too far. And this hodge podge of discordant and often incompatible systems can very often have very real and troubling implications for patients.
For example, one patient recently undergoing a heart transfer had the procedure interrupted for five full minutes after a PC connected to an essential piece of monitoring equipment began a scheduled anti-virus scan:
“According to one such report filed by Merge Healthcare in February, Merge Hemo suffered a mysterious crash right in the middle of a heart procedure when the screen went black and doctors had to reboot their computer. Fortunately, the patient was sedated, and the doctors had five minutes at their disposal to wait for the computer to finish rebooting, start the Merge Hemo application again, and complete their procedure without any health risks for the patient.”
Fortunate, since “death by shitty hospital IT support” doesn’t sound like a particularly fun way to go. The filing with the FDA by the company in question (Merge) notes that the blame was the fault of the hospital’s IT support, who ignored software instructions that state the folders being used by Merge’s software should always be whitelisted from any anti-virus platforms:
“Merge investigated the issue and later reported to the FDA that the problem occurred because of the antivirus software running on the doctors’ computer. The antivirus was configured to scan for viruses every hour, and the scan started right in the middle of the procedure. Merge says the antivirus froze access to crucial data acquired during the heart catheterization. Unable to access real-time data, the app crashed spectacularly.”
Here’s the thing: aging systems and shoddy medical IT support are the least of the medical industry’s problems. The biggest problem continues to be that medical technology security remains little more than an afterthought, leaving underfunded IT support frequently outgunned. That has resulted in a major wave of ransomware attacks that in some instances have actually forced hospitals to revert to using paper only while they get sorted out (underfunded school systems have been having a dramatic uptick in similar attacks).
And as Internet of Things companies push hospitals to embrace even more sophisticated technologies, you can expect things to get worse. After all, this is a sector that can’t even secure doorbells, refrigerators, thermostats or even tea kettles. What could possible go wrong as these technologies are introduced into an already marginally-competent medical IT sector?
Filed Under: anti-virus scan, computers, heart surgery
Comments on “Heart Surgery Stalled For Five Minutes Thanks To Errant Anti-Virus Scan”
Instead of reporting the problem, that is sloppy for safety critical software.
Re: Re:
Yeah. A critical application should not crash. I can understand that it can’t do its job if it can’t access the data, but it doesn’t seem hard to include some sort of file not accessible message and a retry.
The antivirus would finish with that file in a matter of seconds at most. The reboot took minutes. That could, in some cases, be the difference between life and death.
Re: Re:
It’s that or ransomware. Damned either way, I suppose.
Re: Re: Re:
“Please deposit 1 bitcoin to continue with heart transplant procedure…”
On one hand, yeah, that’s kind of stupid of the IT guys.
On the other hand, that’s kind of stupid of the Merge developers, and extremely stupid of them to not fix the problem when it’s brought to their attention.
Re: Re:
Hospital IT guys are damned if they do, damned if they don’t. Specialized software like this tends to be barely operable on a good day.
Many times you can’t patch the systems (even the OS), because it’ll void the support on the system. Install AV? yup, can’t do that either. partition it off from the network it so it can’t talk to the internet? Nope. Not allowed, vendor won’t support it if it can’t phone home.
But when the thing comes down with malware or crashes in the middle of a procedure, it’s somehow always IT’s fault for not “doing things right”.
It never seems to come back to procurement, for not engaging IT on the system, or on the vendor for using undocumented (and unsupported) operating system API’s to tie into custom hardware that requires crash-prone 3rd party licensing app to read a crypto key off of a thumbdrive.
Medical device software is pretty much crap. In many cases, even if IT had the authority to do so (and they usually don’t), they couldn’t fix the issues these systems see.
Re: Re: Re:
“partition it off from the network it so it can’t talk to the internet? Nope. Not allowed, vendor won’t support it if it can’t phone home.”
Well there’s your problem. That should be an immediate “No way”. Why spend money on software when the company is that stupid?
OF COURSE the computer that runs the heart surgery program should never EVER be connected to a network. Yikes.
Re: Re: Re: Re:
So much this. Such a requirement is so obviously a showstopper risk for a mission-critical application that the fact that any hospital accepts it is terrifying.
Re: Re: Re: Re:
Well there’s your problem. That should be an immediate “No way”. Why spend money on software when the company is that stupid?
IT doesn’t get a say-so in it. Doctors who know little to nothing about software and computers listen to vendor sales pitches, make the decisions and then order IT to do it. Of course, when it goes to shit, IT gets the blame, not the doctors. Shit rolls downhill.
Re: Re: Re:2 Re:
I was just about to say this.
Maybe the hospital or doctors were talked/ bribed/ convinced to get this software from this vendor.
And if the vendor knows they have a monopoly on the market, will they do their best to make robust software? Or will the developers say “There’s no chance a virus scan would interrupt the process, but even so, let’s just tell the users not to do it instead of adding code to handle the error”.
Re: Re: Re:
The real problem is that device manufacturers and the regulators do not care is patients die because of an IT cock-up. Now if both had some real skin in the game such as being vulnerable to murder charges when their criminal incompetence caused a patient to die that might help.
A PC?
For example, one patient recently undergoing a heart transfer had the procedure interrupted for five full minutes after a PC connected to an essential piece of monitoring equipment began a scheduled anti-virus scan:
Who the hell thinks it is OK to use a PC to run an essential piece of medical equipment during a heart transplant? I assume said PC was running windows, poorly secured and connected to the hospital network which also was connected to every other PC in the building . . . what could go wrong!
Re: A PC?
It’s actually very common to use Windows for things. It’s rather popular.
Re: Re: A PC?
Like oil platforms…
Guess what it says in the EULA about things like that? It’s one part of such documents worth minding, instead of treating these things like the appliances they still are not.
Re: Re: Re: A PC?
User assumes all responsibility for any damages caused by use of this product — I bet the hospital never considered heart transplants when they agreed to that!
Re: Re: A PC?
It’s actually very common to use Windows for things. It’s rather popular.
Doctors like Windows. It’s like what they have at home.
Re: A PC?
Windows has become very stable in the last several versions. This wasn’t running Windows 95 or anything.
That said, it would be smart to have a second machine running the same disc image ready to go in case of an emergency like this.
Re: A PC?
Talk of lame flame, wow. Unfortunately, the same would have happened, under any system you could name. If a scan is scheduled, it will happen. It doesn’t matter, MS, Apple Android. Or even all the flavors of Linux, the scan would happen. Should it have happened? Yes.
Whitelist? No. After installing a operating software, all the system should be rechecked,what if their were a virus from the materials installed? Or an update change a permission, that’s what the it guy is for. They are supposed to be knowledgeable about the system, and the programs in use. There could be other problems,
Re: Re: A PC?
It’s not the scan that was the problem but its timing. What is not mentioned what time this happened (e.g. 2AM or 1PM). If it was an emergency surgery then there is a risk of this happening since by definition emergency surgery is unscheduled.
Re: Re: A PC?
Talk of lame flame, wow. Unfortunately, the same would have happened, under any system you could name.
When was the last time you heard of virus scan crippling a Linux system?
Yeah, that’s what I thought. Talk about lame.
Re: A PC?
There’s nothing inherently wrong with using a Windows PC for an application like this.
Having it configured so that processes like virus scanning can happen while it is actively in use is a serious problem, but not one that is special to Windows PCs.
One correction
I object to “marginally-competent medical IT sector” and would call it “largely underfunded medical IT sector”
We can do the job – if we’re given the manpower, the authority, and the tools needed.
Also, faxes are used because people in charge (not your IT sector) often think they are MORE secure than email. Yes, that’s wrong, but “Everyone has a fax” and “I don’t know the email of the person these are going to!” are the battlecry of the “Email is too HARD!” crowd.
It’s not like IT has veto authority in these matters.
Re: One correction
“But $EXEC, you don’t understand, if IT goes ahead with this radical plan to implement $BEST_PRACTICE, people will DIE….”
Re: One correction
No, this is very obvious IT staff not knowing what they are doing.
This machine shouldn’t be actively used…so why in holy hell is is actively running AV scans every hour?
Nightly, at most, would suffice. I mean an AV scan would drastically eat up system resources. And as a system ages could it even finish an AV scan in the time it would take before another one starts?
Constant AV scans are idiotic and counter-productive, especially on machines that are not used to browse the web.
Re: Re: One correction
Every hour is crazy, yes. But transplants take place whenever, day or night. My dad got his call around 6pm, had 1.5 hour window to get to the hospital and surgery was started around 9:30 pm and took 6 hours.
Time sensitive, organs are.
Re: Re: One correction
Most AV has at least two modes (usually 3): the basic mode is “on demand” where you manually scan the contents of the device. This can be scheduled, and such schedules are usually nightly or weekly. This is done while the device is unavailable for regular work.
The second mode is “on access” where individual files are scanned as they are read/written. This is the area where you would want to exclude the files requiring real-time access, unless you’re using a really fast drive.
The third mode is “heuristic” where files aren’t really being scanned at all; it just watches the memory and flags up if anything suspicious is seen.
Now some AV software has this nasty habit of displaying a dialog when something suspicious is found, potentially locking things up on the system until a decision is made. This is bad AV design.
But in this case, the AV software was being run every hour, which, as you say, is the IT staff not knowing what they’re doing. There is absolutely no reason to do an on demand scan every hour. This shouldn’t even really be an option. For quality AV software, it will see that other processes are under high utilization, and throttle back the scanning to not impede other operations… and then the scan will likely take more than an hour to complete, resulting in a cascade of scheduled scans.
Re: Re: Re: potentially locking things up on the system
That’s not bad AV app design, that’s bad OS design.
It’s the kind of thing that used to happen with single-tasking OSes in the 1980s. Not a modern multitasking OS that is supposed to be fit for the 21st century.
Re: Re: One correction
I’ve met people who are so clueless about safe browsing that they actually DO need their computer swept hourly.
Re: One correction
Yeah, I read this snippet and thought, ‘wow, my wife works for a big convention center with relatively slim IT support (I do more pro-bono than their full-time guy does, period). They got a new “do-it-all” fax-print-scan, etc. multi-machine and IT refused to set it up (too ‘difficult’, not worth the effort), and scared them all to use it. I came in, set it up with shared folder, e-mail linking, etc., and now they ‘scan-to-mail’ just like their old fax. They now LOVE it.
Lot to be said for competent IT not trying to ‘excuse’ their way out of doing a job, too. (I did the work on my own time, after hours…what was their excuse?)
Re: Re: One correction
Truly good IT departments are very rare, and very often companies that have them are completely ignorant of their value or even consider them to be a nuisance.
In my experience, the problem IT departments face is twofold: first, they don’t directly generate revenue. There’s no line you can point a bean-counter to that says “here’s the value to the company”. This means that they are often viewed as a drain on resources that is to be minimized, rather than the essential utility that it actually is.
Second, if an IT department is excellent and doing its job properly, then there will always be clashes and people pissed off at them — particularly management, because much of their interaction time will consist of raising holy hell in opposition to some stupid idea or another.
It means that being good at IT is as much a political thing as a technical one. Setting up a new network copier is technically easy, but that kind of thing is often littered with various political mines.
Bad IT departments just give up on the political battles and do the minimal amount they are required to do to keep their jobs. You can spot these pretty easily — the people in these departments just look defeated and cranky.
I have immense respect for good IT people. I wouldn’t last a month in their shoes.
As an aside, when I am evaluating a company that I’m unfamiliar with, the three most valuable things I can learn to get an idea of the company’s character are what the custodial staff, the secretarial staff, and the IT staff think about how the company runs.
I know this is a failure of the IT department, but I wish more software was designed with the user’s preference and real life scenarios in mind. The options are always so black and white like “scan automatically at a set time” or “only scan manually.” Why can’t there be a “prompt for scan without taking over the monitor or crashing the computer and don’t run scan if not receiving an affirmative response.” How about “only scan when no one is logged in and pause/cancel scan when someone logs in?”
Yes, the software license probably says “don’t use this for important systems” but the software could still be written to provider more user friendly options. Stories like this are too common for it always to be an IT department problem.
Re: Re:
I won’t say it’s not the IT department’s fault, but I wouldn’t be surprised if someone re-purposed a PC for this without telling them. There are too many places where they don’t lock them down hard enough because the politics of the business demand that “certain people” have access. I’ve seen it with clueless CEOs who think they need access to EVERY FILE EVERY WHERE!
Re: Re:
For one, all those sorts of things, like AV scans, that should delay while the system is in use, need a better definition of “in use” and ways to detect that. What some applications and operating systems consider to be “idle” is beyond the pale.
Re: Re: Re:
In principle, I agree that a system “in use” should have different rules. The problem is that “in use” is difficult to recognize without domain specific knowledge. For example, what if this heart monitoring application was designed to a read specialized input device, render the results on a screen, and do nothing else? The doctors would say it is “in use” as long as they are relying on its screen to report patient statistics. The general definition used by Windows would say it is “in use” only if someone recently moved the mouse or typed on the keyboard. If the screensaver is disabled, the doctors might “use” the device for hours on end without touching it.
The problem comes down to using a device that is inappropriate for the task at hand. Trying to define extra rules to compensate for the inappropriateness is an ugly workaround, not a fix.
Re: Re: Re: Re:
The obvious solution is to set up a screen saver that displays the heart monitoring data.
Re: Re: Re:
This. I’ve seen apps go idle because I stopped typing for ten seconds. That’s just crazy.
I’m sorry mrs smith, he didn’t make it….we used a Dell.
Just give it time
Soon it’ll be “Windows just shut down on its own to do updates”
it's just a flesh wound...
If you value your life don’t go under the knife
While the anti-virus scans your machine,
To do so is folly and you won’t be too jolly
When blood starts coming from places unseen.
It’s good to see hospitals are willing to do everything in their power to combat the MRSA virus. They even ask IT to pinch in.
Just have to add this
Seems that they didn’t learn from the little fiasco that inspired this little gem:
https://xkcd.com/463/
Re: Just have to add this
You are evil. I have finally clawed my way out of the black hole of XKCD just to tell you that you are EVIL.
Thank you.
IT kills - news at 11
It’s not a matter of Windows. The Therac-25 was using a custom real time OS, and it was involved in six radiation-poisoning accidents.
Software that doesn’t recover? Nonsense. Is it really better if the software continues to run while saying, “I can’t tell if the patient is dying!” for the next 5 or 10 or 20 minutes?
Automation that runs “whenever”. Scans of folders that shouldn’t be. Sounds like typical “blanket IT policy” to me. Regardless of what I’m doing, for example, software install will pop up and kindly give me two minutes to shut everything down before the reboot…even if I’m not there.
An accident/near-accident like in the article happens every so often…that a piece of software using Windows (or not Windows) fails or is clobbered by some automated behavior and critical functionality is lost or someone’s life endangered. (Cases I can recall on Windows: A software install that endangered a patient in South Africa, I think; a blue screen reboot of a mission-critical device on an aircraft carrier.)
If your software is really critical, it needs to be treated that way. The sad truth is: that is not where we are. No one gives a care about security or reliability…just so long as the software package is secure and reliable enough to make it through the sales demo without crashing. Since no one buying software looks behind the grandiose GUI interface, it’s going to stay that way.
So in the end, this case is hardly worth mentioning: “Patient might have died, news at 11.” And if not that then, “Hospital settles wrongful death lawsuit, news at 11.” Actually, these days, neither event would make the evening news.
Re: IT kills - news at 11
It shouldn’t simply crash, IMO. The developers should have forseen the possibility that their software might not be able to access the data for some reason, and programmed it to be robust enough to deal with that.
“Release it now, the damn users can beta test it” isn’t really acceptable for medical applications.
Re: Re: IT kills - news at 11
Several people have said that, and it is meaningless.
The software was locked out of something it needed access to, in order to do its job. Locked out, whether it spends the next 20 minutes saying “Unable to connect to patient; retrying” or simply crashes outright…
…IS NOT RELEVANT TO THE PATIENT. Crash or retry: the patient is equally at risk.
Re: Re: Re: The software was locked out of something it needed access to...
It’s not quite clear why Windows has to “lock out” files while one process is accessing them, to prevent other processes from doing so at the same time. Particularly if it’s only read access.
Linux, for example, does not do this. So a background scan need not block higher-priority activities from proceeding at the same time.
Re: Re: Re: IT kills - news at 11
The difference between having a good clue as to what has gone wrong, or having absolutely no clues can make a huge difference to the patient, because an error messages and automatic retry can make recovering the system much much quicker than fumbling about blindly changing things trying to get the application to stay up when you restart it.
Re: Re: Re: IT kills - news at 11
For how long would the files be locked out? A minute? Two minutes tops?
I’ve encountered medical software which, through system reboot and reinitialising the programme, can take ten minutes (although clearly this one is a bit faster than that) to be fully up and running.
I stand by what I said, the software should be robust enough to deal with it. And since Merge said the file folders should be whitelisted, they obviously knew it was a problem.
As an aside; a problem was discovered with the Alaris (also sold under IVAC and Carefusion brands at different times) Signature volumetric infusion pump. It was known as key bounce, and what could happen was that a keystroke would inadvertently be registered twice (the keypad flexed slightly, so two distinct contacts could be made without the button being fully disengaged).
Clearly, if this key bounce would happen while setting the infusion rate, a rate could be entered that was ~10x what it should be (e.g. 99.3ml/hr instead of 9.3). It totally did happen of course.
Now, Alaris said – quite rightly BTW – that the user should have checked what rate they’d programmed before pressing start. But in the end, Alaris were forced to roll out a software upgrade which detected two key presses within a very short time, and gave a warning message and an audible indication.
So anyone who works for Merge, hold onto your ankles ‘cos you might still get your balls felt.
McAfee: The Killer App
It’s not even that. Some doctors and hospitals simply refuse to do it or work with the other party. But hell, i remember doing this stuff manually.
Getting the other doctor(s) / hospital to actually pay any mind to those records they wanted so badly? Whole other fucking ball game.
Typical
Our endocrinologist is on Windows XP.
Re: Typical
The UK government went so far as to pay Microsoft for extended XP support because the National Health Service was so utterly dependent on XP (and indeed Internet Explorer 6!!!) that switching to Win7 took waaay longer than expected.
In fact, I think some systems are still on XP…
If Micro$oft keeps this up, first the weather broadcast interruption for a Windows 10 sales pitch, then this, they’re only going to accelerate the demise of Windows.
IT Department:
NEVER SCHEDULE AV scans on ANY computer. Defender (MSE) will tell you when it would like to scan WITHOUT interrupting your usage.
Re: Re:
Defender (MSE): “Gee, it looks like the system is idle, no user input. Just this one DDOS-suspect program running.”
Getting a virus infection during open-heart surgery could he catastrophic, so its good to see that the hospital has anti-virus procedures in place.
I Thought Windows Was A Multitasking OS ...
… able to do more than one thing at a time.
Even if disk contention is a problem, a rationally-designed OS like Linux offers this feature called ionice, which lets you reduce the I/O priority of the scanning task to minimize disruption to more important activities.
I use this for backup tasks, for example.
Insecure by design
“folders being used by Merge’s software should always be whitelisted from any anti-virus platforms”
Malware writers praise Merge for providing a safe place to hide their stuff!
How can you ensure something is free of malware if you are not allowed to look everywhere malware might be?
Mission-Critical Software
There’s only ONE answer for that. LINUX or BSD, properly set up, with a good tech available, will NEVER fail, unless someone wants it to. Cheaper too.
Apps might be a problem, but I’ve found that most software is available, at least a functional match, for Linux or BSD. Custom software can be ported over to Linux, at least if the author is still available. Worth looking into.
Re: Mission-Critical Software
…with a good tech available…
Do you think this problem would have occurred if a “good tech” was available? Oh, right, companies don’t pay for those…”Mordac the preventer of information services” is cheaper.
RE Hospital IT
I was at the hospital this past week and ended up sitting in a waiting room deep inside the complex.
I pulled out my phone and no service.
No problem I have WiFi calling I’ll just hop on their guest network.
I connect and it wants me to enter my phone number so they can send me a text to gain access.
I still haven’t figured out if this is by design or incompetence.
I suspect somewhere the CIO is telling the rest of the suits that they no longer need a guest network because hardly anyone ever uses it.
I'm Confused!
Granted, I’m NOT a medical IT tech, but I don’t understand why that system needed AV at all. The way things SHOULD be done, The local mission-critical systems would only be connected to the Institution’s internal network, and not have any direct connection to the internet. If it needs data, the request should be passed up to the main system, which would request it from the internet, run any AV and antimalware scans necessary, then pass the clean file to the working system IN NO WAY should the monitoring system be directly connected to the internet! That’s just asking for trouble. In this instance, no AV is necessary because the main, internet-connected system should take care of that.
NBD
just add this
to the List :drug mixups/toxicity, hospital ‘food’, patient name confusion,
hospital acquired infections++…no wonder
the medical establishment is the 3rd biggest killer:
http://www.health-care-reform.net/causedeath.htm
http://www.cnbc.com/2016/05/04/medical-errors-are-third-leading-cause-of-death-in-united-states-study.html
Medical errors are third-leading cause of death in United States: Study
Dan Mangan
4 May 2016
Go to the doctor or hospital when you’re sick in the hopes of getting better, and you might end up dead, instead.
A new study estimates that medical errors are actually the third-leading cause of death in the United States, responsible for a whopping 251,454 fatalities in 2013.
Only heart disease and cancer, which respectively killed 611,000 people and 585,000 people that year, outpaced medical errors, according to the study published in the medical journal The BMJ…
Stupid is as stupid does
Anyone who uses Windows systems to implement/control real-time or safety-critical systems should be sued for reckless endangerment! There are much more robust software systems for such uses, such as Linux RT, QNX, etc.
Re: Stupid is as stupid does
Anyone who uses Windows systems to implement/control real-time or safety-critical systems should be sued for reckless endangerment!
Agreed, but it seems doctors would rather lobby for protection from such lawsuits.
Re: Re: Stupid is as stupid does
The eight most frightening words in the English language are:
we are proud to be a Microsoft house.
Seriously.