A TapRooT® Investigation at System Improvements
Here at System Improvements, we practice what we preach. For example, we are using the Version 5 TapRooT® Web Enterprise Software to perform audits of our own internal processes: from Accounting, to Course Registration, to Information Technology.
But every once in a while we have to react to an actual incident, and perform our own investigation as well.
Challenge
On August 13th we had to react to a situation that seemed unusual.
We have a very large group of people who use both the TapRooT® Process and Software, and we are fortunate to have a community that shares their best practices, as well as their problems, with us.
We received data that was outside what we would classify as ‘normal’ for our standard operations (and our upcoming trending release will allow you to determine this as well). We received several comments from customers that their computers were getting infected with a virus known as the ‘Live Security Platinum’ virus.
The Live Security Platinum virus has been around for years and is a particularly nasty virus that likes to pretend it is acting as an Anti-Virus software and protecting your computer, while in fact it is infecting it further with every action that you take. This virus is often observed when you try to click on an internet link or download a file. It presents a warning message that appears to caution you from exposing your computer to further risk, when in reality any selection you make is propagating the virus.
Action
Our initial reaction when we received the first report of this on Friday August 10th, was to assume that it was an isolated incident, and one of our users had simply contracted the virus. However, between August 11th and August 13th, we received 11 reports of the incident. At this point we had to investigate further:
Were all the users from the same company? The answer was no. Several were from the same company, but not the whole group.
Were all the users using the same internet connection at a hotel? The answer was no. Eight of the 11 users were at the same hotel, attending a course.
Were all the users using the same Operating System? The answer was no. Some were using Windows 7, some XP, some Vista.
Were all the users using the same Anti-Virus Software? The answer was no, there were diverse selections.
Were all the users up to date on their Anti-Virus Software Definitions? The answer was no.
Had all the users accessed the TapRooT® website in the past few days? The answer (of course) was yes.
Through this interrogative process, we were able to determine that two key factors were highly correlated with the infection:
1.) The users who were affected were out-of-date with their virus definitions or had no Anti-Virus Software installed
2.) All the users had visited www.taproot.com in some way.
It would have been easy to conclude from this data that finding number 1 was the root cause of the issue. This is where the investigation stops for most.
As far as finding number 2, ALL people who attend TapRooT® courses have visited our website, either to download the software or to register for the course. So why would that be significant?
In fact 4 of the 5 people consulted (the same group who recommend Trident!) on this issue believed that finding number 1 was the explanation for the issue.
But TapRooT® has taught us that if one person says ‘yes’ and the others say ‘no’, you check ‘yes’ and proceed down that path.
And so we investigated further.
Step 1 – Virtual Machines were set up of all known operating systems (XP, Vista, Windows 7)
Step 2 – All Virtual Machines were fit with all known browsers (Internet Explorer, Firefox, Google Chrome)
Step 3 – All combinations of Operating System and browser were tested with the software downloads, and www.taproot.com links
This test reproduced no instances of virus or any issue whatsoever.
The investigation continued.
We performed a scan of all files on our website to determine if any of them had been updated. This test found no changes or vulnerabilities whatsoever.
And so we investigated further.
Having eliminated the machine/browser, as well as the website, we investigated the server. A scan of the server found no changes or vulnerabilities whatsoever.
At this point in the process is when the 11th report of the incident came in. We felt we had eliminated the Machine/Browser possibility since we could not replicate it. We also felt we had eliminated the website and the server. Yet at this point, we cannot ignore that we had just received our 11th report of an issue.
So despite all the evidence to the contrary, we took immediate corrective action and shut down our website. Even though this virus is running rampant on the internet and we can find no correlation to our website, we shut it down (and by doing so closed down our e-commerce store and lowered our potential website hits on our busiest day of the week).
Anyone who knows Mark Paradies knows that this caused him great pain, but he signed the order.
And so we investigated further.
We contacted our server host and asked for an additional virus software installation and scan. We signed a contract and they promised to expedite the process for us.
Results
Here is where we found after nearly 8 hours of investigation:
Causal Factor 1: The scan had returned a result that a remote virtual directory of our website had been compromised.
Corrective Action 1: We quarantined and subsequently removed all the offending files in this directory. As of this writing, we are performing additional scans to detect any other vulnerability, to perform a validation of this Corrective Action.
Causal Factor 2: The remote virtual directory had different permissions than other folders in our website, as it existed long before our new website.
Corrective Action 2: We updated this directory’s permissions to prohibit further compromise.
So after all this, our website is back up and running.
I certainly do not want to appear to be celebrating the fact that our website was hit by a virus. Believe me, I do not. Yesterday was a very long day.
What I do celebrate is that we can work with the 11 affected individuals to make sure we restore them to complete functionality, and that this malicious virus did not affect the thousands who visit our website each day, especially on ‘Newsletter Tuesday’.
I know for a fact that would not be the case if I worked elsewhere. I’m glad we practice what we preach. It made this a small problem and not a larger disaster.
