Somewhere Between Incompetence And Dereliction

When I worked in IT, I was in and out of both the security and contingency planning areas between the early 1980s and early 2000s. I was in at the start of both fields, and I followed them through the transition from IBM mainfrane to Unix and Windows networking. In light of my experience, I've had some big questions about the story we've been getting about the Colonial Pipeline hack. On one hand, a corporation is prudent to minimize the amount of public information it releases about its internal operations. On the other, based on what I read, I've got to think that at some point, the Colonial board of directors will need to clean house thoroughly. You don't pay $5 million ransom just like that, after all.

The broad outlines of the picture so far are all we've been told: somehow, some freelance Russian hackers were able to disable the company's software that controls its pipeline operations, they demanded a ransom to have the system restored, and the company was forced to operate its pipeline manually until it paid the ransom, reportedly $5 million. In the meantime, gasoline and other product distribution was impacted over much of the US.

Here's the first issue. Pipelines are among the industries that are heavily regulated. This is because they're critical. We've just seen what happens when a pipeline goes down. Banks are the same way. If an ATM network were to go down for several days, it would be even more catastrophic, but the pipeline is clearly bad enough. Bank regulators require that banks have comprehensive plans in place, including physical backup facilities, to recover within hours if any such outage were to happen. A bank can be declared insolvent if regulators don't think its backup measures are adequate, and with good reason.

Since the 1980s, public corporations in particular have had to assess the criticality of their business functions, if only to recognize what an outage for x amount of time will cost them -- this includes not just loss of accounts receivable, but secondary costs in terms of lawsuits, loss of reputation, and political fallout. In general, a business subject to external and internal audit should have long since identified the level of outage that would threaten its existence and have plans and physical backup measures in place to recover from any such outage. This would range, depending on the business, from local power outage to civil unreast to hurricane to anything short of the big asteroid hitting the planet.

These measures can be expensive, but the issue is how much more expensive it would be for a bank or public utility to have its system unavailable for days or weeks. This could threaten a company's ability to remain in business, and its board has a fiduciary responsibility to shareholders to assure it can continue.

As far as I can see, Colonial Pipeline had no such plans in place, or if it did, it didn't have anyone who could implement them. For instance, if the issue was that hackers had locked the system software, there should have been a clean backup copy offsite, and if necessary, a place with a backup system available to download and run the backup software to control the pipeline remotely.

The question isn't just whether the company's managers were incompetent or derelict, the question would be whether the company auditors, both inside and outside, and its regulators were also derelict. Remember that a major impact of the Enron scandal was to destroy the Arthur Andersen audit firm, whose dereliction allowed the scandal.

A secondary question is how the hackers were able to gain access to the computer system that controlled this critical infrastructure. All I can surmise from the information that's been made public is that the hackers were somehow able to lock or disable the operating copy of the system software, but could then either unlock it or provide a clean copy on payment of the ransom.

Someone who wsnts to do this needs both physical and electronic access to the company's computers. Controlling this has been IT Management 101 since the 1970s. The various root or superuser IDs that can change system software are normally carefully controlled and limited. I've read speculation that this was an inside job. Could certainly be, and there should normally be just a few people who'd be the ones to ask. If it were more than a few, the IT director and most of his chain of command should already be on their way out.

Another issue, especially if it were an inside job, is whether drug testing is being done. Pipelines are regulated by the US Department of Transportation, which requires drug testing for employees involved in operations. I've always thought, given my experience in the work environment, that drug testing and alcohol screening would go a long way to solve corporate dysfunction, but maybe because that's the case, it isn't done much, even when it clearly should be.

There really ought to be congressional hearings on this. But then, congress is also stalled somewhere between incompetence and dereliction.