Twitter vs FTX

A past allegation against Twitter has resurfaced, and if it's correct, it raises questions about whether Twitter has been nearly as big a swindle as FTX. The questions involve areas that I worked on in my tech career, so they're particularly interesting to me, and based on my own experience, they at least appear credible.

This past September, during the period when Elon Musk was attempting to get out of his obligaation to buy Twitter, Musk

accused Twitter of fraud by concealing serious flaws in the social media company’s data security, which the entrepreneur said should allow him to end his $44 billion deal for the company, according to a Thursday court filing.

. . . Musk said the claims by the whistleblower, former head of Twitter security Peiter “Mudge” Zatko, amounted to fraud and breach of contract by Twitter.

I don't believe the maneuvers between Musk and Twitter following Musk's September allegations and his late October decision to proceed with the purchase have been made public, or if they have, I haven't seen them. In any case, his September allegation of fraud due to data security shortcomings wasn't sufficient to stop the deal. However, Peiter “Mudge” Zatko's claims have resurfaced in the wake of Musk's Twitter Files revelations.

The whistleblower report has emerged independently of Musk’s “Twitter Files” but was likely inspired by the exposés.

A Twitter user who published the report said: “The stuff uncovered in the Twitter whistleblower report is much crazier than anything in the ‘Twitter files’ but it’s much less politically/tribally salient so it got no attention.

. . . “Twitter didn’t monitor employee computers at all, it was not uncommon for employees to install spyware on work devices.

“Twitter does not have separate development, test, staging, and production environments.

“At least 5,000 employees had privileged access to production systems.

“In 2020, Twitter had security incidents serious enough they had to be reported to the federal government on an almost weekly basis.

The difference between "test" and "production" environments is particularly important, as is the related question of how many employees had privileged access to "production systems". In a programming environment, mistakes will inevitably be made in writing programs, especially given the tricky logical issues programmers have to deal with -- what you actually told the machine to do may not be quite what you meant for it to do; that's just how logic is. If I tell the driver of a car to "go ahead and back up", he'll know what I mean. A computer instead will either give me an error message or maybe move forward, not reverse.

As a result, programs must be tested exhaustively before they're approved and moved to "production" status, at which point they are basically the company product, and they can't be changed without a repeat of the testing and approval process. This means that it's important to get things right the first time, because fixing things afterward is expensive and time-consuming.

A key reason for the corporate data security function is to maintain these controls between "test" and "production" ("Mudge's" use of the term "staging" sounds more company-specific; the distinction between "test" and "production" is the important one).

If there is no enforceable distinction between the two, then there are two posssibilities. The most likely is that software with bugs will become in effect the company product, which at best will make the users angry, but if it results in incorrect account balances, it could be a major problem for the company. The less likely but more serious possibility is that a programmer will deliberately write a program that steals money from the company or its customers (for instance, by incorrectly rounding and putting a tiny percentage of the rounding into the programmer's own account). The absence of a rigorous testing and approval process for "production", or the ability to bypass the process, will make these possibilities much more likely.

That "Mudge" claims that 5,000 employees (in other words, half of Twitter pre-Musk) had privileged access to "production" indicates that there were no serious company controls over its tech environment -- this in a major tech company. This is equivalent to the alleged "back door" between FTX and Alameda that allowed the company to raid customer deposits to make up for investment losses, except that instead of a few FTX executives who could do this, half of Twitter could do that there.

Farther down in the same story,

Mudge realized that a data center failure could potentially cause the permanent loss of all of Twitter’s data.

He shared this fact with senior leadership, who instructed him not to put it in writing for the Board.

. . . After Agrawal became CEO, he wanted to present materially misleading information to the Board, overriding Mudge’s objections.

The question of data center failure is another basic task of data security. If, as relatively recent history demonstrates, corporate data centers can be rendered inoperable by fire, flood, hurricane, sabotage, terrorist act, earthquake, power failure, or any other of a wide range of interruptions, the company must have contingency plans to accommodate this. Clearly if a bank's data center burns down, there are repercussions; the ATMs won't work, payrolls won't be deposited, checks will bounce, and so forth. Banks have elaborate plans to recover operations within hours with full current data in such an event, but banks are just the most visible instance of an overall problem.

"Mudge's" point seems to be that Twitter had no viable plan in place to deal with, say, an 8.0 earthquake on the San Andreas Fault near its data center. If such an event took place, the implication is that Twitter would simply go down and never come up again; the company would no longer have a product to sell and no way to recover it. The effect would be the same as an FTX bankruptcy. This in fact is board-of-directors-level material; if Parag Agrawal concealed it from the board, which is "Mudge's" allegation, he should have been fired, and of course, he was. The story continues; when he learned of this,

Musk responded saying: “Wow.”

“Live & learn,” he added.

My view continues to be that Musk makes things up as he goes along, and he appears to work from instinct. He isn't a rationalist, policy-driven CEO along the lines of John Ray, instead, it's "Live & learn". Nevertheless, at this point, I think it's clear that in firing half or more of Twitter and starting over from scratch, he made the only possible move. In fact, if Twitter had continued under prior management, there would likely have been either a software failure or a data center disaster that would have put the company out of business and, if anything, thrown even more people out of work.

I certainly agree with the guy who said this is "much less politically/tribally salient", but it's actually more important. Musk has said he intends to hire a CEO for Twitter, but this needs to be a rationalist John Ray type, not a Steve Jobs style figurehead.