Cybernetic Entomology: Adware Certificate Authority

In February of 2015, Lenovo, a maker of computers, was installing a piece of software named Visual Discover from the company Superfish with the operating systems with which their laptop computers came preloaded.

Visual Discover software was “adware” – a browser extension/proxy which monitored web browsing and inserted ads.

Adware can’t insert ads into web pages unless it can modify the text of the pages in order to insert ads. But for https and ssl connections, the pages were authenticated using keys from certificates signed by Certificate Authorities. If Superfish modified the bits seen by the browser, it had to sign the modified bits using the private key of a trusted certificate authority, or the modified bits would not check against the website’s certificate. Superfish didn’t have access to those keys, so it deliberately sabotaged system security.

In order to allow its adware to serve ads in SSL and TLS protected web traffic, Superfish’s installer added a (self-signed) root CA into the Windows Trusted Root Certificate store of every machine on which Visual Discover was installed. The adware could then create a certificate on the fly for any SSL or TLS host, modify the pages to insert ads, and sign the modified bits including the ads using the generated certificate. That way, the browser would see a valid certificate signed by a trusted CA, and the pages with the ads would be served even when the user was using an authenticated connection.

The whole purpose of certificates and SSL and HTTPS and TLS and all those things is to ensure that the bits seen by the browser have not been modified between the host server and the user’s browser or client software. What Superfish was doing is therefore deliberate sabotage. They developed a way to defeat the entire purpose of that part of the operating system.

But it’s worse than that.

The CA once installed in the Windows Trusted Root Certificate store, was not uninstalled when users uninstalled Visual Discover. This left the machines vulnerable to bogus certs even after Visual Discover was uninstalled.

But it’s even worse than that.

In order to generate a certificate signed by a CA, you have to have the CA’s private key. And Visual Discover, which Superfish distributed to the public via Lenovo’s bundle, had that key so that it could generate certificates on the fly. The private key was embedded in every copy of the software. This enabled anyone who read it from the executable file to forge a certificate for any website.

This made it possible, for example, to put up a site pretending to be a bank’s website, and fool anyone connecting to it from a machine with Superfish’s CA installed, thus harvesting the user’s banking credentials in real time.

But it’s even worse than that.

While Visual Discover was running, it would modify the bits coming through authenticated connections inserting ads, generate a cert that would be seen as valid, and then re-sign the modified bits with that cert, even if the original cert was not valid. This meant that any cert, even a forged or revoked cert, regardless of its validity or origin, would result in a supposedly authenticated connection for afflicted computers.

When the problems were explained to them, Lenovo issued a statement saying, and this is a direct quote, “We have thoroughly investigated this technology and do not find any evidence to substantiate security concerns.”

Adi Pinhas, the CEO of Superfish, followed up with an absolutely stunning display of shameless sociopathic lying, saying, “Despite the false and misleading statements made by some media commentators and bloggers, the Superfish software does not present a security risk. In no way does Superfish store personal data or share such data with anyone. Unfortunately, in this situation a vulnerability was introduced unintentionally by a 3rd party.” This slanders the security pros who called him on his sabotage, completely ignores what the security problem actually was, claims the software failed to introduce an entirely unrelated kind of vulnerability, and then blames the sabotage on a third party while claiming it was accidental. The idea that any third party is responsible for Superfish’s deliberate sabotage is not even laughable – it’s actionable.

Now, it’s one thing when someone holds up an apple and says it’s a quince. He might not know the difference. You can work around that. When he goes on to hold up an orange and call it a grape, you realize he’s using an entirely different language than you are or possibly insane, and nothing he says should be interpreted according to the rules of normal discourse. That’s becoming a major problem, and a fatal flaw in a supplier or business partner. The more you depend on such a person, the more you should be worried.

But then when the same person holds up a banana and claims that it’s a small off-duty Czechoslovakian traffic warden, there’s really no escaping the obvious conclusion any more. There is no longer any possibility that this is merely some kind of confusion, stupidity, or even insanity. This is even worse.  This is a deliberate, bald-faced lie.  And what Lenovo said in its statement, and what Pinhas followed up with, is even more absurd than claiming a banana is actually a small off-duty Czechoslovakian traffic warden. Robert Graham (CEO of the company Erratta Security) called Lenovo’s statement a bald-faced lie, and let’s face it, he was right.  Pinhas’ statement was a bald-faced lie too, and also a self-serving slander.

There is a long history – perhaps even a tradition – of telling bald-faced lies when caught in security failures. It doesn’t help. At best, your customers will believe that you are insane. And they really don’t want to be doing business with insane people. When it becomes obvious that you’re lying, that’s even worse as far as they are concerned, because absolutely nobody wants to do business with a crook, and as far as the public is concerned, liars are crooks. Apologizing for your failure may leave you vulnerable to lawsuits in this liability-obsessed age, but with an apology it is at least possible to retain your customers’ trust and respect. And if it matters to you, it’s possible to retain your self-respect as well.

Lenovo apparently reached the same conclusion by the following day, when CTO Peter Hortensius, in an interview with the Wall Street Journal, said, “We agree that this was not something we want to have on the system, and we realized we needed to do more.   Obviously in this case we didn’t do enough.”

Lenovo’s next action was to turn the software “off” – meaning it ceased to modify traffic between the hosts and client software. This did not help fix the main security bug, because that root certificate was still in the trusted authorities for the afflicted machines, and a phishing website “authenticating” itself with a certificate signed by that key could still rip off Lenovo customers all day long.

As of this writing, Lenovo has further responded to the issue by ceasing to install Visual Discover or Superfish’s root CA on new machines, has published instructions for removing the offending CA from the Windows Trusted Root Certificate set, and several security specialists have used the CA’s private key, lifted from the Visual Discover software itself, to issue a revocation for the CA.

Unfortunately, the handling of revocations is inconsistent and unreliable. It works so badly, in fact, that you never ever want to be in the position of having to do it. In some cases they are handled by the operating system and in others by the browser or client software, and in some cases there is not agreement between the two, or some types of client software never get the benefits of the revocation. Multiply this by all the combinations of operating system and software that relies on the certificate system for security, and just distributing the revocation to machines on your local network, in a way that allows confidence, becomes difficult.

At this writing Mozilla has an issue on their tracker to install the revocation by default, marking the offending CA as “untrusted”, and presumably other software makers that depend on the SSL certificates (but whose bug databases are not public) must also be responding.

Installing a CA degrades the security of the machine, and only the owner of the machine knows how much the security of each particular machine is worth. Therefore the owner of the machine is the only party who can make an informed decision about whether the security damage is worth it.

Installation of a CA is a huge deal. We have traditionally allowed operating system and browser providers to add CA’s without stringent justifications for each one, but that questionable practice at least results in a well-known set that will be reviewed by many people. Other than those special cases, installing a CA must be done only with the knowledge and consent of the machine’s owner.

Software design checklist:

• Is the purpose or method of operation of your software directly in conflict with the purpose of some part of the operating system it will be running on? If so, what you are doing is called “sabotage.” Stop immediately and develop a different product.
• Have you found a way to accomplish sabotage? If so stop immediately and report it as a bug to the manufacturer of that operating system or the afflicted client software or both.
• If your software deliberately degrades the operation of the computer it runs on, including the security of that machine, does it do so with the express knowledge and consent of the computer’s owner?
• Is there a straightforward, simple, complete uninstallation procedure that doesn’t leave anything else broken?

Software bundling checklist: If you’re creating a “bundle” of software from various vendors:

• Has each vendor signed a contract accepting liability for security bugs afflicting the operating system as a whole or co-signatories’ software in particular, which arise as a result of the installation, operation, or failure of that vendor’s software?
• Has each vendor provided an uninstaller that cleanly and completely uninstalls their software, including any CA’s or other security-affecting bits it has installed?

CA checklist: If your software does anything with security certificates:

• Does it make sure that the cert is valid first?
• If there is ever need to check the validity of a cert, have you tested your software to be sure it refuses every different kind of invalid or malformed cert?

CA installation checklist: If your software installs a root CA:

• Does it do so with the express knowledge and consent of the machine’s owner?
• Does it advance the interests of the machine’s owner?
• Have you refrained from publishing or distributing anything, including executable software, that contains its private key?
• If the certs your CA authenticates are supposed to be used only local to the machine, can you configure it so that they are refused when presented from remote connections?
• When the software you’re adding a CA for is uninstalled, is the CA automatically uninstalled along with it?
• Are you able to publish a revocation for that CA? If so can you publish and distribute it in a way such that it will have the effect of invalidating its use completely for all purposes and all client software?