Integrity Hashing

On Saturday, February 20, 2016, anyone who clicked the download link for one edition of Linux Mint, a free operating system, got a copy someone else had built. Attackers had broken into the Linux Mint website through its blog software and pointed the link for the 17.3 Cinnamon edition at a copy with a backdoor in it, hidden code that would give them control of any computer it was installed on. The copy sat on a server Mint traced to Bulgaria. How could someone who downloaded it that day have found out the file wasn't Mint's?

Linux Mint published a hash for each download, and a file with even one byte changed gives a completely different hash, so hashing the download and comparing it with Mint's real hash would have shown that it wasn't Mint's. The online bookstore downloads an update to its shop software every few weeks. The software company posts a SHA-256 hash beside each download, and also lists it in the release notes it emails to customers. Before installing, the owner computes the hash of the downloaded file and compares the two. The comparison for the latest update:

Published:  dfe9faead73baf1cbfc6f0535ac061cd
            709f7a43e63e695e7e184816331f1179
Downloaded: 9e3b7549e3d97e442bfc597c7eee6361
            2997f52ba1686e60a50ae6f02bbb719b

The two values look nothing alike, so the file on the owner's computer isn't the file the company published. The owner deletes the download and fetches it again, and the second copy matches.

On the bookstore's Mac the owner uses the Terminal, the program for typing commands. Each kind of computer has its own command for the same job.

Downloads aren't the only files that can change. The shop's checkout page runs a script, checkout.js, that sends each order's card details to the card processor. An adversary who could edit that one file could add a line copying every card to a server of their own. So after each update the bookstore records the script's hash, and a small program hashes the live file every hour and compares. In April it reported this:

Recorded after the update:  d5d6c7ed17e3267a...ca9b5b84
This hour's check:           296c9529b421909a...7e2e0770

What does the mismatch tell the owner, and what doesn't it?

It tells the owner that checkout.js changed at some point after the update, since the update was when the first hash was recorded. It doesn't say who changed it, how, or what the change does; for that the owner opens the file and finds a new last line sending each order form to cdn-stats[.]example.net, an address nobody at the bookstore recognizes. The hourly check also bounds the damage: the file was clean on the previous check, so the copying most likely began within the last hour, and the owner can stop it now instead of weeks from now.

A hash comparison is only as good as the hash it is compared against. Linux Mint listed the official hashes for its downloads on its website, the same site the attackers had broken into. Its leader, Clement Lefebvre, listed the site's hashes among what was affected and said the attackers could change anything in the site's database, those hashes included. Suppose an adversary controls both the download link and the hash printed beside it. What does a matching hash prove?

A matching hash would prove only that the file agrees with the page, and the adversary wrote the page. Lefebvre said Mint heard within the day because people found hashes that contradicted each other, which could happen only because copies of the hashes were posted in many places.

Mint told anyone who had downloaded that edition on February 20 to check the file and delete it if it failed. Anyone who had already installed it was told to take the computer offline, reinstall, and change their passwords. The attackers had also copied the site's forum database, with the accounts of its members, and offered it for sale. Mint's mirrors already carried a hash file signed with the project's private key, but the download page never mentioned it. Mint's instructions now tell every user to check that signature as well: the list of hashes itself can be checked with Mint's public key, and only Mint's private key could have signed it.

The bookstore's check works because its second copy of the hash came from somewhere else too. The hash in the emailed release notes doesn't depend on the download page, so an adversary who changed the page would also have to change an email already sitting in the owner's inbox.