Detection Tools and Thresholds

On the bus agency's network, the firewalls write down each connection they allow or refuse, the servers every sign-in, the dispatch system every message to a driver, and the staff computers the programs they run. The firewall at the edge of the trip planner alone writes about 2,800 entries in an ordinary minute. An entry that points to trouble, such as a sign-in from an address no one has used before, could be anywhere among them. At two seconds an entry, how long would a person need to read one minute of that firewall's records?

At 2,800 entries and two seconds each, reading one minute of records would take about an hour and a half, and by the time the reader finished, another 93 minutes of records would be waiting, from that one firewall alone. Nobody reads them that way. Automated tools read them, and people read what the tools pick out.

At the edge of the trip planner's segment, the agency runs a network intrusion detection system, a NIDS. It reads the traffic going in and out and raises an alert when something looks like an attack, such as one address opening thousands of connections to the site in a few minutes. A NIDS only raises the alarm. Its sibling, the NIPS, can also act on its own, before anyone has opened the alert.

Some attacks show up only when records from different places are read side by side. One evening a dispatcher's account signs in to the agency's VPN from another country. On its own that might be a trip. But the depot's door log shows the same dispatcher's badge was used to walk in forty minutes earlier, and no one crosses an ocean in forty minutes. A SIEM, short for security information and event management, gathers records from many sources so that a pattern like this one can be caught. It raises an alert, and an analyst looks into it.

The agency's newest tool uses AI. Trained on examples of attacks and of ordinary activity, its model gives every event a score: how likely, as a percentage, it is to be malicious. The team decides the score at which an event becomes an alert. Set that line high and the team sees only the events the model is surest about, and a real attack that scores lower goes by unseen. Set it low and the alerts pile up faster than three people can read them, mostly ordinary activity. Where to draw the line is a judgment the team has to make for itself.

FireEye, one of the best-known security companies in the world, got an alert in late November 2020 that reported something small. A second phone had just been registered to an employee's account for multifactor sign-in. Whose phone was it?

Phones are added to accounts all the time, and the alert was rated low. An alert rated that low helps only if a person reads it, and a member of the security team did: they called the employee and asked whether the new phone was theirs. It wasn't. Someone else had the employee's password and had added a phone of their own, so that the sign-in codes would come to them. Pulling on that thread, FireEye found that intruders had been inside its network and had stolen the tools its own staff used to test clients' defenses. Then it found how they had gotten in: through an update to SolarWinds Orion, network-management software used by thousands of organizations, which the attackers had secretly altered. SolarWinds later estimated that fewer than 18,000 of its customers had installed the altered update. Among the organizations the attackers went on to break into were several U.S. government departments, including the Treasury and Homeland Security. The United States said in April 2021 that Russia's foreign intelligence service was responsible.