AI-Based Cybersecurity Attacks

This lesson is about how an adversary can use AI-powered tools to augment a cyberattack, and how to defend against some of those attacks.

We follow one public high school through every section. Its principal appears in 40 morning-announcement videos on the school's public channel. Its website carries a staff directory, and a local news story about the school is online.

The adversary starts with an AI-powered reconnaissance tool. It combs the school's social media and every public website that mentions the school, and returns the profile below.

Every entry came from something the public can reach: a video or a web page. The tool gathers them into one profile.

The Principal entry lists 40 videos of about 3 minutes each. That is about 2 hours of one person's face and voice.

The adversary generates an avatar of the principal from the 40 videos. The Records vendor entry names the company that hosts the school's student records. The avatar joins a video call with the company's support desk and asks for an export of every student record. If the desk sends it, private information about every student has gone to the adversary.

The school and the company set a phrase in advance, when they signed their contract. It has never been published, said on video or written on any page, so no reconnaissance tool can find it. Before exporting any records, the support desk asks for the phrase. The avatar has the principal's face and voice but not the phrase, so the request fails. The same defense works between people. Close friends and relatives who agree on a phrase can check each other when a call sounds right but asks for something odd.

The Staff phone entry describes voice-based authentication. Every staff caller says the same phrase, and the phrase is printed on the district help page, so it's no secret. What the line checks is the voice: it compares the caller's voice with a recording on file. The shared secret stopped the avatar because it tested what the caller knew. This line tests only who is speaking.

The entry also says whose voice is public: the office manager recorded the greeting every caller to the school hears, and the help page posts it. From that recording the adversary generates a copy of the office manager's voice, says the printed phrase in it, and the line accepts it. That opens the office manager's staff account. More organizations are adopting voice-based authentication, so the potential impact of voice impersonation grows with them.

The defense is to enable multifactor authentication (MFA) on the line. A matching voice is then one factor. The caller must also type a one-time code from the authenticator app on the office manager's phone. The adversary copied the voice but doesn't hold that phone, so the staff account stays locked.

The Staff entry lists 85 email addresses, 23 of them for Spanish speakers. A large language model (LLM) is a generative AI tool, and it can produce a convincing phishing message in any target language. The adversary has an LLM draft one email in English for 62 staff and one in Spanish for the other 23.

Some traditional phishing was written by non-native speakers of the target's language. So unnatural language has been one feature used to tell a phishing message from a legitimate one. The two drafts have no such feature, since each reads as a native speaker's writing. For these 85 staff, the wording no longer gives the email away.

An LLM is trained on large data sets. Some AI-powered tools also feed user input back into the LLM to provide continuous training. Secure or sensitive information in an LLM can therefore come from either source. Adversaries can craft prompts that extract it.

Both sources matter at the school. Years ago a staff spreadsheet with home phone numbers was posted on a public forum by mistake and taken down a day later. A model trained on a large data set that included the forum could hold those numbers. An adversary who crafts the right prompts could pull them back out. User input is the other source. So the school has a rule for every staff member: personal or sensitive data is never entered into AI-powered tools, such as chatbots or virtual assistants. Data that was never typed in can't be extracted through that path.

The training data can also carry what an adversary put there. The Registration entry says registration for next fall closes March 20. The adversary publishes one website and edits an existing community page, and both give the deadline as April 20. The goal is for that false information to be included in the training sets for LLMs. A model trained on those pages could then repeat the false date. Two pages against the school's own site and a news story may not be enough, and training takes months. An AI assistant that searches the web as it answers is the quicker path. It can find the new pages and tell a parent that registration closes April 20.

Output from AI-powered tools should be carefully evaluated. A parent who gets that answer checks it against a reputable, stable, non-AI-based source: the school's own website, or a call to its front office. The website gives March 20, the date the school published, and that's the one to register by.

AI-enhanced coding tools can help an adversary write new malware. They can also help rewrite part of an existing application to do harm.

The Records system entry is a code base of about 2 million lines, all of it published. The adversary has a coding tool search it for vulnerabilities. A vulnerability found there could expose the same student records that the support desk kept from the avatar.