Principles of AI · Unit 1: Inside the Machine ·  Activity 1.2.1

Password Wars — Attack and Defend the Kiosk

A kiosk that guards a secret, two roles, and a class tournament  ·  Approx. 2 class days

StarringExplore · AI CityDefend · Set GuardrailsAttack · Test the Kiosk
Unit 1: Inside the Machine

Quick reference for deterministic vs. probabilistic systems, the path of a request, and why guardrails can be bypassed.

Password Wars infographic: deterministic versus probabilistic machines, the path of a request through input guardrail, assistant and output guardrail, common attack tricks, why usually-works guardrails fail, and the least-privilege fix

🧭 Plot Summary

AI is probabilistic: it weighs chances rather than following one fixed set of steps, so the same question can get a different answer. That is a strength when you want a fresh story. It is a problem when you need a system to reliably keep a secret.

In this activity you first explore AI City, noticing which parts behave the same every time and which don't. Then you head to the Kiosk in Prompt Plaza. One player is the AI Safety Engineer, who hides a password and builds guardrails. The others are Penetration Testers, who try to make the kiosk give it up. Round by round, the guardrails get stronger and the attacks get sneakier. Your results become points in a class tournament.

What you'll do in this activity

  • Run machines that give the same answer every time, and machines that don't, and learn what makes the difference.
  • Meet prompt injection, guardrails, and data leakage through a kiosk that guards a secret password.
  • Play both sides: the safety engineer who sets the guardrails, and the penetration tester who tries to get around them.
  • Keep a log of each attack, each guardrail, and how many attempts it took.
  • Compare results with another team and find out which guardrail held up better, and why.
  • Work out why "usually works" is not good enough, and what a defender can do instead.

Ground rules

  • Penetration testing is only legitimate with permission. Here, the game is the permission. Don't use these tricks on a real system.
  • Play fair: the tester may not look at the engineer's screen.
  • Use a made-up password. Never use a real one.

Why it matters

Companies now put AI tools in front of customer records, internal documents, and company accounts. If the guard is only a list of forbidden words, a patient attacker walks right past it. Knowing why helps you decide what an AI tool should be trusted with, and you'll use it again in the Red-Team Report (1.2.4).

✅ Self-Check Before the Tournament

Check off each item as you get there. These aren't grades. They're your own signal.