Quick reference for hallucinations, question types that catch them, the five kinds of mistake, and real checks.
🧭 Plot Summary
A chatbot can write a smooth, organized, sure-sounding answer that is simply wrong. People call that a hallucination, and it can include made-up facts, quotes, and sources. The reply doesn't come with a warning label, so the warning has to come from you.
Your mission is to stump the bot. You pick a topic you know well, ask questions designed to catch mistakes, and check each answer against a source that doesn't depend on the bot. Then you count: what kinds of questions fooled it, and how often did it sound completely sure while it was wrong?
What you'll do in this activity
- Pick a topic you know better than most people, so you can spot mistakes the bot would slip past someone else.
- Ask a chatbot a mix of questions designed to catch it: facts, numbers, sources, trick premises, and obscure details.
- Re-ask in different words and see whether the answers agree.
- Check every suspicious answer against a source that is independent of the bot, and log how you checked.
- Count your results and find out which kinds of questions fooled it most, and how often it sounded sure while wrong.
- Write one honest sentence about what your test does and doesn't prove.
Ground rules
- Don't type personal information into the chatbot, about you or anyone else.
- "Are you sure?" and "the bot sounded confident" are not checks. Use a source outside the bot.
- Report what you actually found, including answers you couldn't confirm.
Why it matters
People now use chatbots for homework, news, and decisions. Knowing where they slip, and having a habit for checking, is the difference between using AI and being misled by it. You'll use the same habit in the Red-Team Report (1.2.4) and every time you verify AI output later in the course.
✅ Self-Check Before You Share
Check off each item as you get there. These aren't grades. They're your own signal.