Chatbots can write smooth, organized answers that are wrong. When an AI presents something false as fact, including made-up details, quotes, or sources, people call it a hallucination. It is one of the most important limits to understand, because a wrong answer and a right answer can read exactly the same.
Each reply is written the way a chatbot might state it. Decide, then see.
Bot 1: "Venus is the hottest planet in our solar system, hotter even than Mercury, because of its thick carbon dioxide atmosphere."
Bot 2: "The Great Wall of China is the only human-made structure you can see from the Moon with the naked eye."
Bot 3: "Lightning never strikes the same place twice, which is why tall buildings are safe."
Bot 4: "A day on Venus is longer than a year on Venus: it takes about 243 Earth days to spin once, but only about 225 to orbit the Sun."
Bot 5: "The Eiffel Tower was completed in 1889."
Bot 6: "According to Dr. Helen Marlowe's 2019 study in the Journal of Applied Birds, 83% of crows can recognize individual human faces."
Remember the language-model dice from 1.1.1. A language model builds a reply by choosing likely next words. Training taught it what plausible text looks like, and plausible is not the same as true. By default it isn't looking a fact up in a table. It is producing words that fit.
To catch a mistake you have to know the right answer, or know how to find it fast. That's why you test a chatbot on a topic you know well: a game, a sport, your instrument, your town, a school subject, a hobby. You are the expert in the room.
| Type | What it looks like | Why it's useful |
|---|---|---|
| Specific fact | A name, place, rule or detail from your topic | Ask about something you know cold |
| Number or date | A score, year, count or measurement | Easy for you to check against a reliable reference |
| Source or quote | "Who said this?" or "give me a source for that" | Bots can invent sources that look real |
| False premise | A question that assumes something untrue | Does it go along or push back? |
| Obscure but real | A true detail few people write about | Less common facts are more likely to be guessed |
Then ask again in different words. You did this in 1.1.1 and 1.1.4: reword a prompt and compare. If two versions of the same question get different answers, at least one is wrong.
For every question, write down the tool you used and the date, your exact question, the bot's answer, how sure it sounded, how you checked, and whether it was right. Then count. Counting turns "it got some wrong" into something you can compare.
| # | Question type | Bot sounded | Result |
|---|---|---|---|
| 1 | Specific fact | completely sure | |
| 2 | Specific fact | completely sure | |
| 3 | Number or date | completely sure | |
| 4 | Number or date | completely sure | |
| 5 | Source or quote | completely sure | |
| 6 | Source or quote | hedged | |
| 7 | False premise | completely sure | |
| 8 | False premise | completely sure | |
| 9 | Obscure but real | completely sure | |
| 10 | Obscure but real | hedged |
A check has to come from somewhere independent of the answer. Some things feel like checks but aren't.
For a source or quote, go find it yourself. Does the article, author, and journal exist? Does it say what the bot says it says? If you can't find it, treat it as invented. Do the same for a summary at the top of search results: it is also written by AI and also needs checking.
Ten questions is a small test. Suppose a chatbot is really wrong on 20% of questions like yours. The chance that ten of your questions come up completely clean is still about 11%. Use the sliders to see how test size changes that.
Chance you would see no mistakes at all: 11%
| Questions | 5 | 10 | 20 | 50 |
|---|---|---|---|---|
| at 20% wrong | 33% | 11% | 1% | 0% |
The math: the chance every question is answered correctly is (1 − error rate) multiplied by itself n times. It assumes each question is independent. A short test can look clean even when the bot makes mistakes fairly often.