Pretending to Know Was the Scariest Part
I asked a bot running on my Windows device, 「Where is the folder I made yesterday?」 It confidently gave me a path that did not exist. No error appeared, so I only noticed much later. That led to a small experiment.
Key point: a bot does not have 「I don't know」 as a default state. You have to give it permission to say so.
Steps You Can Follow
1. Fix 10 questions
From the questions you often ask in a project, pick 6 whose answers truly exist in your files, then add 4 that deliberately have no answer. Write them in a notepad. You will reuse this list every time.
2. Ask as they are and count
If a question with no answer gets a plausible reply, mark it as a 「made-up answer」. In my case it was 4 out of 10.
3. Add three lines to CLAUDE.md
• If something cannot be verified, first write 「not verified」.
• If it is a guess, put [guess] at the start of the sentence.
• Write the source file name on the last line.
4. Ask the same 10 in a new conversation
An old conversation would mix the results, so always start fresh. The result was 1 made-up answer.
5. Catch the last one with the source line
I opened the file named on the last line and the content was different. With a source line, even the wrong answer was spotted in 10 seconds.
Wrap-up
• Rules worked better as permission (「you may say this」) than as a ban.
• A fixed question list lets you compare every rule change with the same yardstick.
• Ten questions is a small sample, so the numbers are not absolute. It was still enough to confirm the direction.
Before lunch today, write down just 10 questions. Five minutes is enough.