This week I used homemade exercise cards to explain confidence intervals to colleagues, leaving my old slides unopened. Each card showed an interval for a mean calculated from a random sample, all at the same confidence level. The population mean was printed on the back, which we kept hidden. The data were simulated for the exercise, not taken from a product test.
I asked everyone to choose a card they would be happy to put in a report. Some chose the narrowest interval; others wanted to exchange a wide one. I had intended to discuss precision. Instead, we were debating whether a particular card was right. Holding a single result made it easy to hear a claim about the method’s long-run performance as a probability assigned to that card.
We paused and turned all the cards over. Some intervals covered the population mean and some did not. Then we turned them face up again: the intervals were visible, the answer hidden. With the same population, sampling process and calculation, another sample would produce another interval. We could not stamp this particular one with “a 95% probability of containing the true mean”.
The 95% refers to the long-run proportion of intervals that cover the population mean when sampling and calculation are repeated under the method’s assumptions. Once calculated, an interval either covers it or does not. That is a short sentence in a textbook. Explaining it aloud, I nearly said “we can be more than nine-tenths sure about this one” myself. The convenient wording was exactly where the misunderstanding began.
I had also prepared exercises about sample size and variability, but we did not rush on. We spent longer with the widest card. If its range allowed for business outcomes that differed substantially, what would a report hide by giving only the point estimate? By then, the handsomest card was no longer the main attraction.
The NIST statistical handbook makes the same distinction between an individual interval and the coverage of its calculation method. I sent that page to the group so the next explanation would not depend on my improvised wording.
When I collected the cards, someone had pencilled a question mark on the back of one. I left it there and put it back in the pile. I should recognise it next time.