Simulation vs Experiment

A simulation answers a probability question by generating outcomes from a model you have assumed and then counting. Fix the model, define exactly what one trial is, run many trials, and report $\hat{p}$ (p-hat) as successes divided by trials. What comes back is an estimate of the probability under that model. It is not the probability itself, and it is not evidence the model is true.

Take a player who makes 70 percent of her free throws with shots independent, and ask for the chance she makes all four of her next attempts. The exact answer is $0.70^4 = 0.2401$. One run of 20 simulated trials gave 3 successes, an estimate of 0.15. A separate run of 10,000 trials from the same model gave 2,335 successes, or 0.2335. Same model both times, and the longer run lands closer.

The misreading: my simulation gave 0.15, so the probability is 0.15. An estimate from $n$ trials has standard deviation $\sqrt{p(1-p)/n}$, which at $p = 0.24$ and $n = 20$ is 0.096. Two standard deviations either side of 0.24 spans roughly 0.05 to 0.43, so a 20-trial run landing on 0.15 is unremarkable. Reporting it as the answer treats a number with that much slop as exact.

Precision improves slowly, and the square root is why. The standard deviation falls with $\sqrt{n}$, so halving the error takes four times the trials and cutting it to a tenth takes a hundred times: 0.043 at 100 trials, 0.0043 at 10,000.

The harder limit is the model. That simulation tells you what a 70 percent shooter would do. It cannot tell you whether this player is a 70 percent shooter, and if she is really at 0.60 then every figure it produced answers the wrong question. Topic 2.3 in the Fall 2026 course is Estimating Probabilities Using Simulation.

An experiment imposes something. The researcher assigns conditions called treatments to experimental units and then measures a response, which is the line between an experiment and an observational study, where the groups assemble themselves. The course names four marks of a well-designed experiment: comparison of at least two treatment groups, random assignment of treatments to units, replication, meaning more than one unit per treatment, and direct control of extraneous variables.

A store tests two checkout layouts on 200 shoppers, using a random number generator to send 100 to each. Mean checkout time comes out at 84 seconds under layout A and 97 seconds under layout B, a difference of 13 seconds. Because chance decided who met which layout, no systematic difference in basket size or shopping habit is expected to pile up on one side, so a gap that size, if it is larger than chance variation would ordinarily produce, is fairly credited to the layout.

"It was an experiment, so it proves cause and effect" hides two separate conditions inside one word. Imposing a treatment is what makes a study an experiment. Random assignment is what makes its comparison causal, and the two do not always travel together. Give layout A to the morning shift and layout B to the afternoon shift and you have imposed treatments, so it is still an experiment, but time of day now moves in lockstep with layout and no causal reading is available.

The causal claim also has a limit on who it covers. Random assignment supports a conclusion about the units in the study, and extending that to a wider group takes random selection, a different act at a different stage; scope of inference works through the combinations.

Everything else in experimental design is machinery for those four marks: blocking to pull a known nuisance variable out of the comparison, a control group and a placebo to supply a baseline, and blinding to keep expectations from leaking into the response. Replication means many units per treatment, not repeating the whole study, which is the vocabulary slip the exam catches most often here.