Lesson 3.5 · Collecting Data
Scope of inference
Every study ends with a conclusion, and every conclusion has limits. The scope of inference tells you how far a conclusion can reach: to whom the results apply, and whether you can say one variable caused a change in another. Both answers come directly from how the data were collected.
Two questions, two kinds of randomness
When you read about a study, ask two separate questions.
1. Who can the results be generalized to? This depends on random selection. If the individuals in the study were randomly selected from a population, the results can be generalized to that population. If they were volunteers, a convenience sample or some other non-random group, the results apply only to individuals like those in the study.
2. Can we conclude cause and effect? This depends on random assignment. If the treatments were randomly assigned to the units, a statistically significant difference in responses can be attributed to the treatments. If not (as in every observational study), confounding variables might explain the difference, and only an association can be concluded.
These two kinds of randomness do different jobs, and one cannot substitute for the other.
The scope of inference
| Randomly assigned | Not randomly assigned | |
|---|---|---|
| Randomly selected | Cause and effect; generalize to the population | Association only; generalize to the population |
| Not randomly selected | Cause and effect; only for units like those in the study | Association only; only for individuals like those in the study |
Random selection allows generalization to the population. Random assignment allows cause-and-effect conclusions.
Each cell of the table
Random selection and random assignment. This is the ideal, but it is rare, because randomly selected people usually can't be required to participate in an experiment. Example: a company randomly selects 200 of its 5,000 employees and randomly assigns half to a four-day work week. If productivity differs significantly, the company can conclude the schedule caused the change for all 5,000 employees.
Random selection, no random assignment. This is a well-designed sample survey or observational study. Example: a random sample of 1,000 adults in a state shows that those who work from home report less stress. You can generalize the association to all adults in the state, but you can't say working from home causes lower stress. People who work from home may have different jobs or incomes.
Random assignment, no random selection. This is the most common kind of experiment: volunteers are randomly assigned to treatments. Example: 80 volunteer college students are randomly assigned to study with or without music. A significant difference lets you conclude that the music condition caused the difference, but only for students like these volunteers.
Neither. Example: a teacher compares test scores of students in her morning class with students in her afternoon class. There's no random selection (it's just her classes) and no random assignment (students chose their schedules). The conclusion is limited to an association for these students, and even that may be explained by confounding variables such as sleep or other courses.
Generalize to the right population
Random selection lets you generalize only to the population that was actually sampled. A random sample of seniors at one high school supports conclusions about seniors at that school, not about all high school students in the country. Similarly, an experiment on lab mice supports conclusions about mice like those in the lab, not directly about humans.
Worked example: An experiment with volunteers
A sleep researcher recruits 64 adults through a newspaper ad. She randomly assigns 32 to use a new mattress and 32 to keep their current mattress for a month. The new-mattress group reports significantly fewer nights of back pain. What is the scope of inference?
Solution. The treatments were randomly assigned, so the researcher can conclude that the new mattress caused the reduction in back pain nights. The subjects were volunteers who answered an ad, not a random sample, so the conclusion applies only to adults similar to these volunteers, not to all adults.
Worked example: A random sample, no treatment
A random sample of 600 students at a large university finds that students who work more than 15 hours a week at a paid job have significantly lower mean GPAs than students who work fewer hours. What is the scope of inference?
Solution. The students were randomly selected from the university, so the association can be generalized to all students at this university. The students chose how many hours to work (no random assignment), so we cannot conclude that working more hours causes a lower GPA. For example, students with greater financial need may work more hours and also have less time or support for coursework, which is a possible confounding variable.
Worked example: Which conclusion is justified?
A school district randomly selects 10 of its 40 elementary schools. Within each chosen school, half of the fourth-grade classrooms are randomly assigned to use a new math curriculum and the other half keep the old curriculum. At the end of the year, the new-curriculum classrooms have significantly higher mean test scores. Which conclusion is justified?
Solution. Both kinds of randomness are present. The schools were randomly selected from the district's elementary schools, and the curriculum was randomly assigned to classrooms. So it is reasonable to conclude that the new curriculum caused higher test scores for fourth-grade classrooms in this district's elementary schools. The conclusion should not be extended to other districts, which were never part of the population sampled.
Common mistake
A large sample does not substitute for either kind of randomness. An observational study of 100,000 people still can't show causation, and an experiment with 5,000 volunteers still can't be generalized to a population it wasn't selected from.
Tip
On free-response questions, answer scope of inference in two sentences: one about generalization (random selection? to which population?) and one about causation (random assignment?). State each in the context of the study.
Practice
A researcher randomly selects 400 adults from a city's voter registration list and asks whether they own a pet and how many times they visited a doctor last year. Pet owners reported significantly fewer doctor visits. Which conclusion is appropriate?
Seventy volunteers at a fitness center are randomly assigned to one of two stretching routines, and flexibility is measured after six weeks. Routine B produced significantly greater improvement. Which statement correctly describes the scope of inference?
Which feature of a study's design allows its results to be generalized to a larger population?
A random sample of 250 students at Lincoln High School finds that students who eat lunch off campus have a significantly lower mean attendance rate. A newspaper reports, "Eating lunch off campus lowers attendance at high schools across the state." What are the two problems with this headline?
Consider these four studies. In how many of them is a cause-and-effect conclusion justified (assuming the observed difference is statistically significant)?
- Study 1: A random sample of 1,000 drivers is asked about phone use and accident history.
- Study 2: 50 volunteer students are randomly assigned to take a quiz with or without background music.
- Study 3: A company randomly selects 100 employees and randomly assigns half to a standing desk.
- Study 4: A doctor compares recovery times of patients who chose surgery with those who chose physical therapy.
Enter a number. Fractions like 3/4 and sqrt(2) are OK.
A state agency randomly selects 300 of the state's 12,000 dairy cows on registered farms. Half of the selected cows are randomly assigned to a new feed and half to the standard feed. Cows on the new feed produce significantly more milk. Which conclusion is best supported?