Data Collectionmedium
0:00.0
A researcher hypothesizes that a new tutoring method improves standardized test scores. She designs a study, recruits 150 participants, and administers a pre-test. Before analyzing data, she realizes she collected scores on 12 different test domains (reading, math, science, etc.). She had not pre-registered her analysis plan. She then tests whether the tutoring effect is significant in each of the 12 domains individually and reports results for all 12. She finds that 3 domains show statistically significant improvements (p < 0.05) and 9 domains show no significant effect. She concludes: 'The tutoring method is effective for reading, math, and science.' Which statement best identifies a risk to validity?
A researcher hypothesizes that a new tutoring method improves standardized test scores. She designs a study, recruits 150 participants, and administers a pre-test. Before analyzing data, she realizes she collected scores on 12 different test domains (reading, math, science, etc.). She had not pre-registered her analysis plan. She then tests whether the tutoring effect is significant in each of the 12 domains individually and reports results for all 12. She finds that 3 domains show statistically significant improvements (p < 0.05) and 9 domains show no significant effect. She concludes: 'The tutoring method is effective for reading, math, and science.' Which statement best identifies a risk to validity?