tag:blogger.com,1999:blog-2166457665608842386.post4263461941877773737..comments2021-09-18T07:11:18.542-05:00Comments on Daniel Simons: What effect size would you expect?Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-2166457665608842386.post-85119732750920844732013-03-15T11:46:41.356-05:002013-03-15T11:46:41.356-05:00Josh -- I don't think the thought experiment m...Josh -- I don't think the thought experiment makes any assumptions at all about the truth value of the initial study. Rather, it just asks what the odds of getting an effect size of a given magnitude/sign would be if the null hypothesis were true. The larger thought experiment asks whether a second study can be considered to have replicated the pattern of the first, whether or not the first result was true or a false positive. It isn't a conditional probability because I'm not making any assumptions about whether the initial result is real or a false positive. I'm just asking whether a second study replicates the pattern shown in the first one. We don't know the ground truth about the size of the actual effect in this thought experiment (just as we can only estimate it through experimental results in reality).<br /><br />In a real-world situation, we just don't know whether an initial result is a false positive or even an accurate estimate of an effect. And, no one study can tell us whether the original was a false positive. The broader goal, then, is to estimate the true underlying effect size. The best approach, without pre-existing knowledge of the ground truth is to conduct a meta-analysis across many studies. If the cumulative effect size approaches zero across many studies, then the original likely was a false positive, but we never know that for certain (we're estimating reality). <br /><br />My point in this thought experiment is to note that the sign of an effect in a replication does not provide strong confirmation of the original effect.Daniel J. Simonshttps://www.blogger.com/profile/02968898312917472467noreply@blogger.comtag:blogger.com,1999:blog-2166457665608842386.post-48071909209692982842013-03-15T10:03:32.846-05:002013-03-15T10:03:32.846-05:00This is an interesting thought experiment. I'm...This is an interesting thought experiment. I'm not sure it contains the right numbers. What you are looking at is what is the probability, given that the first experiment had a false positive, the second experiment of a given sample size will be in the same direction.<br /><br />I think that typically what we care about is the non-conditional probability: what is the joint probability that the first results was a false positive and the second one is the same direction? Answering that question involves a lot more guesswork (we need to know the prior probability of false positives, which we don't, and people's guestimates seem to run pretty much the whole spectrum). Joshhttps://www.blogger.com/profile/12072145030980202405noreply@blogger.com