tag:blogger.com,1999:blog-21664576656088423862024-02-20T11:51:04.470-06:00Daniel Simons <b> | </b><a href="http://dansimons.com">dansimons.com</a><b> | </b><a href="http://simonslab.com">simonslab.com</a><b> | </b><a href="http://invisiblegorilla.com">invisiblegorilla.com</a><b> | </b> <hr>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.comBlogger56125tag:blogger.com,1999:blog-2166457665608842386.post-27593230358137286762022-06-23T08:39:00.000-05:002022-06-23T08:39:00.139-05:00 Earliest proposal for a new registered report format?<p><span style="font-size: large;"><b>Earliest proposal for a new registered report format?</b></span></p><p>Back in March of 2012, about a year before we launched registered replication reports at <i>Perspectives on Psychological Science</i> and Chris Chambers and crew <a href="https://www.journals.elsevier.com/cortex/news/registered-reports-a-new-article-format-from-cortex">launched</a> registered reports at <i>Cortex</i>, Alex Holcombe and I had been discussing ways to increase the incentives for replication studies. The discussion happened on the now-defunct Google+ platform. Those discussions led to plans for a new journal which soon after morphed into the registered replication reports at <i>Perspectives</i>. <br /></p><p>Yesterday, Alex uncovered what I think is my first public post about the idea, dated 14 March 2012. I've copied it in full below, editing only to remove now broken Google+ tags/links. </p><p>The post describes the value of what are now called "registered reports," detailing what form such articles would take, how they would be reviewed, and how the new format would improve the publishing process and incentives (at least for replication studies). Although the post was specifically focused on replication studies, most of the elements described in the post are now a standard part of the <a href="https://osf.io/rr/">registered report model</a> today.<br /></p><p>If you know of earlier proposals for adoption of registered reports as an article type, let me
know. It would be nice to know the full history of this format.<br /></p><p><br /></p><p></p><span><a name='more'></a></span><p><i>Originally posted to Google+ on 14 March 2012</i><br /></p><p><b>Outlets to publicize replication attempts</b><br /><br />Yesterday, <span><span>Alex Holcombe</span></span> started a thread discussing how best to encourage people to post replication attempts to <a data-saferedirecturl="https://www.google.com/url?q=http://psychfiledrawer.org&source=gmail&ust=1656054378293000&usg=AOvVaw3tbPiReTkiD6eRTEMyrVwJ" href="http://psychfiledrawer.org" target="_blank">psychfiledrawer.org</a>.
PsychFileDrawer.org is great site where you can post the details of
your successful and failed replication attempts of other studies. Alex's
question: If you had a little bit of money to encourage people to post,
how would you use it. Lots of interesting comments, there. <br /><br />After
some discussions of this and related issues with faculty and students
at Illinois, I've been wondering whether a new type of journal might be
successful. Below is an idea I posted (in slightly modified form) to Alex Holcombe's thread. What do you think? <br />------<br />I
know that most "null results" journals generally haven't been
successful. I wonder, though, whether an open access journal that
published both replication successes and failures might be. (Note:
Journals like PLoS go part of the way toward what I'm thinking.)<br /><br />Here's
my idea: Researchers submit the intro (extremely short -- no need to
review the literature) and method section for review, along with an
analysis plan that specifies details like the sample size, assumed
effect size, methods for eliminating outliers, a-priori power, etc. They
would not submit results. Only the intro, method, and analysis plan
would undergo peer review. Once the replication plan passed peer review,
the results would be published regardless of outcome. But, in order to
be published, the method and results would have to follow the
pre-approved plan exactly. <br /><br />Here are the benefits of this approach:<br /><br />1)
It increase the incentives for people to do replications—it could
result in an actual journal article, so it might be worthwhile as an
undergraduate thesis project or a grad student project. <br /><br />2) It
would encounter less resistance from the authors of the original
publication during the review process (a major problem when publishing
failed replications) -- their goal would be to verify that the methods
are acceptable to them given what they had done originally. If they
thought the method and analysis plan to be acceptable in advance, they
wouldn't have grounds to object if the result didn't support them (and
they should be excited if it supports them).<br /><br />3) It would
encourage direct replication rather than conceptual replications that
differ in both method and analysis from the original.<br /><br />4) It would
lead to more details from original authors about their methods during
the review process, avoiding the inevitable complains that surface when
trying to publish a failed replication in a traditional journal (the
incentive for the original authors in that case is to highlight any
method difference and claim it to be <i>the reason</i> for the failure). <br /><br />5)
The submission and review process could be relatively quick as well
given that there wouldn't be lengthy reviews, and the original authors
could always be reviewers. I would favor all reviews being signed as
well so that there can be no objections later.<br /><br />6) It's possible
that such a journal would publish a lot of replication attempts for the
same paper, but that's okay -- better cumulative effect size estimate
that way. <br /><br />7) The end result could be posted to sites like
PsychFileDrawer as well, making meta-analysis of the size of an effect
possible (and more accurate).<br /><br />What do you think? Any academic society publishers out there think that this might be a viable model.</p>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-33938132754169040902021-09-19T09:44:00.002-05:002021-09-19T09:44:24.871-05:00Happy to be wrong (Covid update)<p>Earlier this semester, I thought we likely had out-of-control spread
on campus from people with breakthrough cases that were going
undetected.<br /></p><p>It looks like I was wrong.</p><p>The University of Illinois decided not to do surveillance testing this fall semester, instead focusing on regular testing only of unvaccinated people and selective testing of vaccinated people if they were in a place with an outbreak. (I still think that's a mistake - see below). The campus also returned almost all of its students—including a record-size first-year class—and encouraged attendance at several mass gathering events including football games. I expected this combination to be disastrous. And, at first, it looked like I was right.<br /></p><p>In the first weeks of the semester, we saw a substantial spike in cases. Despite daily testing numbers that were just a fraction of those from last year, we saw new cases at a rate comparable to the February surge and daily positivity rates as high as during parts of the fall 2020 surge. </p><p>Over the past week or so, though, <a href="https://go.illinois.edu/COVIDTestingData">the daily numbers</a> of cases have dropped dramatically—we're currently down to an average of under 7 detected cases per day. What we don't know, because we're not testing everyone, is whether those 7/day constitute all of the cases or if we would detect many more than that if we were testing everyone. That is, would we still see only 7 cases/day if we were conducting 10,000 to 20,000 tests/day necessary for surveillance rather than the 2,000 to 5,000 per day we're currently conducting. Or would the number of cases scale proportionally with the number of tests. </p><p><b>Aside</b>: <i>We wouldn't even need to test everyone to know—we could randomly select a few thousand vaccinated people to test each week and that would give us an estimate. As far as I know, were not doing that, so we remain blind to what's actually happening. I don't know why we're not doing that sort of testing (other than cost).</i> </p><p> Here are some possible reasons things might be going better than I had expected they would:<br /></p><ul style="text-align: left;"><li>Maybe high vaccination rates coupled with masking requirements were enough to eliminate spread on campus.<br /></li><li>Maybe spread from breakthrough cases to people who also are vaccinated is even less likely than thought, so undetected breakthrough cases aren't spreading Covid further (given high vaccination rates on campus).<br /></li><li>Maybe those people who are most likely to engage in riskier behavior—the ones who were most responsible for the outbreaks in the fall/spring—have extra immunity because they both had covid already and have been vaccinated. So, even if they're taking risks, they aren't getting Covid and spreading it. And, everyone else is doing what they were all of last year, taking appropriate precautions to avoid getting Covid or giving it to others.<br /></li><li>Maybe breakthrough cases acquired from another breakthrough case are more likely to be asymptomatic, in which case we could have spread on campus but it wouldn't be detected because those students wouldn't seek out a test. (worst case that could be detected via the random sampling strategy I noted above.).<br /></li></ul><p>Regardless of the reasons, it's great that case numbers are down, and I hope it means that spread on campus is under control. </p><p>That said, the pandemic is far from over. Illinois Region 6, the set of counties that include Champaign, is hovering around 8% positivity (not great). We still have many county residents hospitalized, and we've seen increased deaths over the past month. Southern Illinois has zero open ICU beds for a 20-county region. Other Midwestern states (e.g., Ohio) are seeing a huge surge, and neighboring states are not taking the same precautions that Illinois is (no indoor mask mandates). </p><p>For those reasons, I'm not thrilled with unrestricted attendance at sporting events (especially the indoor ones). If people who aren't vaccinated attend those events, including those traveling from outside the area, we could see spread from them to the campus or vice versa. I'd like to see UIUC require negative tests or proof of vaccination to attend campus events (just like the United Center will). </p><p>It's a relief and great news that campus appears to be doing relatively well right now. I'm glad that my concerns about the fall semester appear to have been overly alarmist. It'd be wonderful to be proven wrong about that. But, that doesn't mean we shouldn't take precautions, especially when the campus and broader community interact. <br /></p>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.comtag:blogger.com,1999:blog-2166457665608842386.post-73171388069250987232021-08-31T08:46:00.004-05:002021-09-19T09:44:38.663-05:00Covid responses at Duke and UIUC<p>According to the Duke University <a href="https://coronavirus.duke.edu/covid-testing/">dashboard</a>, they had 349 positive cases among students last week. Duke has about 15000 students (about 2/3 are grad students) and a <a href="https://www.washingtonpost.com/education/2021/08/30/liberty-university-covid-virtual-quarantine/">98% vaccination rate</a>. You might recall from last year that Duke implemented a robust testing process last year and had far fewer cases than the University of Illinois did. They appear to have continued their approach from last year of testing all of their students. Although their dashboard doesn't break down cases based on vaccination status, most of these positive cases are likely to be breakthrough cases. In other words, they are testing everyone so that they will have some idea of how bad things are on campus. That has led them to implement mitigation (according to Washington Post <a href="https://www.washingtonpost.com/education/2021/08/30/liberty-university-covid-virtual-quarantine/">article</a>): Indoor and outdoor masking, allowing faculty to shift to virtual teaching for two weeks, suspending indoor dining, limits on student activities. </p><p>Duke has a higher vaccination rate than the University of Illinois, and when they tested everyone last week, they found 349 positive cases. When the University of Illinois tested primarily the unvaccinated students, we found 165 cases. Yet, unlike Duke, we don't know how many people are actually infected because we're testing only a small subset of our campus population. <br /></p><p>Duke is doing surveillance testing to monitoring the covid situation on its campus and they're taking steps to detect cases among the entire campus population and to stop spread. The University of Illinois is not, even though we could be. We have made no effort to detect cases among the vast majority of students who are vaccinated even though breakthrough cases can spread Covid. And, if we don't know when someone is infected, we can't isolate them to break the chain of transmission. If I had to guess based on our vaccination rate and positive tests among the unvaccinated students, I'd estimate that we have
500+ cases on campus right now and that we've detected only about a third of
them. And, those cases we haven't detected will be spreading Covid rapidly. <br /></p><p>The University of Illinois has regularly touted our "success" last year in preventing community spread (although the data on that are debatable), so they recognize how important that is. Yet, they went ahead and encouraged a full-capacity crowd at a football game with no meaningful mitigations in place. Many people who attend football games come from the surrounding communities that have low vaccination rates and high levels of spread. If preventing community outbreaks is important to the university, maybe that wasn't such a good idea...</p><p>Yesterday, Awais Vaid of the Champaign-Urbana Public Health District (who has worked closely with the university SHIELD team) told the <a href="https://www.news-gazette.com/coronavirus/as-crowds-return-so-does-call-for-testing/article_c79bce87-c493-59e1-941a-267c569b08fd.html">News-Gazette</a> that we'll know in a week whether having 41,000 fans at the game will result in more Covid cases for Champaign County, and he encouraged fans who attended to consider getting tested. Perhaps the possibility that the game would be a super spreader event is something that the university should have raised and addressed <i>before</i> the game. Instead, the university offered free tickets to faculty and staff in order to fill seats (thanks for the invite, but I noped as far away from that one as possible). </p><p>Unlike Duke, the University of Illinois appears to have adopted a "let's wait and see what happens" strategy, while simultaneously blinding ourselves so that we can't see clearly. We have the capability to test everyone, but the university has chosen not to. Hoping everything will go well isn't a strategy—Covid doesn't care what we hope will happen.</p>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.comtag:blogger.com,1999:blog-2166457665608842386.post-30473030232677534432021-08-30T11:31:00.001-05:002021-09-19T09:44:51.780-05:00Covid numbers - August 30, 2021<p>During the first week of classes at the University of Illinois, <a href="https://go.illinois.edu/COVIDTestingData">campus testing</a> identified <b>165</b> cases of Covid on a total of 22,296 tests. <br /></p><p>For comparison, in the first week of the spring semester (Jan 25 - 31), we had a total of <b>128</b> cases with 63,208 tests. We're finding more cases than at the comparable point last spring despite testing far fewer people. Whether that's due to Delta or to reduced restrictions and more
potential super spreader events (e.g., football games with unrestricted
attendance, massive new student orientations with lots of yelling, open
bars, etc.) is hard to know.</p><p>And, as noted in <a href="http://blog.dansimons.com/2021/08/covid-at-illinois-30-august-2021.html">my post</a> earlier today, we're missing the vast majority of breakthrough cases because we are not doing surveillance testing of vaccinated students. If the vaccines have 60% efficacy against infection and the exposure risks are comparable for vaccinated and unvaccinated students (an assumption that might not be justified), we could have up to 3x as many breakthrough cases that have gone undetected. That is, had we tested everyone on campus, we could well have found a much higher number of cases (perhaps hundreds). Any breakthrough cases that that we didn't detect (because we're not looking) potentially could infect others. Even if people with breakthrough cases do not spread covid as easily as unvaccinated people with Covid, that's still a lot infected people who come into close contact with many other people.<br /></p><p>We don't definitive evidence of vaccine efficacy against infection in this population, and we lack information from the university about the total numbers of people being tested, the number of each subgroup of people on campus, or the numbers of vaccinated and unvaccinated people in each group being tested. Without that information, we can only rely on guestimates about efficacy as well as infection rates. But, even if we ignore the possibility of large numbers of undetected breakthrough cases, the first week numbers aren't great - they look worse than at the start of the spring semester. If there are a sizable number of undetected breakthrough cases, as seems inevitable, we won't know how much spread we're seeing on campus and we won't be able to disrupt chains of transmission. I hope breakthrough cases aren't as infectious, but if they are even somewhat infectious, we're risking undetected exponential growth.<br /></p>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.comtag:blogger.com,1999:blog-2166457665608842386.post-67586682198016083692021-08-30T07:43:00.002-05:002021-08-30T07:43:58.615-05:00Covid at Illinois - 30 August 2021<h1 style="text-align: center;"></h1><p>During the 2020-2021 academic year, I posted <a href="http://www.dansimons.com/covid.html">regular updates</a> about Covid at the University of Illinois. I tracked the number of positive cases on campus, estimated the percentage of students infected, and computed various metrics of the infection that were more meaningful than the ones provided by the university (e.g., rolling averages of cases, cumulative cases by semester, cases by week, etc.). Those analyses were necessary because the campus dashboard featured meaningless statistics like 7-day <a href="http://www.dansimons.com/Covid/positivity.html">average positivity</a> and the all-time number of tests, neither or which provided useful information to gauge the extent of outbreaks. The <a href="https://go.illinois.edu/COVIDTestingData">campus dashboard</a> also gave no breakdown of the infection percentages, the numbers of students being tested each week, the compliance rate for testing, etc. </p><p>In addition to those daily updates, I also discussed many of the logistical problems that contributed to the infection of an estimated 20% of the undergrads on campus last year (4935 out of about 24000) as well as some of the misleading claims coming from the administration. In addition to the daily summaries, I posted a <a href="http://www.dansimons.com/Covid/fall2020summary.html">detailed summary</a> of the fall semester issues as well as a more in-depth analysis of how <a href="http://www.dansimons.com/Covid/testTiming.html">delayed testing</a> results likely contributed to the fall 2020 surge. </p><p></p><p>This year, changes in testing policies (not testing everyone) mean that I won't be able to estimate the infection rates on campus, and it will be much harder to evaluate whether or not we're in the midst of a large outbreak. It'll be more guesswork than analysis. I'll try to do that guesswork and will post updates here on <a href="http://blog.dansimons.com">my blog</a>. Occasionally I'll post analyses as well. But, there's not much benefit to the sort of daily tracking I did last semester because there won't be enough information available on the campus dashboard to do it. I fear that we're running blind this semester. Here's where things stand on campus and why we won't have the information we need. <br /></p><h3 style="text-align: left;">Vaccine "requirement"<br /></h3><p>This summer, the campus announced that they would be requiring vaccines. That sounded fantastic, but it turned out that the implementation had no teeth and it was actually more of a "nudge" than a requirement. If you weren't vaccinated — and you didn't need to give any reason why not — you had to continue testing. </p><p>Last week, Gov. Pritzger announced that all school and university faculty and staff in Illinois must be vaccinated. Shortly after that announcement, the university announced that they will now mandate vaccination for everyone (except those with a medical or religious exemption). I had thought that "requirement" and "mandate" are synonyms, but apparently they mean different things to our administration. I also have no idea how hard it is to get a religious exemption. In my view, you should be required to present a holy text from your faith showing that vaccines are not permitted...). In any case, it's great that we finally have an actual vaccine requirement/mandate coming into effect. </p><p></p><h3 style="text-align: left;">Changes to testing <br /></h3><p>Last year, the campus tested everyone twice weekly (allegedly - there was a lot of non-compliance). Over the summer, when cases were low and vaccines were available, the university developed their plan for the fall: They dramatically scaled back on testing by only requiring unvaccinated people to test. The number of testing sites was reduced to 4. The used vaccination as a get-out-of-testing incentive. Unfortunately, they didn't radically change their plans for Delta. They did increase the required testing frequency for unvaccinated undergraduates to every other day because modeling showed that without doing that, people would spread Delta before they knew they were positive. Delta becomes infectious faster than the earlier variants. </p><p>However, by electing not to test vaccinated people on campus, we won't detect breakthrough cases (unless they are symptomatic and elect to get tested, by which point it's too late to stop spread). We also won't be able to break the
chain of transmission by isolating infected students and quarantining or
testing their contacts. We're going to be largely blind to the outbreaks on campus, and they could easily get out of control. The campus has the capacity to test everyone, but they chose not to use it. It's a potentially disastrous decision. Here's why.<br /></p><h4 style="text-align: left;">The consequences of testing only the unvaccinated <br /></h4><p>There are far more vaccinated students than unvaccinated ones on campus. About 88% of students are vaccinated, and that soon will be closer to 100% now that we have an actual mandate. If we estimate vaccine efficacy against <i>infection</i> to be approximately 60%, that means there will be <b>nearly 3x as many</b> breakthrough cases than there are cases among the unvaccinated!</p><p>To see why, imagine that we test all of the unvaccinated students on campus on a single day and that 1% of them test positive. If we assume that the unvaccinated and vaccinated students had the same exposure to covid via close contacts, that means 0.4% of the vaccinated students would have tested positive had we tested them too (60% efficacy against infection). With about 35,000 undergrads on campus, about 30,800 of them are currently vaccinated (88%) and the remaining 4200 are unvaccinated. If 1% of the 4200 tested positive, that would yield 42 cases. If 0.4% of 30,800 vaccinated students tested positive, that'd be 123 cases. So, we should expect about 3x as many cases among the vaccinated population as among the unvaccinated population. There are a lot more vaccinated than unvaccinated students (which is great, of course). </p><p></p><p>In the extreme, if 100% of students were vaccinated, and we're only testing unvaccinated students, we could have a massive outbreak on campus via breakthrough cases and we'd know nothing about it (until people start showing symptoms and seek testing or treatment. But by the time they're showing symptoms, they've been infectious for a while).</p><p>Last year, the University was (somewhat) able to contain outbreaks by quickly identifying positive cases—often before they were infectious—and then isolating them. We saw the surge, implemented mitigations, and brought numbers down at least somewhat. This semester, up to 3/4 of the initial cases could go undetected. Unlike last year, we'll be blind to spread from those cases until it's too late to stop it. </p><p>Vaccines are highly effective at preventing severe illness, hospitalization, and death. And, students might be at lower risk than older people as well. But, the vaccines aren't airtight protection, and even with the campus's indoor mask requirement (again, that's great!), there are many opportunities to spread Covid both on and off campus (see the massive unmasked crowds at the football game, at bars, etc.). If the University allows unmitigated spread on campus, it likely will reach the broader community this semester. </p><p>As was the case last year, the University of Illinois is in much better shape than many other universities. First, Illinois is in better shape than many other states. We have a governor who is implementing good public health policies (mandating vaccines for all schools and requiring masking indoors) and a <a href="https://www.c-uphd.org/">local health district</a> that is proactive and communicates well. We have a campus mask mandate and are implementing a vaccine mandate. Due to the amazing test that researchers developed here and the testing capacity that was built up over the past year, we have the potential to know a lot more about how bad the outbreaks on campus are and to end them before they spread too far. We have the capacity to limit the spread and consequences of Delta, but we're not doing everything we need to be doing.</p><p>Even if the University prefers not to test everyone due to costly logistics, there are approaches we could take that would help at least determine how much spread we have on campus. For example, the campus could test randomly selected, somewhat large subsets of vaccinated students each week. If those tests revealed a high rate of positives, we might need to adopt more extensive testing and restrictions. The dashboard could also separate results for vaccinated and unvaccinated people. That would show whether we're seeing a spike in vaccinated people seeking tests (a sign that we have spread). Without some form of systematic surveillance testing, though, we won't know how bad things are until people start getting sick. </p>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.comChampaign, IL, USA40.1164204 -88.243382911.806186563821157 -123.3996329 68.426654236178848 -53.0871329tag:blogger.com,1999:blog-2166457665608842386.post-90042520704254139192020-12-10T11:46:00.005-06:002020-12-21T09:25:46.836-06:00<h1 style="text-align: center;"><b> Analyses of Covid trends <br />at the University of Illinois</b></h1><p style="text-align: left;">Since the university reopened in Fall 2020, I have closely monitored the numbers of Covid cases on campus and the reasons for spread. My analyses are intended to be a reliable source of publicly available information. The University has not always been transparent about the situation on campus, so I have provided additional estimates of the information that was not available. I also provide daily updates about hospitalizations and spread in the community surrounding campus. <br /></p><p style="text-align: left;">Here are links to the various summaries and write-ups I've been tweeting about and posting publicly on Facebook. <br /></p><ul style="text-align: left;"><li>I post almost daily updates at <a href="http://www.dansimons.com/covid.html">http://www.dansimons.com/covid.html</a>. Monitor that page for the most recent daily case numbers, 7-day averages, hospitalizations/deaths in the community, etc. The page includes daily updated graphs and explanations of them.<br /></li></ul><div><ul style="text-align: left;"><li>My summary of the entire fall semester is available at <a href="http://www.dansimons.com/Covid/fall2020summary.html">http://www.dansimons.com/Covid/fall2020summary.html</a></li><ul><li>That summary includes a discussion of the <a href="modelinhttp://www.dansimons.com/Covid/fall2020summary.html#models">modeling</a> that was used to justify reopening as well as how it was both right and badly wrong. </li><li>It also discusses how the University placed <a href="modelinhttp://www.dansimons.com/Covid/fall2020summary.html#blame">blame</a> for the initial surge in cases on a handful of non-compliant students without taking any responsibility for logistical failures and inaccurate initial assumptions about compliance.</li></ul></ul><ul style="text-align: left;"><li>My analyses of delays in test processing and how they contributed to the spread of Covid on campus is at <a href="http://www.dansimons.com/Covid/testTiming.html">http://www.dansimons.com/Covid/testTiming.html</a></li></ul><ul style="text-align: left;"><li>My essay on how to interpret positivity ratios correctly, including a discussion of why positivity is the wrong metric to use when you are testing everyone in your population of interest repeatedly (as was the case at the University of Illinois this fall) is at <a href="http://www.dansimons.com/Covid/positivity.html">http://www.dansimons.com/Covid/positivity.html</a><br /></li></ul><p style="text-align: left;"><br /></p><p style="text-align: left;">I have turned off comments for this post. If you would like to comment, please do so in response to my regular tweets (@profsimons) on this topic. This post is a placeholder so that the links are findable.<br /></p></div>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0Champaign, IL, USA40.1164204 -88.243382911.806186563821157 -123.3996329 68.426654236178848 -53.0871329tag:blogger.com,1999:blog-2166457665608842386.post-49352358888625535952017-05-15T17:17:00.004-05:002020-12-21T08:06:28.927-06:00A new journal at APS: AMPPS<span style="font-size: large;">I am thrilled to announce the launch of the newest journal published by the Association for Psychological Science (APS):</span><br />
<span style="font-size: large;"><br /></span>
<div style="text-align: center;">
<span style="color: blue; font-size: large;"><a href="https://www.psychologicalscience.org/publications/ampps"><i>Advances in Methods and Practices in Psychological Science</i> (<i>AMPPS</i>)</a></span></div>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">I will been named the founding editor, and I have assembled what I think will be a terrific editorial team. </span><span style="font-size: large;">We are now <a href="https://www.psychologicalscience.org/publications/ampps">taking submissions</a>, with the first issue slated for publication in early 2018.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The journal's primary mission is to communicate advances in research methods and practices to the broad membership of APS and beyond. We hope to bridge subfields of psychology, bringing advances from within an area to the rest of the field. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><i>AMPPS</i> will publish articles on research practices, tutorials to help researchers develop new research skills, and empirical papers that illustrate innovative methodological approaches. It will be the new home for the Registered Replication Reports that previously were published at the APS journal Perspectives on Psychological Science </span><span style="font-size: large;">(my co-editors for those papers, Alex Holcombe and Jennifer Tackett, have joined the new editorial team</span><span style="font-size: large;">). </span><span style="font-size: large;">AMPPS will also publish registered reports, adversarial collaborations, multi-lab consortium studies, simulations and re-analyses of existing data, meta-science papers, commentaries, and much more.</span><br />
<br />
<span style="font-size: large;">You can read more about the journal's mission, see the editorial team, and read the submission guidelines on the journal's <a href="https://www.psychologicalscience.org/publications/ampps">website</a>.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">APS also issued a <a href="https://www.psychologicalscience.org/observer/aps-journal-on-research-practices-and-methods-launches#.WRooRBMrIo_">press release</a> about the new journal last week.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Please help spread the word! If you tweet or post about it, please use the hashtag #APS_AMPPS</span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-6373765223958240992015-12-10T11:01:00.006-06:002020-12-21T08:06:41.702-06:00Visual effort and inattentional deafness<div style="text-align: center;">
<span style="font-size: x-large;">Visual Effort and Inattentional Deafness</span></div>
<br />
<span style="font-size: large;">Earlier this week I was asked for my thoughts on a new <i>Journal of Neuroscience</i> paper: </span><br />
<blockquote class="tr_bq">
Molly, K., Griffiths, T. D., Chait, M., & Lavie, N. (2015). Inattentional deafness: Visual load leads to time-specific suppression of auditory evoked responses. <i>Journal of Neuroscience</i>, <i>35</i>, 16046-16054.doi: 10.1523/JNEUROSCI.2931-15.2015</blockquote>
<div>
<span style="font-size: large;">In part due to a widely circulated press release, the paper has garnered a ton of media coverage, with headlines like:</span></div>
<div>
<blockquote>
<a href="http://www.techtimes.com/articles/114557/20151209/focusing-on-a-task-may-leave-you-temporarily-deaf-study.htm">Focusing On A Task May Leave You Temporarily Deaf: Study</a><br /><br /><a href="http://www.parentherald.com/articles/13822/20151209/did-you-know-watching-something-makes-you-temporarily-deaf.htm">Did You Know Watching Something Makes You Temporarily Deaf</a>?<br /><br /><a href="http://abcnews.go.com/GMA/video/study-explains-screen-time-inattentional-deafness-35665681">Study Explains How Screen Time Causes 'Inattentional Deafness'</a></blockquote>
<br />
<span style="font-size: large;">The main contribution of the paper was a link between activation in auditory cortex and the behavioral finding of reduced detection of a sound (a brief tone) when performing a more difficult visual task. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This brain-behavior link, not the behavioral result, is the new contribution from this paper. Yet, </span><span style="font-size: large;">almost all of the media coverage has focused on the behavioral result which isn't particularly novel. That's unsurprising given </span><span style="font-size: large;">that most of the stories just followed the lede of the press release, which was titled:</span><br />
<blockquote class="tr_bq">
"Why focusing on a visual task will make us deaf to our surroundings: Concentrating attention on a visual task can render you momentarily 'deaf' to sounds at normal levels, reports a new UCL study funded by the Wellcome Trust"</blockquote>
<br />
<span style="font-size: large;">Here are a few points about this paper that have largely been lost or ignored in the media frenzy (and the press release):</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">1. The study did not show that people were "deaf to their surroundings." In the study (Experiment 2), people performed an easy or hard visual task while also trying to detect a tone that occurred on half of the trials. When performing the easy visual task, they reported the tone accurately on 92% of the trials. When performing the harder visual task, they reported it accurately on 88% of trials. </span><span style="font-size: large;">The key behavioral effect was a <b>4% reduction in accuracy</b> on the secondary, auditory task when the primary visual task was harder.</span><span style="font-size: large;"> </span><span style="font-size: large;">In other words, </span><span style="font-size: large;">people correctly reported the tone on the vast majority of trials even with the hard visual task</span><span style="font-size: large;">. That's not deafness. It's excellent performance of a secondary task with just a slight reduction when the primary task is harder. </span><br />
<span style="font-size: large;"><br /></span>
<u>Aside</u>: much of that small effect on accuracy could be due to a difference in response bias between the conditions (Beta of 3.2 compared to 1.3, a difference reported as p = 0.07 with an underpowered study of only 11 subjects).<br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">2. The behavioral effect of visual load on auditory performance is not original to this paper. In fact, it has been <a href="http://www.ncbi.nlm.nih.gov/pubmed/25287617">reported</a> by the same lab.</span><br />
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">3. A number of other studies have demonstrated costs to detection in one sensory modality when focusing attention on another modality. This paper is not the first to show such a cross-modal effect. See, for example, <a href="http://www.ncbi.nlm.nih.gov/pubmed/17546744">here</a>, <a href="http://www.sciencedirect.com/science/article/pii/S1053810015300581">here</a>, <a href="http://www.ncbi.nlm.nih.gov/pubmed/16255124">here</a>, <a href="http://hfs.sagepub.com/content/56/4/631">here</a>, <a href="http://www.ncbi.nlm.nih.gov/pubmed/18686705">here</a> (none of which were cited in the paper). Many other studies have shown that increasing primary task difficulty decreases secondary task performance. Again, the behavioral result touted in the media is not new, something the press release acknowledges in passing.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">4. The study doesn't actually involve</span><span style="font-size: large;"> <i>inattentional deafness</i>; the term is misused. Inattentional deafness or blindness refers to a failure to notice an otherwise obvious but unexpected stimulus when focusing attention on something else. The "unexpected" part is key to ensuring that the critical stimulus actually is unattended (the justification for claiming the failure is due to <i>inattention</i>); people can't allocate attention to something that they don't know will be there. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">In this study, tone detection was a secondary task. People were asked to focus mostly on the visual task, but they also were asked to report whether or not a tone occurred. In other words, people were actively trying to detect the tone and they knew it would occur. That's not inattentional deafness. It's just a reduction in detection for an attended stimulus when a primary task is more demanding. And, as I noted above, it's not really a demonstration of deafness either given participants were really good at detecting the tone in both conditions (they were just slightly worse when performing a harder visual task). </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Note that the same lab previously published an <a href="http://link.springer.com/article/10.3758/s13414-011-0144-4">paper</a> that actually did show an effect of visual load on inattentional deafness.</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<span style="font-size: large;"><b>Conclusion</b>: </span><span style="font-size: large;">There's nothing fundamentally wrong with this paper, at least that I can see (I'm not an expert on neuroimaging, though). The link between the behavioral results and brain imaging results is potentially interesting. I would have preferred a larger sample size and ideally measuring the link between brain and behavior in the same participants performing tasks with the same demands, but those issues aren't show stoppers.</span><span style="font-size: large;"> I can see why it is of interest to specialists (like me). That said, I'm not sure that it makes a contribution of broad interest to the public, and the novelty and importance of the behavioral result has been overplayed.</span></div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-81967906918891557012015-11-30T09:57:00.002-06:002020-12-21T08:06:50.657-06:00HI-BAR: A gold standard brain training study?<span style="font-family: "arial" , "helvetica" , sans-serif;"><span id="docs-internal-guid-8731cccf-499a-1fbd-a408-e3773074427c"><span style="font-family: inherit; font-size: large;"><br /></span></span>
</span><br />
<h2 style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<div style="text-align: center;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-large; line-height: 1.38; white-space: pre-wrap;">A gold-standard brain training study? </span></div>
<span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><div style="text-align: center;">
<span style="line-height: 1.38;">Not without some alchemy</span></div>
</span></span></h2>
<br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">A <a href="http://blog.dansimons.com/2013/09/hi-bar-had-i-been-reviewer.html">HI-BAR</a> (Had I Been A Reviewer) of: </span><br />
<div>
<blockquote class="tr_bq">
<span style="font-family: "arial" , "helvetica" , sans-serif;">Corbett, A., Owen, A., Hampshire, A., Grahn, J., Stenton, R., Dajani, S., Burns, A., Howard, R., Williams, N., Williams, G., & Ballard, C. (2015). The effect of an online cognitive training package in healthy older adults: An online randomized controlled trial. JAMDA, 16(11), 990-997.</span></blockquote>
<br /><b><br /></b></div>
<div>
<b>Edit 12-3-15: </b>The planned sample was ~1 order of magnitude larger than the actual one, not 2. (HT Matthew Hutson in the comments)</div>
<div>
<br /></div>
<div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">A recent large-scale brain training study, <a href="http://www.jamda.com/article/S1525-8610(15)00435-1/fulltext">published</a> in the Journal of the American Medical Directors Association (JAMDA), has garnered a lot of attention. A <a href="http://www.kcl.ac.uk/ioppn/news/records/2015/november/Brain-training-improves-memory-and-performance-of-everyday-tasks-in-older-people.aspx">press release</a> was picked up by major media outlets, and a blog post by Tom Stafford on the popular <a href="http://mindhacks.com/2015/11/05/a-gold-standard-study-on-brain-training/">Mind Hacks blog</a> called it “a gold-standard study on brain training” and noted that “this kind of research is what ‘brain training’ needs.”*</span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Tom applied the label “gold standard” because of the study’s design: It was a large, randomized, controlled trial with an active control group and blinding to condition assignment. From the gold-standard monicker, though, people might infer that the research methods and results provide solid evidence for brain training benefits. They do not. <br /><br />Tom's post identified several limitations of the study, such as differential attrition across conditions and the use of a self-report primary outcome measure. Below I discuss why these and other analysis and reporting problems undermine the claims of brain training benefits. </span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<br />
<div>
<div style="text-align: center;">
<b><span style="color: blue;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: medium;"><span style="background-color: white; font-size: large; vertical-align: baseline; white-space: pre-wrap;">Problems that undermine interpretability of the study</span></span></span></span></b></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; color: black; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; color: black; font-size: large; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b><span style="color: #38761d;">Differential Attrition</span> </b></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">The analysis was based on the 6-month testing point, but the study was <b>missing data from about 70% of the participants</b> due to attrition. To address this problem, the authors carried forward data from the final completed testing session for each participant and treated it as if it were from the 6-month point. Critically, the control group had substantially greater attrition than the intervention groups—more of their scores were carried forward from earlier points in the intervention.</span><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<div style="line-height: 1.38;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">For the control group, only 27% of the data for the primary outcome and 17% of the data for the secondary outcomes came from participants who actually completed their testing at 6 months. For the Reasoning group, those numbers were 42% and 40%. For the General Cognition group, they were 40% and 30%. <br /><br />The extent of the differential attrition and rates of carrying forward results from earlier sessions were only discoverable by inspecting the Consort diagram. This analysis choice and its implications were not fully discussed, and the paper did not report analyses of participants with comparable durations of training. This analysis approach introduces a major confound that could entirely account for any differential benefits.</span></div>
<div style="line-height: 1.38;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><span style="background-color: white; color: #38761d; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b style="background-color: white; font-style: normal; font-variant: normal; line-height: 1.38; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Unclear sample sizes and means</b></span></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Tables 3 and 4 list different control group means next to each training condition. There was only one control group, so it is unclear why the <b>critical baseline means differed for the two training interventions</b>. Without knowing why these means differed (they shouldn't have), the differential improvements in the training groups are uninterpretable.<br /><br />The Ns listed in the tables also are inconsistent with the information provided in the Consort diagram. In a few cases, the Tables list a larger N than the consort diagram, meaning that there were <b>more subjects in the analysis than in the study</b>.<br /><br />I emailed the corresponding author (on Nov. 10 and Nov. 23) to ask about the each of these issues, but I received no response. I also emailed the second author. His assistant noted that the corresponding author's team was "was responsible for that part of the study" and said the second author "can be of no help with this." I’m hoping this post will prompt an explanation for the values in th</span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">e tables.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large; line-height: 1.38;">For me, those reporting and analysis issues are show stoppers, but the paper has other issues.</span></div>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: large;"><br /></span></span></div>
<div>
<div style="text-align: center;">
<b style="color: blue; font-family: Arial, Helvetica, sans-serif; font-size: x-large;">Other issues</b></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="background-color: white; color: #38761d; font-family: "arial" , "helvetica" , sans-serif; font-size: large; font-weight: 700; line-height: 1.38; white-space: pre-wrap;">Limitations of the pre-registration</span><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">The study was pre-registered, meaning that the recruiting, testing methods, and analysis plans were specified in advance. Such pre-registrations are required for clinical trials, but they have been relatively uncommon in the brain training literature. Have a pre-registered plan is ideal because it eliminates much of the flexibility that otherwise can undermine the interpretability of findings. The use of pre-registration is laudable. But, the <b>registration was underspecified and the reported study deviated from it</b> in important ways. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br />For example, the protocol called for 75,000 - 100,000 participants, but the reported study recruited fewer than 7000. That’s still a great sample, but it’s <strike>2 orders</strike> <u>an order</u> of magnitude smaller than the planned sample. Are there more studies resulting from this larger sample that just aren’t mentioned in the pre-registration? </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">The study also called for a year of testing, but it had to be cut short at 6 months and more than 2/3 of the participants did not undergo even 6 months of training. The pre-registration did not include analysis scripts, and the data from the study do not appear to have been posted publicly.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">The pre-registered hypotheses predicted greater improvements in the reasoning training group than the general cognition group and it predicted that the general cognition group would not outperform the control group. <b>The paper reports no tests of this predicted difference</b>.</span><br />
<span style="background-color: white; color: black; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #38761d; font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><span style="background-color: white; font-weight: 700; line-height: 1.38; white-space: pre-wrap;"><br /></span></span></span>
<span style="background-color: white; color: black; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #38761d; font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><span style="background-color: white; font-weight: 700; line-height: 1.38; white-space: pre-wrap;">Underreporting for the primary measure (IADL)</span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">The primary outcome measure consisted of self-reports of performance in daily activities (known as the Instrumental Activities of Daily Living or IADL). As Tom's post noted, such <b>self-reports are subject to demand characteristics</b> — people expect to do better following training, so they report having done better. The study did not test for different expectations across the training and control groups, so the benefits could be due to such demands or to a differential placebo effect (e.g., the control group might have found the study less worthwhile). </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br />The reported benefits for IADLs were small, and the <b>data provide little evidence for any benefit of training</b>. The study reported statistically significant benefits for both training groups relative to the control group, but statistical significance is not the same as evidence. With samples this large, we <a href="http://daniellakens.blogspot.com/2014/05/the-probability-of-p-values-as-function.html">should expect</a> a substantially lower p-value than .05 when an effect actually is present in the population. If the Ns and means reported in the table were consistent with the method description, it might be possible to compute a Bayes Factor for these analyses. My bet is that the difference between the training groups and the control group would provide weak evidence at best for a meaningful training benefit (relative to the null).</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">The paper provides <b>no information about baseline scores</b> on the primary outcome measure (IADL). Although the analyses control for baseline scores, training papers must provide the pre-test scores and post-test scores. Without doing so, it is impossible to evaluate whether apparent training benefits resulted in part from baseline differences.<br /><br />The paper also states that “Data from interim time points also show significant benefit to IADL at 3 months, particularly in the GCT group, although this difference was not significant.” I take this to mean both training groups outperformed the control group at 3 months, but they did not differ significantly from each other. <b>No statistical evidence is provided in support of this claim</b>.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<span style="background-color: white; color: #38761d; font-family: "arial" , "helvetica" , sans-serif; font-size: large; font-weight: 700; line-height: 1.38; white-space: pre-wrap;">Limited evidence from the secondary measures</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<div style="line-height: 1.38;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Only one of the secondary cognitive outcome measures (a reasoning measure) showed a training benefit. The paper refers to it as “the key secondary measure,” but that designation does not appear in the pre-registration (http://www.isrctn.com/ISRCTN72895114). Moreover, the pre-registration predicts better performance for reasoning training than general cognition training or the control group, but the paper found improvements for both interventions. A few other measures showed significant effects, but given the large sample sizes, the high p-values might well be more consistent <b>with the absence of a training benefit</b> than the presence of one.</span></div>
<div style="line-height: 1.38;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Despite providing no statistical evidence of differential benefits for Reasoning training and General Cognition training, the paper claims that “Taken together, these findings indicate that the ReaCT package confers a more generalized cognitive benefit than the GCT at 6 months. That claim appears to come from finding no effect on a digit secondary task in the Reasoning group and a decline in the General cognition group. However, a <b>difference in significance is not a significant difference</b>.</span></div>
<div style="line-height: 1.38;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Almost all of the measures showed declining performance from the pre-test to the post-test. That is, participants were not getting better. They just declined less than the control participants. It is unclear why we should see such a pattern of <b>declining performance over a short time window with relatively young participants</b>. Although cognitive performance does decline with age, presumably those declines should be minimal over 1-6 months, and they should be swamped by the benefits of taking the test twice. One explanation might be differential attrition -- those subjects who did worse initially were more likely to drop out early. </span></div>
<div style="line-height: 1.38;">
<span style="background-color: white; color: black; font-family: "arial"; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="line-height: normal;"></span><span style="background-color: white; color: black; font-family: "arial"; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span><span style="line-height: normal;"></span><span style="background-color: white; color: black; font-family: "arial"; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">* Thanks to Tom Stafford for emailing a copy of the paper. The journal is obscure enough that the University of Illinois library did not have access to it.</span></div>
</div>
</div>
</div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-2485618190670815692014-12-15T10:42:00.002-06:002020-12-21T08:06:58.438-06:00Response from Ballesteros et al to my HI-BAR<b>Update 12-15-14</b>: I used <strike>strikethru</strike> to correct a couple of the F test notes below. The crossed out ones were fine.<br />
<span style="font-size: large;"><br /></span>
<b>Update 5-26-15</b>: Frontiers has published a <a href="http://goo.gl/nqk73g">correction</a> from Ballesteros et al that acknowledges the overlap among their papers. It doesn't admit the inappropriateness of that overlap. It mostly echoes their response below, but does not address my questions about that response.<br />
<br />
<br />
<span style="font-size: large;">In late November, I <a href="http://blog.dansimons.com/2014/11/hi-bar-more-benefits-of-lumosity.html">posted</a> a <a href="http://blog.dansimons.com/2013/09/hi-bar-had-i-been-reviewer.html">HI-BAR</a> review of a paper by <a href="http://journal.frontiersin.org/Journal/10.3389/fnagi.2014.00277/abstract">Ballesteros et al (2014)</a> that appeared in <i>Frontiers</i>. In addition to a number of other issues I discussed there and in an <a href="http://blog.dansimons.com/2014/04/hi-bar-benefits-of-lumosity-training.html">earlier review</a> of another paper, I raised the concern that the paper republished the training data and some outcome measures from an <a href="http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0092269#pone-0092269-g002">earlier paper</a> in <i>PLoS One</i> without adequately conveying that those data had been published previously. I also noted this concern in a few <a href="http://journal.frontiersin.org/Journal/10.3389/fnagi.2014.00277/abstract">comments</a> on the <i>Frontiers</i> website. On my original HI-BAR post, I asked the original authors for clarification about these issues. </span><br />
<span style="font-size: large;"><br /></span>
<b><span style="font-size: large;">I have now received their response as a pdf file. You can read it <a href="https://drive.google.com/file/d/0B8SdDrkCc9RiV3FZNXZWN2ZGaGc/view?usp=sharing">here</a>. </span></b><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Below I quote some sections from their reply and comment on them. I encourage you to read the full reply from the original authors as I am addressing only a subset of their comments below. For example, their reply explains the different author lists on the two papers in a satisfactory way, I think. Again, I would be happy to post any responses from the original authors to these comments. As you will see, I don't think this reply addresses the fairly major concerns about duplication of methods and results (among other issues).</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Quotes from the authors' reply are indented and italicized, with my responses following each.</span><br />
<span style="font-size: large;"><br /></span>
<br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">As you noted, both papers originated from the same randomized controlled intervention study (clinical trial number: NCT02007616). We never pretended that the articles were to be considered as two different intervention studies. Part of this confusion could have been generated because in the Frontiers article the clinical trial number did not appear on the first page, even though this number was included in the four preliminary versions of the manuscript under revision. We have contacted Frontiers asking them to include the clinical trial number in the final accepted version, if possible. </span></i></blockquote>
<span style="font-size: large;">Although that would help, it's not sufficient. The problem is that the data analyses are republished. It would be good to note, explicitly, both in the method section and in the results section, that this paper is reporting outcome measures from the same intervention. And, it's essential to note when and where analyses are repeated.</span><br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">If it is not possible at this stage, we asked them to publish a note with this information and to acknowledge in the results section the overlap as well as in Figure 3b, mentioning that the data in the Figure were published previously in PLoS.</span></i></blockquote>
<span style="font-size: large;">This seems necessary regardless of whether or not the link to the clinical trial number is added. Even if it is made clear that the paper is based on the same intervention, it still is not appropriate to republish data and results without explicitly noting that they were published previously. Actually, it would be better not to republish the data and analyses. Period.</span><br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">You also indicated in your first comments posted in Frontiers that the way we reported our results could give the impression that they come from two different and independent interventions. To avoid this misunderstanding, as you noticed, we introduced two references in our Frontiers ́ article. We inserted the first reference to the PLoS article in the Method section and a second reference in the Discussion section. Two references that you considered were not enough to avoid possible confusions in the reader.</span></i></blockquote>
<span style="font-size: large;">As I discussed in my HI-BAR post, these citations were wholly inadequate. One noted only that the results of the oddball task were reported elsewhere, yet the same results were reported again in <i>Frontiers</i> and results section included no mention of the duplication. The other citation, appearing in the discussion, implied that the earlier paper provided additional evidence for a conclusion about the brain. Nowhere did the <i>Frontiers</i> paper cite the <i>PLoS</i> paper for the details of the intervention, the nature of the outcome measures, etc. It just published those findings as if they were new. The text itself should have noted, both in the method and results sections, whenever procedures or results were published elsewhere. Again, it would have been better not to republish them at all.</span><br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">In relation to the results section in which we describe the cross-modal oddball attention task results in the Frontiers article, we acknowledge that, perhaps, it would have been better to avoid a detailed presentation of the attentional results that were already published and indicate just that attentional data resulting from the task developed in collaboration with the UBI group were already published. We could have asked the readers to find out for themselves what happened with attention after training. We were considering this possibility but in the end we decided to include the results of the oddball task to facilitate ease of reading.</span></i></blockquote>
<span style="font-size: large;">Acknowledging the repetition explicitly in the text would have helped avoid the interpretation that these were new results. Repeating the statistics from the earlier paper isn't an "ease of reading" issue -- it's duplicate publication. You could easily summarize the findings of the earlier paper, with clear citation, and note that the details were described in that paper. I don't see any justification for republishing the actual statistics and findings. </span><br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">Regarding the last paragraph of the oddball results section, we said “New analyses were conducted....” As we said above, we tried (perhaps in an unfortunate way) to give in this paper a brief summary of the results obtained in the attention task, so this paragraph refers to the epigraphs “Distraction” and “Alertness” of results section in PLoS publication, where differential variables were calculated and new analyses were performed to obtain measures of gains and losses in the ability to ignore infrequent sounds and to take advantage of the frequent ones. Once again, we apologize if this sentence has led to confusion, and we are in contact with the Journal concerning this.</span></i></blockquote>
<span style="font-size: large;">Yes, that sentence added to the impression these analyses were new to the <i>Frontiers</i> paper. But, the statistical analyses should not have been duplicated in the first place. That's also true for the extensive repetition of all of the training data.</span><br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">Another comment in your exhaustive review referred to the differences in the samples shown in the Consort diagram between the two publications. This has a simple explanation. The diagram of the PLoS article refers only to the cross-modal oddball task designed to assess alertness and distraction while the Frontier ́s Consort diagram refers to the whole intervention study. In the control group, one of the participants was removed due to the large number of errors in the oddball task, but he was included in the rest of the study in which he reached inclusion criteria. The same occurred in the trained group. As attentional measures were analyzed separately by the UBI group, by the time we sent the results only fifteen participants completed the post-evaluation (we could not contact a participant and the other was travelling that week). A few days later, these two participants completed the post-evaluation, but we decided not to include them in the PLoS sample as the data were already analyzed by the UBI group.</span></i></blockquote>
<span style="font-size: large;">Thank you for these clarifications. I'm curious why you decided not to wait for a few days for those remaining participants if that was part of your intervention design. If their results came in a few days later, why not re-do the analysis in the original paper to include all participants who were in the intervention. Presumably, by publishing the PLoS paper when you did, you deemed the intervention to be complete (i.e., it wasn't flagged as a preliminary result). It seems odd to then add these participants to the next paper. This difference between papers raises two questions. First, would the results for the PLoS paper have been different with those two participants? Second, would the results of the Frontiers paper have differed if they had been excluded? And, if those data were in by the time the Frontiers paper was written, why were these participants not included in the oddball analysis? At least that would have added new information to those duplicated analyses.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This clarified information should have been included in the </span><span style="font-size: large;"><i>Frontiers</i> paper to make it clearer that both papers reported the same intervention with the same participants.</span><br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">We would like to explain the clinical trial registration process. As you pointed out in your comments to Mayas et al. (2014), we registered the clinical trial after the attention manuscript was submitted to PLoS. The journal (PlosOne) specifically required the registration of the intervention study as a clinical trial during the revision process in order to publish the manuscript, and told us about the possibility of post-registration. The Editor of PLoS sent us the link to register the study.</span></i></blockquote>
<span style="font-size: large;">Post-registering a study completely undermines the purpose of registration. I find it disturbing that <i>PLoS</i> would permit that as an option for a clinical trial. Looking at the <i>PLoS</i> guidelines, they do make an exception for post-study registration provided that the reasons for "failing to register before participant recruitment" are spelled out clearly in the article (emphasis from original on <a href="http://www.plosone.org/static/editorial#clinical">PLoS website</a>): </span><br />
<blockquote class="tr_bq" style="background-color: white; color: #333333; font-family: arial, sans-serif; line-height: 18.0049991607666px; margin-bottom: 1em; padding: 0px;">
<em>PLOS ONE</em> supports prospective trial registration (i.e. before participant recruitment has begun) as recommended by the ICMJE's <a href="http://www.icmje.org/about-icmje/faqs/clinical-trials-registration/" style="color: #3c63af; text-decoration: none;">clinical trial registration policy</a>.<strong>Where trials were not publicly registered before participant recruitment began</strong>, authors must:<br />
<ul style="background-color: white; color: #333333; font-family: arial, sans-serif; line-height: 18.0049991607666px; margin: 0px 0px 1em; padding: 0px 0px 0px 45px;">
<li style="line-height: 1.5em; margin: 0px; padding: 0px;">Register all related clinical trials and confirm they have done so in the Methods section</li>
</ul>
<ul style="background-color: white; color: #333333; font-family: arial, sans-serif; line-height: 18.0049991607666px; margin: 0px 0px 1em; padding: 0px 0px 0px 45px;">
<li style="line-height: 1.5em; margin: 0px; padding: 0px;">Explain in the Methods the reason for failing to register before participant recruitment</li>
</ul>
</blockquote>
<span style="font-size: large;">It's also clear that their policy is for clinical trials to be pre-registered, not post-registered. And, the exception for post-registration requires a justification. </span><span style="font-size: large;">Neither the <i>PLoS</i> paper nor the <i>Frontiers</i> paper provided any such justification. The <i>Frontiers</i> paper didn't mention registration at all, and the <i>PLoS</i> one didn't make clear that the registration occurred after submission of the finished study. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The idea of registration is to specify, in advance, the procedures that will be followed in order to avoid introducing investigator degrees of freedom. Registering after the fact does nothing to address that problem. It's just a re-description of an already completed study. It's troubling that a <i>PLoS</i> editor would instruct the authors to post-register a study.</span><br />
<blockquote class="tr_bq">
<i><span style="font-size: large;">We would like to clarify some questions related to the data analysis. First, multiple tests in all ANOVAs were Bonferroni corrected although it is not made explicit in the text. </span></i></blockquote>
<span style="font-size: large;">As best I can tell, the most critical multiple testing issues, the ones that could have been partly remedied by pre-registration, were not corrected in this paper. </span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">There are at least four distinct multiple testing issues:</span><br />
<br />
<ol>
<li><span style="font-size: large;"> There are a large number of outcome measures, and as best I can tell, none of the statistical tests were corrected for the number of tests conducted across tasks. </span></li>
<li><span style="font-size: large;">There are multiple possible tests for a number of the tasks (e.g., wellbeing has a number of sub-scales), and there don't appear to have been corrections for the number of distinct ways in which an outcome could be measured. </span></li>
<li><span style="font-size: large;">A multi-factor ANOVA itself constitutes multiple tests (e.g., a 2x2 ANOVA involves 3 tests: each main effect and the interaction). Few studies correct for that multiple testing problem, and this paper does not appear to have done so. </span></li>
<li><span style="font-size: large;">There is a multiple tests issue with pairwise comparisons conducted to follow-up a significant ANOVA. I assume those are the Bonferroni-corrected tests that the authors referred to above. However, it's impossible to tell if these tests were corrected because the paper did not report the test statistics — it just reported p < xx or p= xx. Only the F tests for the main effects and interactions were reported.</span></li>
</ol>
<span style="font-size: large;">If the authors did correct for the first three types of multiple testing issues, perhaps they can clarify. However, based on the ANOVA results, it does not appear that they did.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">A related issue, one mentioned in my HI-BAR on the PLoS paper but not on the Frontiers paper, is that some of the reported significance levels for the F tests are incorrect. Here are some examples from the <i>Frontiers</i> paper in which the reported significance level (p<xx or p=xx) is wrong. For each, I give the correct p value in <b><span style="color: red;">red</span></b> (uncorrected for multiple tests). None of these calculations led to less statistical significance:</span><br />
<br />
<ul>
<li>[F(1, 28) = 3.24, MSE = 1812.22, p = 0.057, η2p = 0.12]. <b><span style="color: red;">p=.0826</span></b></li>
<li>[F(2, 50) = 5.52, MSE = 0.001, p < 0.005, η2p = 0.18]. <b><span style="color: red;">p=.0068</span></b></li>
<li>[F(1, 28) = 4.35, MSE = 0.09, p < 0.001, η2p = 0.89]. <span style="color: red;"><b>p=.0462</b></span></li>
<li><strike>[F(1, 28) = 17.98, MSE = 176.74, p < 0.001, η2p = 0.39].<span style="color: red;"><b> p=.0002</b></span></strike></li>
<li><strike>[F(1, 28) = 13.02, MSE = 61.49, p < 0.01, η2p = 0.32]. <span style="color: red;"><b>p=.0012</b></span></strike></li>
<li>[F(1, 28) = 3.42, MSE = 6.47, p = 0.07, η2 = 0.11].<b><span style="color: red;"> p=.0750</span></b></li>
</ul>
<span style="font-size: large;">The following two results were reported as statistically significant at p<.05, but actually were not significant with that alpha level:</span><br />
<ul>
<li>[F(1, 28) = 3.98, MSE = 0.06, p < 0.01, η2p = 0.12]. <span style="color: red;"><b>p=.0559</b></span></li>
<li>[F(1, 28) = 3.40, MSE = 0.15, p = 0.04, η2p = 0.10]. <span style="color: red;"><b>p=.0758</b></span></li>
</ul>
<div>
<span style="font-size: large;">I don't know that any of these errors dramatically changes the interpretation of the results, but they should be corrected.</span></div>
<blockquote class="tr_bq">
<i><span style="font-size: large;">Second, the RTs filtering was not an arbitrary decision. The lower limit (200 ms) reflects the approximate minimum amount of time necessary to respond to a given stimulus (due to speed limitations of the neurological system). RTs below this limit may reflect responses initiated before the stimulus onset (guessing). The selection of the upper limit (1100 ms) is a more difficult decision. Different criteria have been proposed (statistical, theoretical...) in the literature (see Whelan, 2008). Importantly none of them seem to affect type I errors significantly. In this case, we made the following theoretical assumption: RTs longer than 1100 ms might depend on other cognitive processes than speed of processing.</span></i></blockquote>
<span style="font-size: large;">There is nothing wrong with this choice of cutoffs, and it might well have been principled. Still, it is arbitrary. Any number of cutoffs would have been just as appropriate (e.g., 150ms and 1200ms, 175ms and 3000ms, ± 3SD, etc). My point wasn't to question the choice, but instead to note that it introduces flexibility to the analysis. This is a case in which pre-registering the analysis plan would help -- the choice of cutoffs is reasonable, but flexible. Registration eliminates that flexibility, making it clear to readers that the choice of cutoffs was principled rather than based on knowing the outcome with different cutoffs. Another approach would be to assess the robustness of the results to various choices of cutoff (reporting all results).</span><br />
<div class="page" title="Page 3">
<div class="layoutArea">
<div class="column">
</div>
</div>
</div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-91319332994570348722014-11-20T13:56:00.004-06:002020-12-21T08:07:07.902-06:00HI-BAR: More benefits of Lumosity training for older adults?<h2 class="tr_bq" style="background-color: white; font-stretch: normal; margin: 0px 0px 1em; position: relative;">
<div style="font-family: 'Trebuchet MS', Trebuchet, Verdana, sans-serif; font-size: 11px; text-align: center;">
<span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: x-large;">HI-BAR (Had I Been A Reviewer)</span></div>
</h2>
<div style="text-align: center;">
<i><b><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">For more information about HI-BAR reviews, see my </span><a href="http://blog.dansimons.com/2013/09/hi-bar-had-i-been-reviewer.html" style="font-family: Arial, Helvetica, sans-serif; font-size: large;">post</a><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> introducing the acronym</span></b></i></div>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"></span><br />
<span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><a href="http://journal.frontiersin.org/Journal/10.3389/fnagi.2014.00277/abstract">Ballesteros S, Prieto A, Mayas J, Toril P, Pita C, Ponce de León L, Reales JM and Waterworth J (2014) Brain training with non-action video games enhances aspects of cognition in older adults: a randomized controlled trial. Front. Aging Neurosci. 6:277. doi: 10.3389/fnagi.2014.00277</a></span><br />
<div>
<br /></div>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif;"><b>Update:</b> <i>I posted a brief summary of this HI-BAR as a comment on the paper at Frontiers, and Dr. Ballesteros responded to indicate that this was the same training study. Her response quoted the one-sentence citation described below and noted that they did not intend to give the impression that this was a new intervention. Her reply did not address the overlap in presentation of methods and results.</i></span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif;"><i><br /></i></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif;"><b>Update 2:</b><i> Dr. Ballesteros sent me a response to my questions [</i><a href="https://drive.google.com/file/d/0B8SdDrkCc9RiV3FZNXZWN2ZGaGc/view?usp=sharing" style="font-style: italic;" target="_blank"><span id="goog_1471992043"></span>pdf<span id="goog_1471992044"></span></a><i>]. I have just posted a </i><a href="http://blog.dansimons.com/2014/12/response-from-ballesteros-et-al-to-my.html" style="font-style: italic;">new blog post</a><i> in which I quote from her response and add new comments. </i></span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">This paper testing the benefits of Lumosity training for older adults </span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">just appeared in </span><a href="http://journal.frontiersin.org/Journal/10.3389/fnagi.2014.00277/abstract" style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><i>Frontiers</i></a><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">. The paper is co-authored two of the same people (</span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">Ballesteros and Mayas)</span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> as a recent <i>PLoS One </i>paper on Lumosity training that I critiqued in a <a href="http://blog.dansimons.com/2014/04/hi-bar-benefits-of-lumosity-training.html">HI-BAR in April</a> (note that the new paper includes six additional authors who were not on the <i>PLoS</i> paper and the <i>PLoS</i> paper includes two who weren't on the <i>Frontiers</i> paper). </span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">I was hopeful that this new paper would address some of the shortcomings of the earlier one. Unfortunately, it didn't. In fact, it couldn't have, because the </span><span style="background-color: white;"><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">"new" Frontiers paper is based on <b>the same intervention</b> as the <i>PLoS</i> paper.</span></span><br />
<span style="background-color: white;"><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span></span>
<span style="background-color: white;"><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">And, when I say, "the same intervention," I don't mean that they directly replicated the procedures from their earlier paper. The <i>Frontiers</i> paper reports data from the same intervention with the same participants. It would be hard to know that</span></span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> from reading just the <i>Frontiers</i> paper, though, because it does not mention that these are additional measures from the same intervention. </span><br />
<span style="background-color: white;"><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span></span>
<span style="background-color: white;"><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">As my colleagues and I have noted (ironically, <a href="http://journal.frontiersin.org/Journal/10.3389/fpsyg.2011.00226/full">in <i>Frontiers</i></a>), this sort of partial reporting of outcome measures gives the misleading impression that there are more interventions demonstrating transfer of training than actually exist. If a subsequent meta-analysis treats these reports as independent, it will produce a distorted estimate of the size of any intervention benefits. Moreover, whenever a paper does not report all outcome measures, there's no way to appropriately correct for multiple comparisons. At </span></span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">a minimum, all publications derived from the same intervention study should state explicitly that it is the same intervention and identify all outcome measures that will, at some point, be reported. </span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">The <i>Frontiers</i> cites the <i>PLoS</i> paper only twice. The first citation appears in</span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> the method section in reference to the oddball outcome task that was the focus of the <i>PLoS</i> paper:</span><br />
<blockquote class="tr_bq">
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">Results from this task have been reported separately (see Mayas et al., 2014).</span></blockquote>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">That is the only indication in the paper that there might be overlap of any kind between the study presented in <i>Frontiers</i> and the one presented in <i>PLoS</i>. It's a fairly minimalist way to indicate that this was actually a report of different outcome measures from the same study. </span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">It's also not entirely true: The</span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> results section of the <i>Frontiers</i> paper describes the results for that oddball task in detail as well, and all of the statistics for that task reported in the <i>Frontiers</i> paper were also reported in the <i>PLoS</i> paper. The results section does not acknowledge the overlap. </span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">Figure 3b in the <i>Frontiers</i> paper is nearly identical to Figure 2b in the <i>PLoS</i> paper, reporting the same statistics from the oddball task with no mention that the data in the figure were published previously.</span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">After repeating the main results of the oddball task from the earlier paper, the <i>Frontiers</i> paper states:</span><br />
<blockquote class="tr_bq">
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">"</span><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">New analyses were conducted on distraction (novel vs. standard sound) and alertness (silence vs. standard sound), showing that the ability to ignore relevant sounds (distraction) improved significantly in the experimental group after training (12 ms) but not in the control group. The analyses of alertness showed that the experimental group increased 26 ms in alertness (p < 0.05) but control group did not (p = 0.54)."</span></blockquote>
<span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">I thought "new analysis" might mean that these analyses were new to this paper, but they weren't. Here's the relevant section of the <i>PLoS</i> paper (which reported the actual statistical tests):</span><br />
<blockquote class="tr_bq">
<span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">"In other words, the ability to ignore irrelevant sounds improved significantly in the experimental group after video game training (12 ms approximately) but not in the control group....Pre- and post-training comparisons showed a 26ms increase of alertness in the experimental group..."</span></blockquote>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">Stating that "results from this task have been reported separately" implies that any analyses of that task reported in </span><i style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: large;">Frontiers</i><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> are new, but they're not.</span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">The <i>Frontiers</i> paper also reports the same training protocol and training performance data without reference to the earlier study.</span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> The <i>Frontiers</i> paper put the training data </span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">in a figure whereas the <i>PLoS</i> paper put them in a table. Why not just cite the earlier paper for the training data rather than republishing them without citation?</span><br />
<br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">The same CONSORT diagram appears as Figure 1 in both papers. The <i>Frontiers</i> paper CONSORT diagram does report that the experimental condition had data from 17 participants rather than the 15 participants reported in the <i>PLoS</i> paper. From the otherwise identical figures, it appears that those two additional participants were the ones that the <i>PLoS</i> paper noted were lost to followup due to "personal travel" and "no response from participant." I guess both responded after the <i>PLoS</i> paper was completed, although the method sections of the two papers noted that testing took place over the same time period (January through July of 2013). The presence of data from these two additional participants did not change any of the statistics for the oddball task that were reported in both papers. </span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">One other oddity about the subject samples in the consort diagram: The <i>PLoS</i> paper excluded one participant from the control condition analyses due to "Elevated number of errors," leaving 12 participants in that analysis. The <i>Frontiers</i> paper did not exclude that participant. If they made too many errors for the <i>PLoS</i> paper, why were they included in the <i>Frontiers</i> paper?</span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">Other than the one brief note in the method section about the oddball task, the only other reference to the <i>PLoS</i> paper appeared in the discussion, and gives the impression that the <i>PLoS</i> paper provided </span><i style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: large;">independent</i><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> evidence for the contribution of frontal brain regions to alertness and attention filtering:</span><br />
<blockquote class="tr_bq">
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">We also found that the trainees reduced distractibility by improving alertness and attention filtering, functions that decline with age and depend largely on frontal regions (Mayas et al., 2014).</span></blockquote>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">That citation and the one in the method section are the only mentions of the earlier paper in the entire article. </span><br />
<br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><span style="background-color: white;">Given that I was already familiar with the PLoS paper, had I been a reviewer of this paper, here's what I would have said:</span></span><br />
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><br /></span>
<br />
<ol>
<li><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">The paper suffers from all of the same criticisms I raised about the intervention in the </span><i style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: large;">PLoS</i><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> paper (e.g., inadequate control group among many other issues), and in my view, those shortcomings should preclude publication of this paper. See </span><span style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"><a href="http://blog.dansimons.com/2014/04/hi-bar-benefits-of-lumosity-training.html">http://blog.dansimons.com/2014/04/hi-bar-benefits-of-lumosity-training.html</a></span></li>
<li><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">The paper should have explained that this was the same intervention study, with the same participants, as the earlier </span><i style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: large;">PLoS One</i><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> paper.</span></li>
<li><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">The paper should have noted explicitly which data and results were reported previously and which were not.</span></li>
</ol>
<span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">It's possible that the authors notified the editor of the overlap between these papers upon submission. If so, it is surprising that the paper was even sent out for review before these issues were addressed. If the editor was not informed of the overlap, I cannot fault the </span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">editor and reviewers for missing the fact that the intervention, training results, and some of the outcome results were reported previously. Unless they happened to know about the </span><i style="color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: large;">PLoS One </i><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;">paper, they could be excused for missing the one reference to that paper in the method section.</span><span style="background-color: white; color: #1b1b1b; font-family: Arial, Helvetica, sans-serif; font-size: medium;"> Still, the many other major problems with this intervention study probably should have precluded publication, at least without major corrections and qualifications of the claims.</span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-54593221840961123002014-04-02T13:39:00.001-05:002020-12-21T08:07:18.094-06:00HI-BAR: Benefits of Lumosity training for older adults?<h2>
<div style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;">HI-BAR (Had I Been A Reviewer)</span></div>
<div style="text-align: center;">
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: small;"><br /></span></b></div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><div style="font-weight: normal; text-align: center;">
A post-publication review of Mayas J., Parmentier, F. B. R., Andrés P., & Ballesteros, S. (2014) Plasticity of Attentional Functions in Older Adults after Non-Action Video Game Training: A Randomized Controlled Trial. PLoS ONE 9(3): e92269. <a href="http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0092269#pone-0092269-g002">doi:10.1371/journal.pone.0092269</a></div>
</span></h2>
<div style="text-align: center;">
<i><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">For more information about HI-BAR reviews, see my <a href="http://blog.dansimons.com/2013/09/hi-bar-had-i-been-reviewer.html">post</a> introducing the acronym.</span></i></div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;">
<span style="font-size: large;">This paper explored "whether the ability to filter out irrelevant information...can improve in older adults after practicing non-violent video games." In this case, the experimental group played 10 games that are part of the Lumosity program for a total of 20 hours. The control group did not receive any training. Based on post-training improvements on an "oddball" task (a fairly standard attention task, not a measure of quirkiness), the authors claim that training improved the ability to ignore distractions and increased alertness in older adults. </span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Testing whether commercial brain training packages have any efficacy for cognitive enhancement is a worthwhile goal, especially given the dearth of robust, reliable evidence that such training has any measurable impact on cognitive performance on anything other than the trained tasks. I expect that Lumosity will tout this paper as evidence for the viability of their brain training games as a tool to improve cognition. They probably shouldn't.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Below are the questions and concerns I would have raised had I been a reviewer of this manuscript. If you read my <a href="http://blog.dansimons.com/2013/09/19-questions-about-video-games.html">earlier HI-BAR</a> review of Anguera et al (2013), you'll recognize many of the same concerns. Unfortunately, the problems with this paper are worse. A few of these questions could be answered with more information about the study (I hope the authors will provide that information). Unfortunately, many of the shortcomings are more fundamental and undermine the conclusion that training transferred to their outcome measure.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
I've separated the comments into two categories: Method/Analysis/Inferential issues and Reporting issues.</span><br />
<h3 style="text-align: center;">
</h3>
<div>
<h2 style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></h2>
<div style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;"><b>Method/Analysis/Inferential Issues</b></span></div>
<div style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;"><b><br /></b></span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><b>Sample size</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The initial sample consisted of 20 older adults in the training group and 20 in the control group. After attrition, the analyzed sample consisted of only 15 people in the training group and 12 in the control group. That's a really small sample, especially when testing older adults who can vary substantially in their performance.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span></div>
<div>
<div>
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Inadequate control group</span></b></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The control group for this paper is inadequate to make any causal claims about the efficacy of the training. The experimental group engaged in 20 hours of practice with Lumosity games. The control group "attended meetings with the other members of the study several times along the course of the study;" they got to hang out, chat, and have coffee with the experimenters a few times. This sort of control condition is little better than a no-contact control group (not even the amount of contact was equated). <a href="http://pps.sagepub.com/content/8/4/445.full">Boot et al (2013) explained</a> how inadequate control conditions like this "limited contact" one provide an inadequate baseline against which to evaluate the effectiveness of an intervention. Here's the critical figure from our paper showing the conclusions that logically follow from interventions with different types of control conditions:</span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhK137Vsz9-67Nc0Rc8fAjljgiElA2WyTbbvMNAqQB8rRbBL6iKfDot-9Rcqk9qO-06lB52WT2yE5FjMi5uq3Dwfw-lbqKaG_V20Bmh9QtwbGZ1BuRCkBX9sauETtqqBGTscO0lSibW5Tg/s1600/flow_chart.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhK137Vsz9-67Nc0Rc8fAjljgiElA2WyTbbvMNAqQB8rRbBL6iKfDot-9Rcqk9qO-06lB52WT2yE5FjMi5uq3Dwfw-lbqKaG_V20Bmh9QtwbGZ1BuRCkBX9sauETtqqBGTscO0lSibW5Tg/s1600/flow_chart.jpg" height="320" title="What can we conclude from an intervention" width="275" /></span></a></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">When inferring the causal potency of any treatment, it must be compared to an appropriate baseline. That is why drug studies use a placebo control, ideally one that equates for side effects too, so that participants do not know whether they have received the drug or a placebo. For a control condition to be adequate, it should include all of the elements of the experimental group excepting the critical ingredient of the treatment (including equating for expectation effects). Otherwise, any differences in outcome could be due to other differences between the groups. That subtractive method, first described by Donders more than 150 years ago, is the basis of clinical trial logic. Unfortunately, it commonly is <a href="http://pps.sagepub.com/content/8/4/445.full">neglected</a> in psychological interventions.</span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">In video game training, even an active control group in which participants play a different game might not control for differential placebo effects on outcome measures. But, the lack of an active control group allows almost no definitive conclusions: It does not equate the experience between the training group and the control group in any substantive way. This limited-contact control group accounts for test-retest effects and the passage of time, and little else. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><span style="font-family: Arial, Helvetica, sans-serif;">Any advantage observed for the training group could result from many factors that are not specific to the games involved in the training condition or even to games at all: Any benefits could have resulted from doing something intensive for 20 hours, from greater motivation to perform well on the outcome measures, from greater commitment to the tasks, from differential placebo effects, from greater social contact, etc. </span><span style="font-family: Arial, Helvetica, sans-serif;">Differences between the training group and this limited-contact control group do not justify any causal conclusion about the nature of the training.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Interventions and training studies that lack any control condition other than a no-contact or limited-contact control group should not be published. Period. They are minimally informative at best, and misleading at worst given that they will be touted as evidence for the benefits of training. The literature on game training is more than 10 years old, and there is no justification for publishing studies that claim a causal benefit of training if they lack an adequate baseline condition.</span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span></div>
<div>
<div>
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Multiple tests without correction</span></b></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The initial analysis consisted of two separate 2x2x3 ANOVAs on accuracy and response times on the oddball task, accompanied by follow-up tests. A 3-factor ANOVA consists of 7 separate F-tests (3 main effects, 3 two-way interactions, and 1 three-way interaction). Even if the null hypothesis were true and all of the data were drawn from a single population, we would expect a significant result on at least one of these tests more than 30% of the time on average (1 - .95^7). In other words, each of the ANOVAs has a 30% false positive rate. For a thorough discussion of this issue, see <a href="http://deevybee.blogspot.co.uk/2013/06/interpreting-unexpected-significant.html">this excellent blog post</a> from Dorothy Bishop.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><span style="font-family: Arial, Helvetica, sans-serif;">The critical response time analysis, the only one to show a differential training effect, produced two significant tests out of the 7 conducted in the ANOVA: a main effect of condition as for accuracy (but not with the predicted pattern), and a significant 3-way interaction. </span><span style="font-family: Arial, Helvetica, sans-serif;">The results section does not report any correction for multiple tests, though, and</span><span style="font-family: Arial, Helvetica, sans-serif;"> the critical 3-way interaction would not have been significant (it was p=.017 without correction).</span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Possible speed-accuracy tradeoff</span></b></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The accuracy ANOVA showed a marginally significant 3-way interaction (p=.077 without correction for multiple tests), but the paper does not report the means or standard deviations for the accuracy results. Is it possible that the effects on accuracy and RT were in opposite directions? If so, the entire difference between training groups could just be a speed-accuracy tradeoff, with no actual performance difference between conditions.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span><br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><b>Flexible and arbitrary analytical decisions</b>. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">For response times, the analysis included only correct trials and excluded times faster than 200ms and slower than 1100ms. Standard trials after novel trials were discarded as well. These choices seem reasonable, but arbitrary. Would the results hold with different cutoffs and criteria? Were any other cutoffs tried? If so, that introduces additional flexibility (investigator degrees of freedom) that could spuriously inflate the significance of tests. That's one reason why pre-registration of analysis plans is essential. It's all too easy to rationalize any particular approach after the fact if it happens to work.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span><br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Unclear outcome measures</span></b><br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div>
<div style="margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The three way interaction for response time was followed up with more tests that separated the oddball task into an alertness measure and a distraction measure analyzed separately for the two groups. It's not clear how these measures were derived from the oddball conditions, but I assume they were based on different combinations of the silent, standard, and novel noise conditions. It would be nice to know what these contrasts were as they provide the only focused tests of differential transfer task improvements between the experimental group and the control group. </span></div>
</div>
</div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: large;">
</span>
</span><br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="font-style: normal;">
<div style="margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><b>A difference in significance is not a significant difference</b></span></div>
<div style="font-weight: normal; margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The primary conclusions about these follow-up outcome measures are based on significant improvements for the training group (reported as p=.05 and p=.04) and the absence of a significant improvement for the control group. Yet, significance in one condition and not in another does not mean that those two conditions differed significantly. No test of the difference in improvements across conditions was provided.</span></div>
</div>
<div style="font-weight: normal;">
<div style="font-style: normal; margin: 0px;">
</div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
</div>
<div style="font-style: normal;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="margin: 0px;">
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Inappropriate truncation/rounding of p-values</span></b></div>
</div>
<div style="margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The critical "significant" effect for the training group actually wasn't significant! The authors reported "<em style="background-color: white; color: #333333; font-style: normal; line-height: 18.0049991607666px;">F</em><span style="background-color: white; color: #333333; font-style: normal; line-height: 18.0049991607666px;">(1,25) = 4.00, </span><em style="background-color: white; color: #333333; font-style: normal; line-height: 18.0049991607666px;">MSE</em><span style="background-color: white; color: #333333; font-style: normal; line-height: 18.0049991607666px;"> = 474.28, </span><em style="background-color: white; color: #333333; font-style: normal; line-height: 18.0049991607666px;">p</em><span style="background-color: white; color: #333333; font-style: normal; line-height: 18.0049991607666px;"> = .05, d = 0.43]</span>." Whenever I see a p value reported as exactly .05, I get suspicious. So, I checked. F(1,25) = 4.00 gives <i>p = .0565</i>. Not significant. The authors apparently truncated the p-value. </span></div>
<div style="font-style: normal; margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div style="font-style: normal; margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">(The reported p=.04 is actually p=.0451. That finding was rounded, but rounding down is not appropriate either.)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Of the two crucial tests in the paper, one wasn't actually significant and the other was just barely under p=.05. Not strong evidence for improvement, especially given the absence of correction for multiple tests (with which, neither would be significant). </span></div>
</div>
</div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><b><br /></b><b>Inappropriate conclusions from correlation analyses</b></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The paper explored the correlation between the alertness and distraction improvements (the two outcome measures) and each of the 10 games that made up the Lumosity training condition. The motivation is to test whether the amount of improvement an individual showed during training correlated with the amount of transfer they showed on the outcome measures. Of course, with N=15, no correlation is stable and any significant correlation is likely to be substantially inflated. The paper included no correction for the 20 tests they conducted, and neither of the two significant correlations (p=.02, p<.01) would survive correction. Moreover, even if these correlations were robust, correlations between training improvement and outcome measure improvement logically <i>provide no evidence for the causal effect of training on the transfer task</i> (See <a href="http://link.springer.com/article/10.3758/s13423-013-0560-7">Tidwell et al, 2013</a>).</span></div>
</div>
</div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><b><span style="font-size: large;"><br /></span></b>
<b><span style="font-size: large;">Inaccurate conclusions</span></b></span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The authors write: </span><br />
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">"The results of the present study suggest that training older adults with non-action video games reduces distractibility by improving attention filtering (a function declining with age and largely depending on frontal regions) but also improves alertness." </span></blockquote>
<span style="font-size: large;"><span style="font-family: Arial, Helvetica, sans-serif;">First, the study provides no evidence at all </span><span style="font-family: Arial, Helvetica, sans-serif;">that any differences resulted from improvements in an attention filtering mechanism. It provided not tests of that theoretical idea.</span><span style="font-family: Arial, Helvetica, sans-serif;"> </span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Furthermore, the study as reported does not show that training differentially reduced distractibility or increased alertness. The improved alertness effect was not statistically significant when the p-value isn't truncated to .05. The effect on the distraction measure was 12ms (p=-.0451 without correction for multiple tests). Neither effect would be statistically significant with correction for multiple tests. But, even if they were significant, without a test of the difference between the training effect and the control group effect, we don't know if there was any significant difference in improvements between the two groups; significance in one condition but not another does not imply a significant difference between conditions.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div>
<div>
<div style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;"><b>Reporting Issues</b></span></div>
<h3 style="text-align: center;">
<div style="font-weight: normal; text-align: start;">
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Pre-Registration or Post-Registration</span></b></div>
<div style="font-weight: normal; text-align: start;">
<span style="font-size: large;"><span style="font-family: Arial, Helvetica, sans-serif;">The authors <a href="http://clinicaltrials.gov/ct2/show/NCT02007616?term=Ballesteros&rank=3">registered their study</a> on ClinicalTrials.gov on December 4, 2013; i</span><span style="font-family: Arial, Helvetica, sans-serif;">t's listed and linked under the "trial registration" heading in the published article. </span><span style="font-family: Arial, Helvetica, sans-serif;">That's great. </span><span style="font-family: Arial, Helvetica, sans-serif;">Pre-registration is the best way to eliminate p-hacking and other investigator degrees of freedom. </span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
But, this wasn't a pre-registration: The manuscript was submitted to PLoS three months <i>before</i> it was registered! What is the purpose of registering an already completed study? Should it even count as a registration?</span></div>
<div style="font-weight: normal; text-align: start;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span></div>
<div style="font-weight: normal; text-align: start;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><b>Unreported outcome measures</b> </span></div>
<div style="font-weight: normal; text-align: start;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The only outcome measure mentioned in the published article is the "oddball task," but the <a href="http://clinicaltrials.gov/ct2/show/NCT02007616?term=Ballesteros&rank=3">registration</a> on ClinicalTrials.gov identifies the following additional measures (none with any detail): neuropsychological testing, Wisconsin task, speed of processing, and spatial working memory. Presumably, these measures were collected as part of the study? After all, the protocol was registered after the paper was submitted. Why were they left out of the paper?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Perhaps the registration was an acknowledgment of the other measures and they are planning to report each outcome measure in a separate journal submission. Dividing a large study across multiple papers can be an acceptable practice, provided that all measures are identified in each paper and readers are informed about all of the relevant outcomes (the papers must cross-reference each other). </span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Sometimes a large-scale study is preceded by a peer-reviewed "design" paper that lays out the entire protocol in detail in advance. This registration lacks the necessary detail to serve as a roadmap for a series of studies. Moreover, separating measures across papers without mentioning that there were other outcome measures or that some of the measures were (or will be) reported elsewhere is misleading. It gives the false impression that these outcomes came from different, independent experiments. A meta-analysis would treat them as independent evidence for training benefits when they aren't independent.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Unless the results from all measures are published, readers have no way to interpret the significance tests for any one measure. Readers need to know the total number of hypothesis tests to determine the false positive rate. Without that information, the significance tests are largely uninterpretable.</span><br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
</div>
<div style="font-weight: normal; text-align: start;">
<span style="font-size: large;"><span style="font-family: Arial, Helvetica, sans-serif;">Here's a more troubling possibility: Perhaps the results for these other measures weren't significant, so the authors chose not to report them (or the reviewers/editor told them not to). If true, this underreporting—providing only the outcome measure that showed an effect—constitutes p-hacking, <a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1850704">increasing the chances</a> that any significant</span><span style="font-family: Arial, Helvetica, sans-serif;"> results in the article were false positives. </span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Without more information, readers have no way to know which is the case. And, without that information, it is not possible to evaluate the evidence. This problem of incomplete reporting of outcome measures (and neglecting to mention that separate papers came from the same study) has occurred in the game training literature before—see "The Importance of Independent Replication" section in <a href="http://journal.frontiersin.org/Journal/10.3389/fpsyg.2011.00226/abstract">this 2012 paper</a> for some details. </span></div>
<div style="font-weight: normal; text-align: start;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span></div>
<div style="font-weight: normal; text-align: start;">
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Conflicting documentation of outcome measures</span></b></div>
<div style="font-weight: normal; text-align: start;">
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The supplemental materials at PLoS include a protocol that lists only the oddball task and neuropsychological testing. The supplemental materials also include an author-completed Consort Checklist that identifies where all the measures are described in the manuscript. The checklist includes the following items for "outcomes": </span></div>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">"6a. Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed." </span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">"6b. Any changes to trial outcomes after the trial commenced, with reasons." </span></blockquote>
<div>
<span style="font-size: large;"><span style="font-family: Arial, Helvetica, sans-serif;">For 6a, the authors answered "Methods." Yet, the primary and secondary outcomes noted in the Clinicaltrials.gov registration are not fully reported in the methods of the actual article or in the protocol page. For 6b, they responded</span><span style="font-family: Arial, Helvetica, sans-serif;"> "N/A," Implying that the design was carried out as described in the paper.</span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">These forms and responses are inconsistent with the protocol description at clinicaltrials.gov. Either the paper and supplementary materials neglected to mention the other outcome measures or the clincialtrials.gov registration lists outcome measures that weren't actually collected. Given that the ClinicalTrials registration was completed <i>after</i> the paper was submitted, that implies that other outcome measures were collected as part of the study but not reported. If so, the PLoS supplemental materials are inaccurate.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Final post-test or interim stage post-test? </span></b></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The clinicaltrials.gov registration states that outcome measures will be tested before training, after 12-weeks, and again after 24 weeks. The paper reports only the pre-training and 12-week outcomes and does not mention the 24-week test. Was it conducted? Is this paper an interim report? If so, that should be mentioned in the article. Had the results not been significant at 12 weeks, would they have been submitted for publication? Probably not. And, if not, that could be construed as selective reporting, again biasing the reported p-values in this paper in favor of significance. </span></div>
</div>
<div style="font-weight: normal; text-align: start;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
</span><br />
<div>
<b><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Limitations of the limitations section</span></b></div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">The paper ends with a limitations section, but the only identified limitations are the small sample size, the lack of a test of real-world outcome measure, the use of only the games in Lumosity, and the lack of evidence for maintenance of the training benefits (presumably foreshadowing a future paper based on the 24-week outcome testing mentioned in the clinicaltrials.gov registration). No mention is made of the inadequacy of the control group for causal claims about the benefits of game training, the fragility of the results to correction for multiple testing, the flexibility of analysis, the possible presence of unreported outcome measures, or any of the other issues I noted above.</span></div>
<div style="font-weight: normal; text-align: start;">
</div>
</h3>
<h3 style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></h3>
<div style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;"><b>Summary</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;"><b><br /></b></span></div>
</div>
<div>
<span style="font-size: large;"><span style="font-family: Arial, Helvetica, sans-serif;">Brain training is now a major industry, and companies capitalize (literally) on results that seem to support their claims. </span><span style="font-family: Arial, Helvetica, sans-serif;">Training and intervention studies are critical if we want to evaluate the effectiveness of psychological interventions. But,</span><span style="font-family: Arial, Helvetica, sans-serif;"> intervention studies must include an adequate active control group, one that is matched for expected improvements independently for each outcome measure (to control for differential placebo effects). Without such a control condition, causal claims that a treatment has benefits are inappropriate because it is impossible to distinguish effects of the training task from other differences between the training and control group that could also lead to differential improvement. </span><span style="font-family: Arial, Helvetica, sans-serif;">Far too many published papers make causal claims with inadequate designs, incomplete reporting of outcome measures, and overly flexible analyses.</span><span style="font-family: Arial, Helvetica, sans-serif;"> </span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
In this case, the inadequacy of the limited-contact control condition (without acknowledging these limitations) alone would be sufficient grounds for an editor to reject this paper. Reviewers and editors need to step up and begin requiring adequate designs whenever authors make causal claims about brain training. Even those with an adequate design should take care to qualify any causal claims appropriately to avoid misrepresentation in the media.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br />
Even if the control condition in this study had been adequate (it wasn't), the critical interaction testing the difference in improvement across conditions was not reported. Moreover, one of the improvements in the training group was reported to be significant even though it wasn't, and neither of the improvements would have withstood correction for multiple tests. Finally, the apparent underreporting of outcome measures makes all of the significance tests suspect.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">More broadly, this paper provides an excellent example of why the field needs true pre-registration of cognitive intervention studies. Such registrations should include more than just the labels for the outcome measures. They should include a complete description of the protocol, tasks, measures, coding, and planned analysis. They should specify any arbitrary cutoffs, identify which analyses are confirmatory, and note when additional analyses will be exploratory. Without pre-registration (or, in the absence if pre-registration, complete reporting of all outcome measures), readers have no way to evaluate the results of an intervention because any statistical tests are effectively uninterpretable. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<br />
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Note: Updated to fix formatting and typos</b></span></div>
</div>
<div>
<div style="text-align: center;">
<br /></div>
</div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-63108851261994177772014-02-21T11:36:00.004-06:002020-12-21T08:07:27.556-06:00How experts recall chess positions<i style="background-color: white; font-family: Georgia, 'Times New Roman', serif;">Originally posted to <a href="http://www.theinvisiblegorilla.com/blog">invisiblegorilla blog</a> on 15 February 2012. </i><i style="background-color: white; font-family: Georgia, 'Times New Roman', serif;">I am consolidating some posts from my other blogs onto my <a href="http://blog.dansimons.com/">personal website</a> where I have been blogging for the past year.</i><br />
<span style="font-family: Georgia, Times New Roman, serif;"><i style="background-color: white;"><br /></i></span>
<br />
<div>
<span style="font-family: Georgia, Times New Roman, serif;"><span style="background-color: white;">In 2011, a computer (</span><a href="http://goo.gl/2X8W6" style="background-color: white; color: #000055; font-weight: bold; text-decoration: none;">Watson</a><span style="background-color: white;">) outplayed two human Jeopardy champions. In 1997, chess computer </span><a href="http://goo.gl/IQTfL" style="background-color: white; color: #000055; font-weight: bold; text-decoration: none;">Deep Blue</a><span style="background-color: white;"> defeated chess champion </span><a href="http://goo.gl/vQUuv" style="background-color: white; color: #000055; font-weight: bold; text-decoration: none;">Garry Kasparov</a><span style="background-color: white;">. In both cases, the computer “solved” the game—found the right questions or good moves—differently than humans do. Defeating humans in these domains took years of research and programming by teams of engineers, but only with huge advantages in speed, efficiency, memory, and precision could computers compete with much more limited humans.</span></span></div>
<div>
<div class="post-bodycopy clearfix" style="background-color: white; min-width: 0px;">
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">What allows human experts to match wits with custom-designed computers equipped with tremendous processing power? Chess players have a limited ability to evaluate all of the possible moves, the responses to those moves, the responses to the responses, etc. Even if they could evaluate all of the possible alternatives several moves deep, they still would need to remember which moves they had evaluated, which ones led to the best outcomes, and so on. Computers expend no effort remembering possibilities that they had already rejected or revisiting options that proved unfruitful.</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">This question, how do chess experts evaluate positions to find the best move, has been studied for decades, dating back to the groundbreaking work of <a href="http://goo.gl/IOEMm" style="color: #000055; font-weight: bold; text-decoration: none;">Adriaan de Groot</a> and later to work by William Chase and <a href="http://goo.gl/Mqs3D" style="color: #000055; font-weight: bold; text-decoration: none;">Herbert Simon</a>. de Groot interviewed several chess players as they evaluated positions, and he argued that experts and weaker players tended to “look” about the same number of moves ahead and to evaluate similar numbers of moves with roughly similar speed. The relatively small differences between experts and novices suggested that their advantages came not from brute force calculation ability but from something else: knowledge. According to De Groot, the core of chess expertise is the ability to recognize huge number of chess positions (or parts of positions) and to derive moves from them. In short, their greater efficiency came not from evaluating more outcomes, but from considering only the better options. <em>[Note: Some of the details of de Groot’s claims, which he made before the appropriate statistical tests were in widespread use, did not hold up to later scrutiny—experts do consider somewhat more options, look a bit deeper, and process positions faster than less expert players (Holding, 1992). But de Groot was right about the limited nature of expert search and the importance of knowledge and pattern recognition in expert performance.]</em></span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">In de Groot’s most famous demonstration, he showed several players images of chess positions for a few seconds and asked the players to reconstruct the positions from memory. The experts made relatively few mistakes even though they had seen the position only briefly. Years later, Chase and Simon replicated de Groot’s finding with another expert (a master-level player) as well as an amateur and a novice. They also added a critical control: The players viewed both real chess positions and scrambled chess positions (that included pieces in implausible and even impossible locations). The expert excelled with the real positions, but performed no better than the amateur and novice for the scrambled positions (later studies showed that experts can perform slightly better than novices for random positions too if given enough time; Gobet & Simon, 1996). The expert advantage apparently comes from familiarity with real chess positions, something that allows more efficient encoding or retrieval of the positions.</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">Chase and Simon recorded their expert performing the chess reconstruction task, and found that he placed the pieces on the board in spatially contiguous chunks, with pauses of a couple seconds after he reproduced each chunk. This finding has become part of the canon of cognitive psychology: People can increase their working memory capacity by grouping together otherwise discrete items to form a larger unit in memory. In that way, we can encode more information into the same limited number of memory slots.</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">In 1998, Chris Chabris and I invited two-time US Champion and International Grandmaster Patrick Wolff (a friend of Chris’s) to the lab and asked him to do the chess position reconstruction task. Wolff viewed each position (on a printed index card) for five seconds and then immediately reconstructed it on a chess board. After he was satisfied with his work, we gave him the next card. At the end of the study, after he had recalled five real positions and five scrambled positions, we asked him to describe how he did the task.</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">The video below shows his performance and his explanations (Chris is the one handing him the cards and holding the stopwatch—I was behind the camera). Like other experts who have been tested, Wolff rarely made mistakes in reconstructing positions, and when he did, the errors were trivial—they did not alter the fundamental meaning or structure of the position. Watch for the interesting comments at the end when Wolff describes why he was focused on some aspects of a position but not others.</span><br />
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/rWuJqCwfjjc?feature=player_embedded' frameborder='0'></iframe></div>
<span style="font-family: Georgia, Times New Roman, serif;"><br /></span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif; font-style: italic;">HT to Chris Chabris for comments on a draft of this post</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;"><strong>Sources cited</strong>:</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">For an extended discussion of chess expertise and the nature of expert memory, see Christopher Chabris’s dissertation: Chabris, C. F. (1999). Cognitive and neuropsychological mechanisms of expertise: Studies with chess masters. Doctoral Dissertation, Harvard University. http://en.scientificcommons.org/43254650</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">Chase, W. G., & Simon, H. A. (1973). Perception in chess. <em>Cognitive Psychology</em>,<em>4</em>, 55-81.</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">de Groot, A.D. (1946). <em>Het denken van de schaker.</em> [The thought of the chess player.] Amsterdam: North-Holland. (Updated translation published as <em>Thought and choice in chess</em>, Mouton, The Hague, 1965; corrected second edition published in 1978.)</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">Holding, D.H. (1992). Theories of chess skill.<em> Psychological Research</em>,<em> 54(1)</em>, 10–16.</span></div>
<div style="margin-bottom: 1em; margin-top: 1em; padding: 0px;">
<span style="font-family: Georgia, Times New Roman, serif;">Gobet, F., & Simon, H.A. (1996a). Recall of rapidly presented random chess positions is a function of skill. <em>Psychonomic Bulletin & Review, 3(2),</em> 159–163.</span></div>
</div>
</div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-57138359462975055582014-02-14T13:37:00.002-06:002020-12-21T09:13:45.209-06:00HI-BAR: 10 questions about Inattentional blindness, race, and interpersonal goals<h2>
<div style="text-align: center;">
<b style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;">HI-BAR (Had I Been A Reviewer)</span></b></div>
<div style="text-align: center;">
<b style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><br /></span></b></div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: small;"><b><div style="text-align: center;">
<b>A post-publication review of Brown-Iannuzzi et al (2014). The invisible man: Interpersonal goals moderate inattentional blindness to African Americans. <i>Journal of Experimental Psychology: General, 143</i>, 33-37. [<a href="http://www.ncbi.nlm.nih.gov/pubmed/23294347">pubmed</a>] [<a href="http://www.apa.org/pubs/journals/features/xge-a0031407.pdf">full paper</a>]</b></div>
</b></span></h2>
<div style="text-align: center;">
<i style="font-family: Arial, Helvetica, sans-serif;">For more information about HI-BAR reviews, see this earlier <a href="http://blog.dansimons.com/2013/09/hi-bar-had-i-been-reviewer.html">post</a>.</i></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">In a <a href="http://www.apa.org/pubs/journals/features/xge-a0031407.pdf">paper</a> published this week in the <i><a href="http://www.apa.org/pubs/journals/xge/">Journal of Experimental Psychology:General</a></i>, Brown-Iannuzzi and colleagues reported a study in which participants (White women) first were asked to think about their interpersonal goals and then completed an inattentional blindness task in which the unexpected event was either the appearance of a White man or a Black man. For these participants, their idealized interpersonal goals presumably included same-race romantic partners or friends, so the prediction was that priming participants to think about these idealized interpersonal goals would make them less likely to notice an unexpected Black "interloper" than a White one in a basketball counting task similar to our earlier <a href="http://www.youtube.com/watch?v=vJG698U2Mvo">selective attention task</a>. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">This approach is interesting and potentially important for several reasons that have nothing to do with race or interpersonal goals. Most studies showing variability in noticing rates as a function of expectations manipulate expectations by varying the task itself (e.g., counting the white shapes rather than the black ones, attending to shape rather than color. See Most et al, <a href="http://www.ncbi.nlm.nih.gov/pubmed/11294235">2001</a>, <a href="http://www.ncbi.nlm.nih.gov/pubmed/15631594">2005</a>). In this study, Brown-Iannuzzi and colleagues manipulated expectations not by changing the task instructions, but by priming people using an entirely unrelated task. In effect, their priming task was designed to get people to envision a White person without calling attention to race and then used that more activated concept to induce a change in noticing rates as a function of race. </span><span style="font-family: Arial, Helvetica, sans-serif;">If this approach proves robust, it could change how we think about the detection of unexpected objects because it implies that an attention set could be induced (subtly) in a powerful way without actually changing the primary task people are doing. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I don't really have any specific comments about the use of race in particular or the use of interpersonal goals to accomplish this form of priming, but given the broader theoretical importance of this claim, I do have a number of questions about the methods and results in this paper. Most of my questions arise from the relatively sparse reporting of methods and results details in the paper, so I hope that they can be addressed if the authors provide more information. I am concerned that the evidence for the core conclusions is somewhat shaky given the fairly small samples and the flexibility of the analysis choices. Given the potential importance of this claim, I would like to see the finding directly replicated with a pre-registered analysis plan and a larger sample to verify that the effects are robust. </span><br />
<br />
<h3>
<b style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 1.083em;">10 Questions and Comments </b></h3>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">1) The method section provides almost no information about the test videos. What did the test video look like? How long did it last? </span></span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;">Were all the players in the video White? </span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;">How many passes did each team make? Did the two teams differ in the number of passes they made? Did the two videos differ in any way other than the race of the unexpected person? What color clothes did the players wear? How were the videos presented online to MTurk participants? (i.e., were they presented in Flash or some other media format?) Was there any check that the videos played smoothly on the platform on which they were viewed? Were there any checks on whether participants followed instructions carefully? That can be an issue with MTurk samples, but the only check on attentiveness appears to be the accuracy of counting. Was there any check to make sure people actually did think about their interpersonal goals? No subjects appear to have been excluded due to the failure to do. How long did the study take to complete? All of these factors potentially affect the results, so it's hard to evaluate the study without this information.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">2) Were the players in the video wearing black shirts and white shirts as in our </span><a href="http://www.youtube.com/watch?v=vJG698U2Mvo" style="line-height: 17.327999114990234px;">original video</a><span style="line-height: 17.327999114990234px;">? If so, which team's passes did people count? Was that counter-balanced, and if so, did it matter? If the players were wearing white/black shirts, the finding that the shirt color worn by the unexpected person didn't matter is really surprising (and somewhat odd given Most et al's findings that similarity to the attended items matters). The task itself induces an attention set by virtue of the demand to focus selectively on one group and not the other, and it would be surprising if the subtle race priming yielded a more potent attention set than the task requirements. We know that the attention demands of the task (what's attended, what's ignored) affect noticing based on the similarity of the unexpected item to the attended and ignored items. That's a <a href="http://www.ncbi.nlm.nih.gov/pubmed/11294235">pretty big effect</a>. Why wouldn't it operate here too? Shouldn't we also expect some interaction between the priming effect and the color of the attended team.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">3) The analyzed sample consisted of 209 MTurk subjects. I have no objection to using MTurk for this sort of study. But, the method section doesn't report enough details about the assignment to conditions to evaluate the nature of these effects. For example, how many participants were in each of the conditions? It appears that the sample was divided across 5 (personal closeness) x 2 (race) x 2 (shirt color) combinations of conditions, for a total of 20 conditions. If so, there were approximately 10 subjects/condition. Did half of the participants attend to each team in the video? If so, that would mean there were approximately 5 subjects/combination of conditions. Depending on how many factors there were in the design, the sample size in each condition becomes pretty small. And, that's important because the core conclusion rests largely on the effect in just one of the priming conditions.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">4) The paper does not report the number of participants in each condition or the number of those who noticed (it just reports percentages in a figure, but the figure does not break down the results across each of the combinations of conditions). Perhaps the authors could provide a table with that information as part of supplemental materials? Although the paper reports no effect of factors like shirt color, it's quite possible that such factors interacted with the other ones, but there probably is not adequate power with these sample sizes to test such effects. Still, it would be good to know the exact N in each combination of conditions, along with the number of participants who noticed in each condition.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">5) Was there a reason why missing the total count by 4 was used the cutoff for accurate performance? That might well be a reasonable cutoff (it led to 35 exclusions out of the original 244 participants), but the paper doesn't report the total number of passes in the video, so we don't know how big an error that actually is.</span></span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;"> The analysis excluded participants who were inaccurate (by 4 passes), and footnote 3 reports that the simple comparisons were weaker if those participants were included. Does that mean that the effect in the Friend condition was not significant with these subjects included? Did that effect depend on this particular cutoff for accuracy? What if the study used a cutoff of 3 missed passes rather than 4? Would that change the outcome? How about 5? or 2? What if the study also excluded people familiar with the original basketball-counting video? Would it be reliable then? The flexibility of these sorts of analysis decisions are one reason I strongly favor pre-registered analysis plans for confirmatory studies.</span><br />
<div>
<br /></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">6) It seems problematic to include the 23% of participants who were familiar with inattentional blindness in the analyses, especially if they were familiar with one of the variants in which people count ball passes by one of two teams of players. Although one of my own papers was cited as justification for including rather than excluding these participants, I didn't understand the reasoning. It's true that people can fail to notice an unexpected event even if they are familiar with the phenomenon of inattentional blindness more generally, but that typically depends on them not making the connection between the current test and the previous one. That is, they shouldn't have reason to expect an unexpected event in this particular task/scenario (e.g., people familiar with the "gorilla" video might still fail to detect an unexpected target in Mack & Rock's original paradigm because the two tasks are so different that they have no reason to expect one). When I show <a href="http://www.youtube.com/watch?v=IGQmdoK_ZfY">another variant</a> of the gorilla task to people who are already familiar with the original gorilla video, they do notice the gorilla (<a href="http://i-perception.perceptionweb.com/fulltext/i01/i0386.pdf">Simons, 2010</a>). They expect an unexpected interloper and look for it. The same likely is true here. The method section reports that excluding these 23% of the participants did not matter for the interaction, but the analyses with those data excluded are not reported. And, given that excluding those subjects would reduce the sample size by 23% and that the critical simple comparison was p=.03 (see below), it seems likely that the exclusion would have some consequences for the statistical significance of the critical comparisons underlying the conclusions. Perhaps the authors could report these analyses more fully in supplemental materials.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;"><br /></span></span>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">7) It is not appropriate to draw strong conclusions from the difference in noticing rates in the control condition for the White and Black unexpected person. The paper suggests that the difference in the no-prime control group results from racial stereotyping: White subjects view a Black person as a greater threat, so he garners more attention and is noticed more by default. But, this comparison is of two different videos that presumably were not identical in every respect other than the race of the unexpected person. It's quite possible that the actor just stood out more in that video due to the choreography of the players. Or, that particular Black actor might have stood out more due to features having nothing to do with his race (e.g., maybe he was taller, moved with a different gait, etc.). It's risky to draw strong comparisons about baseline differences in noticing rates across videos because many things could differ between any two videos like this. It's not justified to assume that any baseline differences must have been due to race, especially with just one example of an unexpected person of each Race. </span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">8) If the control condition is treated as a default for noticing against which the priming conditions are compared (really, the appropriate way to do it given that direct comparison of different stimuli is questionable), it would be nice to have a more robust estimate of that baseline condition with a substantially larger sample. Otherwise, the noisy measurement of the baseline could artificially inflate the estimates of interaction effects. In the analyses, though, the baseline is treated just like the other priming conditions. If anything, the effects might be larger if the baseline were subtracted from each of the other conditions, but I would be hesitant to do that without first increasing the precision of the baseline estimate. </span></span></div>
<div>
<br /></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;">9) The primary conclusions are driven by the condition in which subjects were primed by thinking about an idealized Friend (the interaction effects are much harder to interpret because the priming conditions really are categories rather than an equally-spaced continuum, and even if they were a continuum, the trends are not consistent with one). Ideally, these simple effect analyses would have been relative to a baseline condition to account for default differences in noticing rates. Still, with the direct comparison, the Friend priming condition was the only one to yield significantly greater detection of the White unexpected person (p=.03). The effect for the Romantic Partner condition was not statistically significant at .05, nor were the neighbor or co-worker conditions. I don't see any a-priori reason to expect an effect just for the Friend condition, and with corrections for multiple tests, that effect would not have been significant either. </span></span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;">This amounts to a concern about analysis flexibility: T</span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;">he authors could have drawn the same conclusion had there been a difference in the Romantic Partner condition but not the Friend condition. It might even be possible to explain an effect of priming in the more remote interpersonal relationship conditions or in some combination of them. Correcting for multiple tests helps to address this issue, but I would prefer pre-registration of the specific predictions for any confirmatory hypothesis test. Then, any additional analyses could be treated as exploratory. With this level of flexibility, correction for these alternative outcomes seems necessary when interpreting the statistical significance of the findings.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;">10) Without correcting for multiple tests, the paper effectively treats the secondary analyses as confirmatory hypothesis tests, but any of a number of other outcomes could have also been taken as support for the same hypothesis. Given the relatively small effect in the one condition that was statistically significant (although, again, see my notes about using a baseline), I worry that the effect might not be robust. </span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;"> Would the effect in that condition no longer be significant if one or two people who noticed were shifted to the miss category? My guess is that the significance in the one critical condition hinges on as little as that. </span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17.327999114990234px;">A confirmatory replication would be helpful to verify that this effect is reliable. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17.327999114990234px;"><br /></span></span>
<h3>
<span style="font-family: Arial, Helvetica, sans-serif;">Conclusions</span></h3>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">This study provides suggestive evidence for a new way to moderate noticing of unexpected events. If true, it would have substantial theoretical and practical importance. However, the missing method details make the finding hard to evaluate. And, the flexibility of the analysis choices coupled with a critical finding that only reaches statistical significance without correction for that flexibility make me worried about the robustness of the result. Fortunately, the first of these issues would be easy to address by adding additional information to the supplemental materials. I hope the authors will do that so that readers can more fully evaluate the study. They could also provide more information about the results and the consequences of various analytical decisions. Given the importance of the finding, I hope they will build on this finding by conducting a pre-registered, highly powered, confirmatory test of the effect.</span></div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-7373747922942521892014-01-25T10:21:00.004-06:002020-12-21T09:13:56.423-06:00Replication, Retraction, and Responsibility<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 18.200000762939453px;">Congrats/thanks to Brent Donnellan, Joe Cesario, and Rich Lucas for their tenacity and perseverance. They conducted 9 studies with more than 3000 participants in order to publish a set of direct replications. Their paper challenged an original report (study 1 in <a href="http://psycnet.apa.org/journals/emo/12/1/154/">Bargh & Shalev, 2012</a>) claiming that loneliness is correlated with preferred shower temperatures. The new, just-accepted paper did not find a reliable correlation. </span></span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">Donnellan describes their findings and the original studies in what may be the most measured and </span><a href="http://traitstate.wordpress.com/2014/01/24/warm-water-and-loneliness/" style="font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">understated blog post</a><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;"> I've seen. You should read it.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br style="background-color: white; line-height: 18.200000762939453px;" /><span style="background-color: white; line-height: 18.200000762939453px;">The original study had fewer than 100 subjects (51 from a Yale undergraduate sample and a replication with 41 from a community sample), underpowered to detect a typical effect size in a social psychology experiment. But there are bigger problems with the original results.</span></span><br />
<span style="background-color: white; line-height: 18.200000762939453px;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></span><span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 18.200000762939453px;">According to the description in Donnellan's post, the data from the Yale sample were completely screwy: 46/51 Yale students reported taking fewer than 1 shower/bath per week!</span><span style="background-color: white; line-height: 18.200000762939453px;"> Either Yale students are filthy, or something's wrong with the data. More critical for the primary question, 42/51 Yale students apparently prefer cold (24 students) or lukewarm (18 students) showers. How many people do you know who prefer cold showers to reasonably hot ones? Again, something's out of whack. In a comment on Donnellan's blog post, Rich Lucas noted that the original distribution of preferred temperatures would mirror what Donnellan et al found if the original data were inadvertently reverse coded. Of course, that would mean the correlation reported in the paper was backwards, and the effect was the opposite of what was claimed.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 18.200000762939453px;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;">From an <a href="http://traitstate.wordpress.com/2012/09/20/whats-the-first-rule-about-john-barghs-data/">earlier Donnellan post</a>, we know that Bargh was aware of these issues back in 2012, but that he prevented </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">Donnellan and his colleagues from discussing the problems until recently (you should read that post too)</span><span style="font-family: Arial, Helvetica, sans-serif;">. </span><span style="font-family: Arial, Helvetica, sans-serif;">In a semi-private email discussion among priming researchers and skeptics, </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">Bargh </span><span style="font-family: Arial, Helvetica, sans-serif;">claimed that his prohibition on discussing his data was just a miscommunication, but he didn't get around to correcting that misconception until he was pressed to respond on that email thread. In the same thread, </span><span style="font-family: Arial, Helvetica, sans-serif;">Bargh claimed to have first learned of these errors from Joe Cesario (who initially requested the original data). Although it's odd that he didn't notice the weird distribution in the frequency responses, I can understand how someone might miss something obvious when they were focusing attention elsewhere... Bargh said that he provided an explanation to the editor at Emotion during the review process: He claimed that Yale students misunderstood the </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">bathing frequency item as asking specifically about baths (not showers). According to Joe Cesiaro's response in that same email thread, though, that doesn't accord with the survey wording about showers/baths that Bargh provided.</span><br />
<span style="background-color: white; font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 18.200000762939453px;"><br /></span></span>
<span style="background-color: white; font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 18.200000762939453px;">Still, whenever and however Bargh learned of the problems with the data, he and Shalev</span></span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;"> had an obligation to retract the original study and issue an erratum (unless they actually believe Yale students prefer rare, cold showers). Even if the subjects misinterpreted the frequency question, the results are bogus. The problems could well have resulted from an honest oversight, a slip up in coding, a misinterpretation of a poorly worded question, or an Excel copy/paste error. Regardless of the reason, authors have a responsibility to own up to mistakes and to correct the record. Posting to a semi-private email list is not sufficient—the public record in the journal must be amended. </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">Authors have an obligation to correct mistakes once they know of them, and the failure to do so in the published record is troubling.</span><br />
<span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;"><br /></span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">Note that I am not arguing the original study should be retracted just because Donnellan and colleagues didn't replicate it. A failure to replicate is not adequate grounds for questioning the integrity of the original finding. The original effect size estimates could be wrong, but that's just science working properly to correct itself (that's why direct replications are useful and important). Yet, obviously flawed data like those described by Donnellan should not have to await replication, and scholars reading the literature should be informed that they should not place any stock in that first study with Yale students. That finding should be withdrawn until it can be verified so that it doesn't mislead the field.</span><br />
<span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;"><br /></span><span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 18.200000762939453px;">One thing I find troubling about this story is that Donnellan, Cesario, and Lucas needed to conduct 9 studies with more than 30x the original number of participants in order to get this paper accepted at <i>Emotion</i>. They should be applauded for replicating with enough power to be sure that their effect size estimates are precise, but <i>each</i> of their studies had more than 2.5x the sample size of the original! If their efforts were entirely voluntary and not a consequence of appeasing reviewers, kudos to them for making sure they got it right. </span></span></span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">I'm glad that this paper was accepted, and our field owes them gratitude for their efforts. I just hope they haven't set an overly high standard and precedent for what's needed to publish a direct replication.</span><br />
<br />
<span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">I </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">would encourage Bargh to issue a public explanation (accessible to the whole field, not just an email thread) for the data issues in their original study. The problems could well have been an accidental coding or interpretation problem, and mistakes are excusable even if they do undermine the claims. </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">More importantly, he should retract the original study (not the whole paper, necessarily -- just the study with problematic data) and issue an erratum in the journal. Out of curiosity, I would like to see an explanation for why the study was not retracted immediately upon learning of the problems more than a year ago.</span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;"> </span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18.200000762939453px;">Perhaps there is a good reason, but I'm having trouble generating one. I hope he will enlighten us. </span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-52906750755545606342013-10-27T10:37:00.003-05:002020-12-21T09:14:13.126-06:00New Posts on IonPsych - October 27, 2013<i><span style="font-family: Arial, Helvetica, sans-serif;">This fall, I am teaching a graduate seminar on speaking/writing for general audiences. As part of the class, students blog at <a href="http://www.ionpsych.com/">www.ionpsych.com</a>. </span><span style="font-family: Arial, Helvetica, sans-serif;">Each week, I'll provide a short summary of the latest posts.</span></i><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>The latest posts on <a href="http://www.ionpsych.com/">www.ionpsych.com</a>: </b></span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Anna Popova</b> <a href="http://www.ionpsych.com/2013/10/in-search-for-perfect-match-or-why-bad.html">discusses</a> the value of negative experiences and imagination in decision making.</span><br />
<div>
<br /></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Jim Monti</b> <a href="http://www.ionpsych.com/2013/10/seven-across-exercise.html">describes</a> the best way to stave off the cognitive costs of aging.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Luis Flores</b> <a href="http://www.ionpsych.com/2013/10/do-people-with-depression-enjoy.html">describes</a> a recent study of depression and argues that people with depression may enjoy activities just as much as those without depression, but they are less willing to work to experience those activities.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Christina Tworek</b> <a href="http://www.ionpsych.com/2013/10/are-you-there-society-its-me-science.html">muses</a> on the causes of the large disparity between what scientists know and what the public knows of science.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Emily Hankosky</b> <a href="http://www.ionpsych.com/2013/10/shaming-addicts-clean.html">critiques</a> claims in a recent book espousing personal responsibility for overcoming addiction. She makes a case that discounting neurobiological bases of addiction is irresponsible.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Lindsey Hammerslag</b> <a href="http://www.ionpsych.com/2013/10/dogma-sciences-greatest-enemy.html">discusses</a> the importance of teaching about certainty and uncertainty in science.</span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-25251995936797446902013-10-13T10:18:00.003-05:002020-12-21T09:14:27.329-06:00New Posts on IonPsych - 13 October 2013<i><span style="font-family: Arial, Helvetica, sans-serif;">This fall, I am teaching a graduate seminar on speaking/writing for general audiences. As part of the class, students blog at <a href="http://www.ionpsych.com/">www.ionpsych.com</a>. </span><span style="font-family: Arial, Helvetica, sans-serif;">Each week, I'll provide a short summary of the latest posts.</span></i><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>The latest posts on <a href="http://www.ionpsych.com/">www.ionpsych.com</a>: </b></span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>H. A. Logis</b> <a href="http://www.ionpsych.com/2013/10/beyond-company-that-you-keep.html">explores</a> how we mold our behaviors to the people around us.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Brian Metzger</b> <a href="http://www.ionpsych.com/2013/10/when-brain-chooses-without-us.html">demonstrates</a> how perception often isn't subject to free will.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Emily Kim</b> <a href="http://www.ionpsych.com/2013/10/find-your-happy-career-in-social.html">describes</a> alternatives to academia for social psychologists.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Joachim Operskalski</b> <a href="http://www.ionpsych.com/2013/10/why-neuroscience-should-not-end-prozac.html">evaluates</a> how new developments in neuroscience might improve treatment of psychiatric disorders and why we shouldn't be so quick to dismiss Prozac. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Anna Popova</b> <a href="http://www.ionpsych.com/2013/10/nobody-has-exciting-methodology-section.html">explains</a> how to make statistics interesting.</span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-75078281185797963492013-10-06T09:27:00.004-05:002020-12-21T09:14:34.589-06:00New Posts on www.ionpsych.com (Oct 6, 2013)<i><span style="font-family: Arial, Helvetica, sans-serif;">This fall, I am teaching a graduate seminar on speaking/writing for general audiences. As part of the class, students blog at <a href="http://www.ionpsych.com/">www.ionpsych.com</a>. </span><span style="font-family: Arial, Helvetica, sans-serif;">Each week, I'll provide a short summary of the latest posts.</span></i><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>The latest posts on <a href="http://www.ionpsych.com/">www.ionpsych.com</a>: </b></span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Jim Monti</b> <a href="http://www.ionpsych.com/2013/10/what-salmon-and-your-brain-have-in.html">discusses</a> why Omega-3 fatty acids are important for your brain, and why diet matters.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Aldis Sipolins</b> <a href="http://www.ionpsych.com/2013/10/we-can-do-better-than-random-assignment.html">presents</a> an alternative to random assignment when dealing with small samples.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Carolyn Hughes</b> <a href="http://www.ionpsych.com/2013/10/baby-when-you-hurt-i-hurt-yeah-right.html">considers</a> whether you really experience pain when you see someone else hurting. </span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-60868897001982984692013-09-30T14:48:00.001-05:002020-12-21T09:14:42.316-06:00New Posts on IonPsych<i><span style="font-family: Arial, Helvetica, sans-serif;">This fall, I am teaching a graduate seminar on speaking/writing for general audiences. As part of the class, students blog at <a href="http://www.ionpsych.com/">www.ionpsych.com</a>. </span><span style="font-family: Arial, Helvetica, sans-serif;">Each week, I'll provide a short summary of the latest posts.</span></i><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>The latest posts on <a href="http://www.ionpsych.com/">www.ionpsych.com</a>: </b></span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Carolyn Hughes</b> <a href="http://www.ionpsych.com/2013/09/first-impressions.html">discusses</a> the neural basis of first impressions. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Luis Flores</b> <a href="http://www.ionpsych.com/2013/09/is-it-informative-whether-worries-are.html">addresses</a> the difference between worrying and generalized anxiety disorder</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Lindsey Hammerslag</b> <a href="http://www.ionpsych.com/2013/09/sexism-in-science-we-need-to-deal-with.html">considers</a> whether scientists are sexist for failing to deal with monkey cramps. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>H. A. Logis</b> <a href="http://www.ionpsych.com/2013/09/does-bullying-kill.html">revisits</a> claims that bullying kills.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;"><b>Christina Tworek</b> <a href="http://www.ionpsych.com/2013/09/boys-wearing-nail-polish-and-other-sins_25.html">examines</a> why some people have a problem with boys wearing nail polish.</span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-85502486896912053432013-09-23T08:45:00.004-05:002020-12-21T09:14:49.946-06:00New posts on www.ionpsych.com<i><span style="font-family: Arial, Helvetica, sans-serif;">This fall, I am teaching a graduate seminar on speaking/writing for general audiences. As part of the class, students blog at <a href="http://www.ionpsych.com/">www.ionpsych.com</a>. </span><span style="font-family: Arial, Helvetica, sans-serif;">Each week, I'll provide a short summary of the latest posts.</span></i><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>The latest posts on <a href="http://www.ionpsych.com/">www.ionpsych.com</a>: </b></span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Anna Popova</b> <a href="http://www.ionpsych.com/2013/09/choices-choices-decisions-decisions.html">explains</a> why individual and group decisions are treated differently by researchers studying decision making (and why they shouldn't be).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Jim Monti</b> <a href="http://www.ionpsych.com/2013/09/eating-your-way-to-alzheimers.html">examines</a> how your diet might affect your risk of Alzheimer's disease.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Aldis Sipolins</b> <a href="http://www.ionpsych.com/2013/09/this-is-your-brain-on-direct-current.html">covers</a> how you might improve your video game playing (and learning in general) by applying direct current to your head. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Emily Hankosky</b> <a href="http://www.ionpsych.com/2013/09/the-future-of-reproduction-same-sex.html">describes</a> a breakthrough study that might change how people have children in the not-too-distant future.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Judy Chiu</b> <a href="http://www.ionpsych.com/2013/09/stressed-why-you-dont-need-to-sweat.html">argues</a> that stress isn't inherently bad for you - what matters is how you think about stress.</span>Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-44533990485454776312013-09-05T16:20:00.004-05:002020-12-21T09:14:57.359-06:0019 questions about video games, multitasking, and aging (a HI-BAR commentary on a new Nature paper)<h2>
<div style="text-align: center;">
<b style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;">HI-BAR (Had I Been A Reviewer)</span></b></div>
<div style="text-align: center;">
<b style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><br /></span></b></div>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: small;"><b><div style="text-align: center;">
<b>A post-publication review of Anguera et al (2013). Video game training enhances cognitive control in older adults. <i>Nature</i>, 501, 97-101.</b></div>
</b></span></h2>
<div style="text-align: center;">
<i style="font-family: Arial, Helvetica, sans-serif;">For more information about HI-BAR reviews, see my <a href="http://blog.dansimons.com/2013/09/hi-bar-had-i-been-reviewer.html">post</a> from earlier today.</i></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">In a paper published this week in <i>Nature, </i>Anguera et al reported a study in which older adults were trained on a driving video game for 12 hours. Approximately 1/3 of the participants engaged in multitasking training (both driving and detecting signs), another 1/3 did the driving or sign tasks separately without having to do both at once, and the final 1/3 was a no-contact control. The key findings in the paper: </span><br />
<br />
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">After multitasking training, the seniors attained "levels beyond those achieved by untrained 20-year-old participants, with gains persisting for 6 months"</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Multitasking training "resulted in performance benefits that extended to untrained cognitive control abilities"</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Neural measures of midline-frontal theta power and frontal-parietal theta coherence correlated with these improvements</span></li>
</ul>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 17.3125px;">This is one of many recent papers touting the power of video games to improve cognition, published in a top journal, that receives glowing (almost breathless) media coverage: </span>The <a href="http://www.nytimes.com/2013/09/05/technology/a-multitasking-video-game-makes-old-brains-act-younger.html" style="line-height: 17.3125px;">NY Times</a> reports<span style="background-color: white; line-height: 17.3125px;"> "A Multitasking Video Game Makes Old Brains Act Younger." A story in </span><a href="http://www.nature.com/news/gaming-improves-multitasking-skills-1.13674" style="line-height: 17.3125px;">Nature</a><span style="background-color: white; line-height: 17.3125px;"> in nature claims "</span><span style="background-color: white; line-height: 1.083em;">Gaming Improves Multitasking Skills." The <a href="http://www.theatlantic.com/health/archive/2013/09/how-to-rebuild-an-attention-span/279326/">Atlantic</a> titles their story, "</span><span style="line-height: 17.3125px;">How to Rebuild an Attention Span." </span><span style="background-color: white; line-height: 17.3125px;">(Here's </span>one <a href="http://bps-research-digest.blogspot.com/2013/09/driving-video-game-reverses-age-related.html">exception</a> that notes a few limitations). </span></div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 1.083em;">In several media quotes, the senior author on the paper (Gazzaley) admirably cautions against over-hyping of these findings (e.g., "</span>Video games shouldn’t now be seen as a guaranteed panacea" in the <i>Nature</i> story)<span style="background-color: white; line-height: 1.083em;">. Yet overhyping is exactly what we have in the media coverage (and a bit in the paper as well).</span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 1.083em;"><br /></span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 1.083em;">The research is not bad. It's a reasonable, publishable first study that requires a bit more qualification and more limited conclusions: Some of the strongest claims are not justified, the methods and findings have limitations, and none of those are shortcomings are acknowledged or addressed.</span><span style="background-color: white; line-height: 17.3125px;"> If you are a regular reader of this blog, you're familiar with the <a href="http://blog.dansimons.com/2013/07/pop-quiz-what-can-we-learn-from.html">problems that plague most such studies</a>.</span><span style="background-color: white; line-height: 1.083em;"> Unfortunately, it appears that the reviewers, editors, and authors did not address them.</span></span></div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></span></div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;">In the spirit of Rolf Zwaan's recent "<a href="http://rolfzwaan.blogspot.com/2013/08/50-questions-about-messy-rooms-and.html">50 questions</a>" post (although this paper is far stronger than the one he critiqued), here are 19 comments/questions about the paper and supplementary materials (in a somewhat arbitrary order). I hope that the authors can answer many of these questions by providing more information. </span></span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 17.3125px;">Some might be harder to address.</span><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 1.083em;"> I would be happy to post their response here if they would like.</span></div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></span></div>
<h3>
<b style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 1.083em;">19 Questions and Comments </b></h3>
<div>
<span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 1.083em;">1.</span><b style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 1.083em;"> </b><span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 1.083em;">The sample size is small given the scope of the claims, averaging about 15 per group. That's worrisome -- it's too small a sample to be confident that random assignment compensates for important unknown differences between the groups.</span></div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;">2. The no-contact control group is of <a href="http://blog.dansimons.com/2013/07/pop-quiz-what-can-we-learn-from.html">limited value</a>. All it tells us is whether the training group improved more than would be expected from just re-taking the same tests. It's not an adequate control group to draw any conclusions about the effectiveness of training. It does nothing to control for motivation, placebo effects, </span><span style="line-height: 17.3125px;">differences</span><span style="line-height: 1.083em;"> in social contact, differences in computer experience, etc. Critically, the relative improvements due to multitasking training reported in the paper are consistently weaker (and fewer are statistically significant) when the comparison is to the active "single task" control group. According to Supplementary Table 2, out of the 11 reported outcome measures, the multitasking group improved more than the no-contact group on 5 of those measures, and they improved more than the single-task control on only 3. </span></span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;"><br /></span></span></span></div>
<div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;">3. The dual-task element of multitasking is the mechanism that purportedly underlies transfer to the other cognitive tasks, and neither the active nor the no-contact control included that interference component. If neither control group had the active ingredient, why were the effects consistently weaker when the multitasking group was compared to the single task group than when compared to the control group? That suggests the possibility of a differential placebo effect: Regardless of whether or not the condition included the active ingredient, participants might improve because they expected to improve. </span></span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white;"><span style="line-height: 1.083em;">4. The active control group is relatively good (compared to those often used in cognitive interventions) - it uses many of the same elements as the multitasking group and is fairly closely matched. But, the study included no checks for </span></span><span style="background-color: white; line-height: 1.083em;">differential expectations between the two training groups. If participants expected greater improvements on some outcome measures from multitasking training than from single-task training, then some or all of the benefits for various outcome measures might have been due to </span><span style="background-color: white; line-height: 17.3125px;">expectations</span><span style="background-color: white; line-height: 1.083em;"> rather than to any benefits of dual-task training. For details, see our <a href="http://pps.sagepub.com/content/8/4/445.full">paper in Perspectives</a> that discusses this pervasive problem with active control groups. If you want the shorter version, see my <a href="http://blog.dansimons.com/2013/07/pop-quiz-what-can-we-learn-from.html">blog post</a> about it. Just because a control group is active does not mean that it accounts for differential expectations across conditions.</span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;">5. The paper reports that improvements in the training task were maintained for 6 months. That's welcome information, but not particularly surprising (see #13 below). The critical question is whether the <i>transfer</i> effects were long-lasting. Were they? The paper doesn't say. If they weren't, then all we know is that subjects retained the skills they had practiced, and we know nothing about the long-term consequences of that learning for other cognitive skills.</span></span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;">6. According to Figure 9 in the supplement, 23% of the participants who enrolled in the study were dropped from the study/analyses (60 enrolled, 46 completed the study). Did drop out or exclusion differentially affect one group? If participants were dropped based on their performance, how were the cutoffs determined? Did the number of subjects excluded for each reason vary across groups? Are the results robust to different cut-offs? What are the implications for the broad use of this type of training if nearly a quarter of elderly adults cannot do the tasks adequately?</span></span></span></div>
</div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;"><br /></span></span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white;"><span style="line-height: 1.083em;">7. Supplemental Table 2 reports 3 significant outcome measures out of 11 tasks/measures (when comparing to the active control group). Many of those tasks include multiple measures and could have been analyzed differently. </span></span><span style="background-color: white; line-height: 17.3125px;">Consider also that each measure could be compared to either control group and that it also would have been noteworthy if the single task group had outperformed the no-contact group. That means there were a really large number of possible statistical tests that, if significant, could have been interpreted as supporting transfer of training.</span><span style="background-color: white; line-height: 17.3125px;"> </span><span style="background-color: white;"><span style="line-height: 1.083em;">I see no evidence of correction for multiple tests. Only</span></span><span style="background-color: white;"><span style="line-height: 1.083em;"> a handful of these many tests were significant, and most were just</span></span><span style="background-color: white; line-height: 1.083em;"> barely so (interaction of session x group was p=.08, p=.03, and p=.02). For the crucial comparison of the multitasking group to each control group, the table only reports a "yes" or "no" for statistically significant at .05, and they must be close to that boundary. (There also are oddities in the table, like a reported significant effect with d=.67, but a non-significant one with d=.68 for the same group comparison.) With correction for multiple comparisons, I'm guessing that none of these effects would reach statistical significance. A </span><span style="background-color: white; line-height: 17.3125px;">confirmatory replication with a larger sample would be needed to show that the few significant results (with small sample sizes) were not just false positives. </span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white;"><span style="line-height: 1.083em;">8. The pattern of outcome measures is somewhat haphazard and inconsistent with the hypothesis that dual-task interference is the reason for differential improvements. For example, i</span></span><span style="background-color: white; line-height: 1.083em;">f the key ingredient in dual-task training is interference, why didn't multitasking training lead to differential improvement on the dual-task outcome measure? That lack of a critical finding is largely ignored.</span><span style="background-color: white; line-height: 17.3125px;"> Similarly, why was there a benefit for the working memory task that didn't involve distraction/interference? Why wasn't there a difference in the visual short term memory task both with and without distraction? Why was there a benefit for the working memory task without distraction (basically a visual memory task) but not the visual memory task? The pattern of improvements seems inconsistent with the proposed mechanism for improvement. </span></span></div>
<div>
<span style="background-color: white;"><span style="line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;"> </span></span></span></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white;"><span style="line-height: 1.083em;">9. </span></span>The study found that practice on a multitasking video game improves performance on that game to level of a college student. Does that mean that the game improved multitasking abilities to the level of a 20 year old? No, although you'd never know that from the media coverage. The actual finding is that after 12 hours of practice on a game, seniors play as well as a 20 year old who is playing the game for the first time. The finding does not show that multitasking training improved multitasking more broadly. In fact, it did not even transfer to a different dual task. Did they improve to the level of 20 year olds on any of the transfer tasks? That seems unlikely, but if they did, that would be bigger news.</span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;">10. The paper reports only difference scores and does not report any means or standard deviations. This information is essential to help the reader decide whether improvements were contaminated by regression to the mean or influenced by outliers. Perhaps the authors could upload the full data set to <a href="http://openscienceframework.org/">openscienceframework.org</a> or another online repository to make those numbers available?</span></span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;"><br /></span></span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;">11. Why are there such large performance decreases in the no-contact group (Figures 3a and 3b of the main paper)? This is a slowing of 100ms, a pretty massive decline for just one month of aging. Most of the other data are presented as z-scores, so it's impossible to tell whether the reported interactions are driven by a performance <i>decrease</i> in one or both of the control groups rather than an improvement in the multitasking group. That's another reason why it's essential to report the performance means and standard deviations for both the pre-test and the post-test.</span></span></span></div>
</div>
</div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;"><br /></span></span></span></div>
<div>
<span style="background-color: white; line-height: 1.083em;"><span style="font-family: Arial, Helvetica, sans-serif;">12. It seems a bit generous to claim (p.99) that, in addition to the significant differences on some outcome measures, there were trends for better performance in other tasks like the UFOV. Supplementary Figure 15 shows no difference in UFOV improvements between the multitasking group and the no-contact control. Moreover, because these figures show Z-scores, it's impossible to tell whether the single-task group is improving less or even showing worse performance. Again, we need the means for pre- and post-testing to evaluate the relative improvements.</span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">13. Two of the core findings of this paper, that multitasking training can improve the performance of elderly subjects to the levels shown by younger subjects and that those improvements last for months, are not novel. In fact, they were demonstrated nearly 15 years ago in a paper that wasn't cited. <a href="http://www.ncbi.nlm.nih.gov/pubmed/10344190">Kramer et al (1999)</a> found that giving older adults dual-task training led to substantial improvements on the task, reaching the levels of young adults after a small amount of training. Moreover, the benefits of that training lasted for two months. Here are the relevant bits from the Kramer et al abstract: </span></div>
<blockquote class="tr_bq">
<span style="background-color: white; color: #2e2e2e; line-height: 20px; text-align: justify; word-spacing: -1px;"><span style="font-family: Arial, Helvetica, sans-serif;">Young and old adults were presented with rows of digits and were required to indicate whether the number of digits (element number task) or the value of the digits (digit value task) were greater than or less than five. Switch costs were assessed by subtracting the reaction times obtained on non-switch trials from trials following a task switch.... First, large age-related differences in switch costs were found early in practice. Second, and most surprising, after relatively modest amounts of practice old and young adults switch costs were equivalent. Older adults showed large practice effects on switch trials. Third, age-equivalent switch costs were maintained across a two month retention period. </span></span><br />
<div>
<span style="background-color: white; color: #2e2e2e; line-height: 20px; text-align: justify; word-spacing: -1px;"><span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></span></div>
</blockquote>
<span style="font-family: Arial, Helvetica, sans-serif;">14. While we're on the subject of novelty, the authors state in their abstract: "These findings ... provide the first evidence, to our knowledge, of how a custom-designed video game can be used to assess cognitive abilities across the lifespan, evaluate underlying neural mechanisms, and serve as a powerful tool for cognitive enhancement." Unfortunately, they seem not to have consulted the <a href="http://scholar.google.com/scholar?hl=en&q=%22space+fortress%22&btnG=&as_sdt=1%2C14">extensive literature</a> on the effects of training with the custom-made game Space Fortress. That game was designed by cognitive psychologists and neuroscientists in the 1980s to study different forms of training and to measure cognitive performance. It includes components designed to train and test memory, attention, motor control, etc. It has been used with young and old participants, and it has been studied using ERP, fMRI, and other measures. The task has been used to study cognitive training and transfer of training both to laboratory tasks and to real world performance. It has also been used to study different forms of training, some of which involve explicit multitasking and others that involve separating different task components. There are dozens of papers (perhaps more than 100) using that game to study cognitive abilities, training, and aging. Those earlier studies suffered from many of the same problems that most training interventions do, but they do address the same issues studied in this paper. The new game looks much better than Space Fortress, and undoubtably is more fun to play, but it's not novel in the way the authors claim.</span><br />
<div>
</div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;">15. Were the experimenters who conducted the cognitive testing blind to the condition assignment? That wasn't stated, and if they were not, then experimenter demands could contribute to differential improvements during the post-test. </span></span></span></div>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;">16. Could the differences between conditions be driven by differences in social contact and computer experience? The extended methods state, "if requested by the participant, a laboratory member would visit the participant in their home to help set up the computer and instruct training." How often was such assistance requested? Did the rates differ across groups? Later, the paper states, "All participants were contacted through email and/or phone calls on a weekly basis to encourage and discuss their training; similarly, in the event of any questions regarding the training procedures, participants were able to contact the research staff through phone and email." Presumably, the authors did not really mean "all participants." What reason would the no-contact group have to contact the experimenters, and why would the experimenters check in on their training progress? As noted earlier, differences like this are one reason why no-contact controls are entirely inadequate for exploring the necessary ingredients of a training improvement.</span></span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;"><br /></span></span></span></div>
<div>
<span style="background-color: white;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 1.083em;">17. Most of the assessment tasks were computer based. Was there any control for prior computer experience or the amount of additional assistance each group needed? If not, the difference between these small samples might partly be driven by baseline differences in computer skills that were not equally distributed across conditions. The training tasks might also have trained the computer skills of the older participants or increased their comfort with computers. If so, improved computing skills might account for any differences in improvement between the training conditions and the no-contact control.</span></span></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">18. The paper states, "Given that there were no clear differences in sustained attention or working memory demands between MTT and STT, transfer of benefits to these untrained tasks must have resulted from challenges to overlapping cognitive control processes." Why are there no differences? Presumably maintaining both tasks in mind simultaneously in the multitasking condition places some demand on working memory. And, the need to devote attention to both tasks might place a greater demand on attention as well. Perhaps the differences aren't clear, but it seems like an unverified assumption that they tap these processes equally.</span></div>
<div style="-webkit-text-stroke-width: 0px; color: black; font-style: normal; font-variant: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<div style="font-weight: normal; margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif;">19. The paper reports a<span style="text-indent: -0.25in;"> significant relationship
between brain measures and TOVA improvement (p = .04). The “statistical analyses”
section reports that one participant was excluded for not showing the expected pattern
of results after training (increased midline frontal theta power). Is this a common practice? What is the p
value of the correlation when this excluded participant is included? Why aren’t correlations reported for the relationship between the transfer tasks and training performance or brain changes for the single-task control group? If the same relationships exist for that group, then that undermines the claim that multitask training is doing something special. The authors report that these relationships aren't significant, but the ones for the multitasking group are not highly significant either, and the presence of a significant relationship in one case and not in the other does not mean that the effects are reliably different for the two conditions. </span></span></div>
<!--[if gte mso 9]><xml>
<o:DocumentProperties>
<o:Revision>0</o:Revision>
<o:TotalTime>0</o:TotalTime>
<o:Pages>1</o:Pages>
<o:Words>76</o:Words>
<o:Characters>437</o:Characters>
<o:Company>Illinois</o:Company>
<o:Lines>3</o:Lines>
<o:Paragraphs>1</o:Paragraphs>
<o:CharactersWithSpaces>512</o:CharactersWithSpaces>
<o:Version>14.0</o:Version>
</o:DocumentProperties>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>JA</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="276">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0in;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment--><div style="margin: 0px;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<h3>
<span style="font-family: Arial, Helvetica, sans-serif;"><b>Conclusions</b></span></h3>
<div style="text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">Is this a worthwhile study? Sure. Is it fundamentally flawed? Not really. Does it merit extensive media coverage due to it's importance and novelty? Probably not. Should seniors rush out to buy brain training games to overcome their real-world cognitive declines? Not if their decision is based on this study. Should we trust claims that such games might have therapeutic benefits? Not yet.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Even if we accept all of the findings of this paper as correct and replicable, nothing in this study shows that the game training will improve an older person's ability to function outside of the laboratory. Claims of meaningful benefits, either explicit or implied, should be withheld until demonstrations show improvements on real or simulated real-world tasks as well. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">This is a good first study of the topic, and it provides a new and potentially useful way to measure and train multitasking, but it doesn't merit quite the exuberance displayed in media coverage of it. If I were to rewrite the abstract to reflect what the study actually showed, it might sound something like this:</span><br />
<blockquote class="tr_bq">
<span style="font-family: Arial, Helvetica, sans-serif;">In three studies, we validated a new measure of multitasking, an engaging video game, by replicating prior evidence that multitasking declines linearly with age. Consistent with earlier evidence, we find that 12 hours of practice with multitasking leads to substantial performance gains for older participants, bringing their performance to levels comparable to those of 20-year-old subjects performing the task for the first time. And, they remained better at the task even after 6 months. The multitasking improvements were accompanied by changes to theta activity in EEG measures. Furthermore, an exploratory analysis showed that multitasking training led to greater improvements than did an active control condition for a subset of the tasks in a cognitive battery. However, the pattern of improvements on these transfer tasks was not entirely consistent with what we might expect from multitasking training, and the active control condition did not necessarily induce the same expectations for improvement. Future confirmatory research with a larger sample size and checks for differential expectations is needed to confirm that training enhances performance on other tasks before strong claims about the benefits of training are merited. The video game multitasking training we developed may prove to be a more enjoyable way to measure and train multitasking in the elderly.</span></blockquote>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-31955737825176897532013-09-05T09:05:00.005-05:002020-12-21T09:15:05.566-06:00HI-BAR (Had I Been A Reviewer)<div style="text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; line-height: 1.083em;"><b>HI-BAR</b><br /><i>Had I Been a Reviewer</i></span></span></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: left;">
If you're a researcher, you undoubtedly have had the experience of reading a new paper in your specialty area and thinking to yourself, "Had I been a reviewer, I would have raised serious concerns about these findings and claims<i>." </i>Or, less charitably, you might ask, "How the hell did that paper survive peer review?"</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Each paper is reviewed by only 2 or 3 people, and small samples can lead to flawed conclusions. Given that I can't insert myself into the review process in advance of publication, I will, on occasion, use my blog to post the sorts of comments I would have made <i>Had I Been A Reviewer </i>of the original manuscript. My comments won't always take the same form that they would have if I had reviewed the paper in advance of publication when constructive comments might improve a manuscript. Rather, they will comment on the strengths and shortcomings of the finished product. On occasion, when I have reviewed a manuscript and the paper was published without addressing major concerns, I might post the reviews here as well (I always sign my reviews, so they won't come as a surprise to the authors in such cases).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Not all of these posts will be take-downs or critiques, although some will be. Post-publication review can help to correct mistakes in the literature, and it also identifies controversies that might have been glossed over in a manuscript media coverage of it and it can inspire future research. I hope that more researchers will take up the call and post their own HI-BAR post-publication reviews.</div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-80780405647647791442013-08-12T13:46:00.004-05:002020-12-21T09:15:14.772-06:00Good resources for science writing/speaking?<div>
For a psychology graduate class I'm teaching this fall (on speaking/writing for general audiences), I'm trying to create a list of good resources on writing, speaking, and blogging about science. I'm hoping that you can help. </div>
<div>
<br /></div>
<div>
I'm particularly interested in finding good discussions of the value and risks of blogging, suggestions for best practices in writing/speaking, etc. Do you have a favorite go-to source for such advice? Do you know of helpful resources for beginning science writers and speakers? If so, please leave them in the comments (or send them to me directly). I'll compile the full list and will post it here.</div>
<div>
<br /></div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-62889944193527116602013-08-07T11:31:00.005-05:002020-12-21T09:15:32.917-06:00Stop the pressesYesterday I encountered something I've never seen before: a formal <a href="http://www.eurekalert.org/pub_releases/2013-08/sfpa-vgb080213.php">press release</a> from an academic society (SPSP) about a conference presentation of unpublished research:<br />
<blockquote class="tr_bq">
<a href="http://www.eurekalert.org/pub_releases/2013-08/sfpa-vgb080213.php">http://www.eurekalert.org/pub_releases/2013-08/sfpa-vgb080213.php</a></blockquote>
A friend of mine forwarded it to me because it makes claims about the cognitive benefits of video game training, an area fraught with methodological problems that my colleagues and I have written about extensively (e.g, here's a <a href="http://blog.dansimons.com/2013/07/pop-quiz-what-can-we-learn-from.html">recent blog post</a> about a <a href="http://pps.sagepub.com/content/8/4/445.full.pdf+html">recent critique</a> of such interventions). My guess is that the design shortcomings we discussed in <a href="http://pps.sagepub.com/content/8/4/445.full.pdf+html">that paper</a> undermine the claims that these authors are making. But, I have no way to know. The actual research isn't available.<br />
<i><br /></i>
<i>Why does this work merit a press release now, before the research has been published? </i><br />
<br />
The purpose of a press release is to draw public (and media) attention to a new finding, but in this case, the press release effectively <i>is</i> the finding because nobody can access the actual research. Journalists or science writers covering this study will have no more information than is available in the release itself, so they cannot verify that the research actually shows what the release claims that it does. In other words, the press release encourages churnalism rather than science reporting.<br />
<br />
In my view, academic societies should not be encouraging media coverage of research until the actual research is available for popular consumption. Doing so risks misleading the public. For this particular release, if the studies suffer from the problems we discussed in our <a href="http://pps.sagepub.com/content/8/4/445.full.pdf+html">recent article</a>, then the conclusions might be unjustified and there would be a reasonable chance that the research would not survive the peer review process (I can only hope that reviewers would nix publication if the claims aren't justified). If that happened, then the press release would have hyped vapor-findings, claims that lack any underlying support. How does that benefit the popular appraisal of our field?<br />
<div>
<br /></div>
Journalists and bloggers are free to discuss research they learn about at conferences, of course. And they typically do a good job in noting when findings are tentative (or giving enough details that others can evaluate the claims). But a formal press release from an academic society about unpublished research that is not available seems to me to be a different beast.<br />
<br />
Are there cases in which an academic society should issue a press release based on a conference presentation? Do you think this sort of press release is acceptable? I'd be curious to hear the perspectives of other scientists and science writers. Let me know what you think?Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0tag:blogger.com,1999:blog-2166457665608842386.post-74433641338906619592013-07-09T10:36:00.004-05:002020-12-21T09:15:41.909-06:00Pop Quiz - What can we learn from an intervention study?<div style="margin-bottom: 9px;">
<div style="background-color: white;">
<h3 style="text-align: center;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">Pop Quiz</span></h3>
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><b>1.</b> Why is a double-blind, placebo-controlled study with random assignment to conditions the <i>gold standard</i> for testing the effectiveness of a treatment?</span></div>
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span></div>
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><b>2.</b> If participants are not blind to their condition and know the nature of their treatment, what problems does that lack of control introduce?</span></div>
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span></div>
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><b>3</b>. Would you use a drug if the only study showing that it was effective used a design in which those people who were given the drug knew that they were taking the treatment and those who were not given the drug knew they were not receiving the treatment? If not, why not?</span></div>
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span></div>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">Stop reading now, and think about your answers. </span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">Most people who have taken a research methods class (or introductory psychology) will be able to answer all three. The gold standard controls for participant and experimenter expectations and helps to control for unwanted variation between the people in each group. If participants know their treatment, then their beliefs and expectations might affect the outcome. I would hope that you wouldn't trust a drug tested without a double-blind design. Without such a design, any improvement by the treatment group need not have resulted from the drug.</span><br />
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span></div>
<div style="color: black;">
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">In a </span><a href="http://pps.sagepub.com/content/8/4/445.full" style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">paper</a><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"> out today in </span><i style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><a href="http://pps.sagepub.com/content/current">Perspectives on Psychological Science</a></i><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">, my colleagues (Walter Boot, Cary Stothart, and Cassie Stutts) and I note that </span><span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">psychology </span><wbr style="color: #222222; font-family: Arial, Helvetica, sans-serif;"></wbr><span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">interventions typically cannot blind participants to the nature of the intervention—you know what's in your "pill." If you spend 30 hours playing an action video game, you know which game you're playing. </span><span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">If you are receiving treatment for depression, you know what is involved in your treatment.</span><span style="color: #222222; font-family: Arial, Helvetica, sans-serif;"> </span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">Such studies almost never confront the issues introduced by the lack of blinding to conditions, and most make claims about the effectiveness of their interventions when the design does not permit that sort of inference. Here is the problem:</span></div>
<blockquote class="tr_bq">
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">If participants know the treatment they are receiving, they may form expectations about how that treatment will affect their performance on the outcome measures. And, participants in the control condition might form different expectations. If so, any difference between the two groups might result from the consequences of those expectations (e.g., arousal, motivation, demand characteristics, etc.) rather than from the treatment itself.</span></blockquote>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">A truly double blind design addresses that problem—if people don't know whether they are receiving the treatment or the placebo, their expectations won't differ. Without a double blind design, researchers have an obligation to use other means to control for differential expectations. If they don't, then a bigger improvement in the treatment group</span><span style="color: #222222; font-family: Arial, Helvetica, sans-serif;"> </span><span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">tells you nothing conclusive about the effectiveness of the treatment. Any improvement </span><span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">could be due to the treatment, to different expectations, or to some combination of the two. No causal claims about the effectiveness of the treatment are justified.</span><br />
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">If we wouldn't trust the effectiveness of a new drug when the only study testing it lacked a control for placebo effects, why should we believe a psychology intervention if it lacked any controls for differential expectations? Yet, almost all published psychology interventions attribute causal potency to interventions that lack such controls. Authors seem to ignore this known problem, reviewers don't block publication of such papers, and editors don't reject them.</span><br />
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="color: #222222; font-family: Arial, Helvetica, sans-serif;">Most psychology interventions have deeper problems than just a lack of controls for differential expectations. Many do not include a control group that is matched to the treatment group on everything other than the hypothesized critical ingredient of the treatment. Without such matching, any difference between the tasks could contribute to the difference performance. Some</span><span style="background-color: transparent; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"> psychology interventions use completely different control tasks (e.g., crosswords puzzles as a control for working memory training, educational DVDs a control for auditory memory training, etc). Even worse, some do not even use an active control group, instead comparing performance to a "no-contact" control group that just takes a pre-test and a post-test. Worst of all, some studies use a wait-list control group that doesn't even complete the outcome measures before and after the intervention.</span><br />
<span style="background-color: transparent; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span>
<span style="background-color: transparent; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">In my view, a psychology intervention that uses a waitlist or no-contact control should not be published. Period. Reviewers and editors should reject it without further consideration -- it tells us almost nothing about whether the treatment had any effect, and is just a pilot study (and a weak one at that). </span><br />
<span style="background-color: transparent; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;"><br /></span>
<span style="background-color: transparent; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">Studies with active control groups that are not matched to the treatment intervention should be viewed as suspect—we have no idea what differences between the treatment and control condition were necessary. Even closely matched control groups do not permit causal claims if the study did nothing to check for differential expectations.</span><br />
<br />
<span style="background-color: transparent; font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif; line-height: 18px;">To make it easy to understand these shortcomings, here is a flow chart from our paper that illustrates when causal conclusions are merited and what we can learn from studies with weaker control conditions (short answer -- not much):</span></div>
</div>
<div style="margin-bottom: 9px;">
<div style="line-height: 18px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieeHJBbUNO3ZnCtR_irwM0CawMDd-t0VQMx-4Seh79Ulkd2w7utxDslJgo0BYJcB7uVVoZMsihUvlcQOS4PahQAZNo-_tYtWHaKEtKy1Rl3aycShnQyki3ZPh8KU6SU52NU-i1ivduNHk/s1047/flow_chart.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="Figure illustrating the appropriate conclusions as a function of the intervention design" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieeHJBbUNO3ZnCtR_irwM0CawMDd-t0VQMx-4Seh79Ulkd2w7utxDslJgo0BYJcB7uVVoZMsihUvlcQOS4PahQAZNo-_tYtWHaKEtKy1Rl3aycShnQyki3ZPh8KU6SU52NU-i1ivduNHk/s1047/flow_chart.jpg" height="640" title="Figure from Boot et al (2013), page 448" width="550" /></a></div>
<div style="text-align: center;">
<br /></div>
</div>
<div style="line-height: 18px; margin-bottom: 9px;">
<span style="font-family: Helvetica Neue, Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br />Almost no psychology interventions even fall into that lower-right box, but almost all of them make causal claims anyway. That needs to stop.</span><span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span><br />
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: 'Helvetica Neue', Arial, Helvetica, sans-serif;">If you want to read more, check out our OpenScienceFramework <a href="http://openscienceframework.org/project/7EB6A/">Page for this paper/project</a>. It includes an answers to a set of <a href="http://openscienceframework.org/project/7EB6A/node/KSzdu/wiki/home">Frequent Questions</a>.</span></div>
Daniel J. Simonshttp://www.blogger.com/profile/02968898312917472467noreply@blogger.com0