Tuesday, December 19, 2017

My Cattle

A while back, Lorne Campbell wrote a blog post  listing the preregistered publications from his lab. This is a great idea. It is easy to talk the talk, but it’s harder to walk the walk.

So under the notion that we don't want to be all hat and no cattle, I rounded up some replications and preregistered original papers that I co-authored.

First the replications.

I find performing replications very insightful. My role in two of the RRRs listed below (verbal overshadowing and facial feedback) was rather minor but the 2016 RRR and the issues surrounding it, on which I've blogged before, felt like an onslaught. The 2012 replication study was used to iron out an inconsistency in the literature. An additional replication study is close to getting accepted and will be added to the list in an update.

These days I use direct replications primarily when I want to build on work by others. As per Richard Feynman, before we move on we first need to attempt a direct replication of the effect we want to build on. We first need to know if we can reproduce it in our own lab.

Zwaan, R.A., Pecher, D. (2012). Revisiting Mental Simulation in Language Comprehension: Six Replication Attempts. PLoS ONE 7(12): e51382.

Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., ... Zwaan, R. A. (2014). Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.

Eerland, A., Sherrill, A.M., Magliano, J.P., Zwaan, R.A., Arnal, J.D., Aucoin, P., … Prenoveau, J.M. (2016). Registered replication report: Hart & AlbarracĂ­n (2011). Perspectives on Psychological Science, 11, 158-171. 

Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Jr., . . . Zwaan, R. A. (2016). Registered Replication Report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11, 917–928.

Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (in press). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review.

Next the original preregistered studies.

I started preregistering experiments several years ago. All in all, I find it an extremely important practice, quite possibly the most important thing we can do to improve the field. After a while preregistration becomes second nature and it becomes odd not to do it.

I have no experience yet with reviewed preregistrations (other than the three RRRs that I’ve participated in). My co-authors and I submitted one over three months ago but we haven’t gotten the reviews yet.

I should add, that I've co-authored quite a few additional empirical papers during this period that were not preregistered. This is mainly because the experiments in those papers were conducted years ago before preregistration was a thing.

Eerland, A., Sherrill, A.M., Magliano, J.P., Zwaan, R.A. (2017). The Blame Game: An investigation of Grammatical Aspect and Blame Judgments. Collabra: Psychology, 3(1): 29, 1–12.
         Only Experiments 3-5 were preregistered. Experiments 1&2 were conducted in 2012.

Eerland, A., Engelen, J.A.A., Zwaan, R.A. (2013). The influence of direct and indirect speech on mental representations. PloS ONE 8(6):  e65480.

Hoeben-Mannaert, L., Dijkstra, K., & Zwaan, R.A. (2017). Is color an integral part of a rich mental simulation? Memory & Cognition, 45, 974–982.

Pouw, W.J.T.L., van Gog, T., Zwaan, R.A., Agostinho. S., & Paas, F. (in press). Co-thought gestures in children’s mental problem solving: Prevalence and effects on subsequent performance. Applied Cognitive Psychology.

Sherrill A.M., Eerland A., Zwaan R.A., & Magliano J.P. (2015). Understanding how grammatical aspect influences legal judgment. PLoS ONE 10(10): e0141181.


And finally, to show that of course I also wear a stetson, here is a theoretical paper on replication. Yeehaw!

Zwaan, R.A., Etz, A., Lucas, R.E., & Donnellan, M.B. (in press). Making replication mainstream. Behavioral and Brain Sciences.


Thursday, December 7, 2017

The Long and Winding Road of our Latest Grammatical Aspect Article

A short blog post that strings together 8 tweets that I sent out today about our new paper.

Today our latest paper on grammatical aspect appeared in Collabra: Psychology. The article reflects the times we psychologists are living in. It does so not from the lofty perspective of the methodologist or statistician, but from the work floor on which the actual scientist (**ducks**) operates.

Our first two experiments were inspired by Hart & Albarricin (2011). This research itself was inspired by some of our own work but took it from cognition into the realm of social psychology, as I described in this blog post.

As the paper explains, these experiments were run in 2012, which is why they were not preregistered. Nobody was doing preregistration at the time. We were thinking to build on Hart and Albarricin (H&A) in what some would call a conceptual replication but which is better thought of as an extension.

For the life of us, we couldn’t get an effect like that of H&A. Then we got down to business and started a registered replication project in which we performed a direct replication of H&A. Along with 11 other labs, we found no effect.

We were sidetracked by the replication project. Especially because there were some troubling issues with the initial response to our RRR, as I describe here . We were sidetracked to the point that I’d completely forgotten about our 2012 experiments.

Luckily my co-authors had not and we decided to pick up the pieces of our study. It was clear that our research could no longer be driven by our H&A-inspired hypothesis, so we took a slightly different tack.

We conducted three more experiments, now all pre-registered, which yielded some interesting new findings, which you can read about in our paper. As usual per Collabra, the data are available and the reviews are open.