Note: This post was written for the March 2016 newsletter of the Australian Psychology Society’s Psychology of Relationships Interest Group (PORIG). I have made a few small changes, and now include the following link to a talk I gave discussing issues similar to this post at the “Navigating the New Era of Social & Personality Psychology” preconference of the 2016 SPSP main conference: https://www.youtube.com/watch?v=QdUtnA8vUn8 (audio is a bit wonky).

This is a question I hear from time to time, particularly from relationship science scholars. Although the bulk of extant relationship research involved data collected from one individual involved in a relationship, typically at one point in time (see Kashy, Campbell & Harris, 2006), a sizeable minority of the field’s work involves data that is much more complex (e.g., longitudinal, self-report and observational dyadic data). Adding to this complexity is the time and expense associated with recruiting dating and/or married couples for these studies, the difficulty of obtaining and coding behavioural/interactive data, and in some cases the expense associated with obtaining particular measures (e.g., hormonal assays of saliva and/or blood). It is often challenging to obtain large samples necessary to increase statistical power and thus reduce the probability of Type II errors, and the false-discovery rate, for these complex studies. And studies of this magnitude rarely set out to test one pre-specified hypothesis; instead, these projects will collect a large amount of data across a number of measures/constructs with the goal of testing many hypotheses to be forged in the future. It is these types of projects (ones in development, those being run presently, as well those already completed that offer a large amount of available data) that are often referred to when researchers inquire, “What if I can’t do open science?”.

What is open science? Briefly, “open science” refers to the public sharing of all aspects of the research process (for more details see: https://en.wikipedia.org/wiki/Open_science; see also https://osf.io/3swkp/). This sharing involves, for example, (a) publicly disclosing study hypotheses prior to actually testing them, (b) making available all of the study materials and procedures (e.g., on the Open Science Framework, https://osf.io/), and (c) publicly disclosing a data analytic plan (i.e., how you plan to test your hypotheses given the measures/procedures of your study). Is it possible to “do” open science when implementing complex study designs as discussed above that are enacted by many relationship scholars?

In my opinion, yes; not only is it possible, it is being done. The complex designs employed by relationship scholars (and others), however, do require unique open science solutions. I, along with my colleagues Timothy Loving and Etienne LeBel, suggested many unique solutions for these complex designs in a paper we published in Personal Relationships in 2014. Instead of reiterating the points we made in that paper, I want to instead briefly share how my research team has engaged in open science practices for different types of research projects in the field of relationship science.

(1) If my research is largely exploratory, how can I publicly disclose hypotheses and data analytic plans?

When research projects are largely exploratory, you can share your study materials as you can with confirmatory research projects. You can also briefly state that your research project is meant to explore possible associations among certain study variables, and the reason(s) for the exploratory nature of the study. You can also provide, if appropriate, a set of guidelines for how you plan to explore the data collected (templates for different types of disclosures, for both confirmatory and exploratory research, can be found here: https://osf.io/m7f8d/).

For example, Kiersten Dobson (a graduate student in my lab), has posted the following information on the OSF (link here: https://osf.io/4xcpy/): description of the study (including planned sample and analytic goals for the exploratory analyses), study materials, and methods. She then posted a copy of the obtained data set, and discussed the follow-up research currently being conducted that followed from the results of the initial exploratory study.

(2) What if the data I am using to test my hypotheses comes from a large dataset that already exists?

In this instance, if the dataset is not your own it may not be possible to publicly post all of the study materials and methods. You can, however, post a document that outlines all of the measures you plan to use from this dataset to test your hypotheses (note that it’s not necessary to include this information in your manuscript; rather, post this information on the OSF and simply link to it in your manuscript). You can also disclose the hypotheses you plan to test and the proposed data analytic plan. If the dataset is your own, you can also post a copy of all study materials.

Early in 2015 a graduate student from the VU Amsterdam (Asuman Buyukcan-Tetik; now a Professor at Sabanci University, Istanbul) visited my lab for three months. We proposed testing new hypotheses by analyzing already existing data—a large dataset collected under the supervision of the PI Catrin Finkenauer. Prior to conducting analyses, Asuman publicly posted, and pre-registered (i.e., the files are “frozen” and cannot be edited or deleted), information about (a) the project and hypotheses, (b) the method, and (c) the strategy of planned analyses. I want to point out that the strategy of analysis contained a few different options that were dependent to some degree on the outcome of the initial analyses planned (i.e., we were only partly certain of what we expected to find; link here: https://osf.io/d7x2p/).

Also in 2014, Rhonda Balzarini, a graduate student in my lab, was given the opportunity to use a large dataset (over 3000 participants) collected by a group of researchers based in Universities throughout the USA (PIs, Bjarne Holmes, Justin Lehmiller, Jennifer Harman, and Nicole Atkins). Rhonda and I met regularly for about two months in the fall of 2014 to discuss our research interests with respect to the dataset and to derive specific hypotheses prior to looking at any of the actual data (i.e., no peeking). Rhonda then publicly disclosed (and pre-registered), prior to analyses, our hypotheses, the methods and measures used in our analyses (link here: https://osf.io/vs574/).

(3) What if in my study I plan to test more than one set of hypotheses? Also, maybe I don’t know what the other hypotheses are yet so how could I possible “pre-register” them?

Big, complex dyadic studies, as mentioned above, rarely set out to test one set of hypotheses. It is entirely possible, however, to make all study materials/procedures publicly available, and to disclose the first set of planned hypotheses along with a data analytic plan. This is the approach taken by Taylor Kohut, a post-doc in my lab, for a large scale longitudinal dyadic study with an experimental intervention (3 conditions) initiated at the midpoint of the study. Instead of trying to explain the study here, I will simply refer you to all of the study information (and I do mean ALL of it) posted on the OSF: https://osf.io/yksxt/ (including study rationale, methods/measures, analytic strategy, and our recruitment plan). It is guaranteed that we will develop new hypotheses in the future, hypotheses that have not yet been considered or discussed. When that time comes we will simply add a new component to the OSF project that discusses our new hypotheses and data analytic plan, and what variables from the original study we plan to use.

(4) What if I can’t make data available? Or code?

There are at least a few concerns with sharing data: (1) other people can use it and benefit from your efforts, and (2) what if participants can identify their own, or their partner’s, sensitive data? With respect to the first concern, the OSF allows users to create a DOI for all files, including datasets, so that if someone does choose to use your data they can properly cite the dataset. Additionally, users on the OSF can license their datasets in seconds, making it legally mandatory for anyone using the dataset to properly cite its use. With respect to the second concern, there are many ways to de-identify data sets, and ways to restrict the use of datasets to other researchers (e.g., post the data on the OSF to a private project page or component, and then grant access to that page or component when asked by other researchers). This is admittedly a big issue that warrants a much longer discussion, and is something we discuss in more detail in a paper in press at the Journal of Personality and Social Psychology (LeBel, Campbell, & Loving, in press; for a pre-accepted draft of this manuscript click here: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2616384).

In my lab we are moving toward posting data needed to reproduce the results of any analyses reported in our manuscripts, and we also post the code/syntax that we used to run our models. At this link https://osf.io/ryfse/ you can find data sets and code needed to reproduce the analyses in a paper in press at the Journal of Experimental Social Psychology, and at this link https://osf.io/me7jp/ you can find the data files (available upon request, but they are physically present in a component linked to this OSF page) and code (for SAS) needed to reproduce the analyses presented in this publication: http://www.collabra.org/articles/10.1525/collabra.24/. One benefit of this practice is that we independently run all study analyses to ensure we can reproduce the results reported in the manuscript prior to submitting for peer review.

There are undoubtedly other scenarios not discussed here that require novel open science solutions. My goal was to share a few of the questions I have heard most often, and show some of the answers we have come up with in my lab. Open science can be done with complex study designs within the field of relationship science, because it is being done. I therefore suggest that the question I posed at the outset of this piece be changed from “What if I can’t do open science?” to “How can I do open science with this study?”.

We now have the tools available to move away from the more “closed science” practices that have been typical of our field to this time. Using these tools is of course a choice, but not using them is also a choice. I have chosen to engage in open science practices for my own research going forward. My experiences so far suggest to me that open science practices have not stifled my creativity, limited what I choose to study, limited the exploration of ideas, or otherwise burdened my ability to discover new things (such as they were).

References

Campbell, L., Loving, T. J., & LeBel, E. P. (2014). Enhancing transparency of the research process to increase accuracy of findings: A guide for relationship researchers. Personal Relationships, 21(4), 531-545.

Kashy, D.A., Campbell, L., & Harris, D.W. (2006). Advances in data analytic approaches for relationships research: The broad utility of hierarchical linear modeling. In A. Vangelisti & D. Perlman (Eds.), The Cambridge Handbook of Personal Relationships (pp. 73-90). New York: Cambridge University Press.

This summer I looked closely at the books on my bookshelf to  rekindle my affection with these old flames. One book that stood out was written by Morton Deutsch and Robert M. Krauss (1965) simply titled “Theories in Social Psychology”. The first chapter has a section labelled “The Nature of Theory”. In my opinion it is still very relevant today and is a must read for social and personality psychologists. I will quote from this section a little as I paraphrase the main points:

1. Theories in the physical sciences often make definitive hypotheses that are logical extensions of the theory in question.

– For example (my example, not one used by Deutsch & Krauss), Einstein’s General Theory of Relativity made a specific hypothesis regarding how much light should be observed to bend around large objects (among other hypotheses of course). Everyone that understood the theory could derive the same specific hypothesis regarding the effects of gravity on the bending of light.*

2. Theories in Social Psychology do not typically make definitive hypotheses that are logical extensions of the theory in question. Instead, “…the ‘derivations’ from most of the theories in social psychology are usually not unequivocal, or strictly logical, for they skip steps, they depend on unexpressed assumptions, and they rest on the criterion of intuitive reasonableness or plausibility rather than on formal logical criteria of consistency.” (p. 7). As they go on to say on page 11, most theoretically derived hypotheses in social psychology are plausible inferences that can be made from an understanding of the theory rather than logical deductions dictated by the theory itself.

– This was not meant as slight to social psychological theories, but was rather pointed out as a reflection of the nascent state of our theories that does not allow for specific, definitive hypotheses to be derived. As theories are rigorously tested over time, some hypotheses should (hopefully) consistently replicate, others not so much, and a better understanding of the nomological network will emerge (Cronbach & Meehl, 1955).

As Feynman discusses in his Cornell lectures (https://www.youtube.com/watch?v=EYPapE-3FRw), using the scientific method we compute the consequences of new laws/theories, and then compare these consequences with observations. If the observations are not as predicted, then the theory is not confirmed (or as he says in this lecture, they are “wrong”). But as Deutsch and Kraus point out, computing the consequences of social psychological theories is not as straightforward as it is for theories in the physical sciences. Given this fact, it is likely that at times experts could derive discordant, yet plausible, hypotheses from the same social psychological theory. Theory testing and theory building in social psychology is therefore challenging.

Then I ran this simple thought experiment (N =1): imagine asking 20 research teams with expertise on, say, adult attachment theory to independently make specific hypotheses regarding the association between, for example, scores on anxious attachment and avoidant attachment with feelings of acceptance from a romantic partner across three experimental conditions: (1) control (no intervention), (2) relationship threat (e.g., when one partner does not want to spend time with the target participant), and (3) relationship boost (e.g., when one partner does want to spend time with the target participant). Next, have a team of coders rate the similarity of generated hypotheses. How similar would they be? How many different, plausible hypotheses would be put forward? My guess is that there would be some consistency across research teams, and some variability as well. I can generate what I feel are plausible, yet discordant, hypotheses all on my own. Feel free to conduct this thought experiment with your theory of choice. Not every expert, considering the same body of information, will generate the same hypotheses.

I am not blaming the theory for not being to make definitive hypotheses at all times. But it does seem that building theory would benefit from more truly confirmatory research being conducted that tests specific, plausible, and pre-registered hypotheses, with the results published regardless of the outcomes. Because as Feynman also says in the lecture linked to above, vague theories can accommodate almost any result, making them rather useless as theories.

* at this point it is obvious I am not a physicist, p < .00000000000001.

Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.

Deutsch, M., & Krauss, R.M. (1965). Theories in social psychology. New York: Basic Books Inc..

I support the efforts of researchers in our field to conduct direct/close replication attempts of published research. It is something I do myself. It is wonderful to see journal policies starting to embrace replication studies, including special issues in different journals devoted to replication studies in a given area of study. But…not really a but in the sense that I will now soften my support for replication research, but a but in the sense that because less labor/time/cost intensive studies will likely be the focus of the majority of these replication attempts compared to more labor/time/cost intensive studies it seems inevitable that an asymmetry in the precision of effect size estimates will develop over time.

Imagine a researcher that has decided to devote some research effort to replication studies comes across three published studies that pique his interest:

• Publication 1: recruited a large number of participants from either a University subject pool (i.e., undergraduate students that need to participate in research studies for course credit) or online (e.g., Amazon’s Mechanical Turk).
• Publication 2: recruited undergraduate students from the University subject pool, but had each participant come into the lab individually because of complex experimental manipulations as well as the collection of biological samples.
• Publication 3: recruited participants from the community, following participants over a period of two years with multiple testing sessions (both in lab and online).

I do not need to provide more details for these fictional studies to make the point that the labor, time and cost needed to directly replicate the methods of study 3 is much greater than for the other two studies, and is greater for study 2 compared to study 1. Given that researchers do not have access to unlimited resources over a prolonged period of time to conduct their own research let alone direct replications of the research of others (if you do, call me), it is reasonable to conclude that of the fictional studies presented more replication attempts would be made for study 1 than the other two studies. Over time, therefore, more precise estimates of the effect sizes obtained in “easy to run” studies will accumulate compared to “difficult to run” studies. Put another way, one shot correlational and experimental studies involving University students or MTurkers will be the focus of the bulk of replication attempts; studies with special populations (e.g., cross-cultural samples, married couples, parent-child interactions, and many, many others), those collecting “expensive” data (e.g., brain scans, hormonal assays), and studies using longitudinal designs (e.g., daily-diary studies, the early years of marriage, personality development across time, and so on) will be the focus of few, if any in some cases, direct replication attempts. I cannot imagine, for example, obtaining the grant funds necessary to directly replicate a multi-wave study of newly married couples over a period of two or more years [but see comment below–Brent Roberts did receive grant funding along these lines]. Even if funds were on hand to directly replicate a two-week diary study that included pre- and post-diary assessments, the amount of time needed to run the study, and the research assistants needed, would likely dissuade most researchers from endeavouring to replicate this research.

Now that the value of direct/close replication studies is generally recognized, perhaps we need to find ways of incentivizing replication attempts of studies that otherwise would be ignored by most replicators.

I am a replicator. I have undertaken, and continue to undertake, close replications of the research of others, as well as my own. What is my motivation? (see https://traitstate.wordpress.com/2015/03/13/how-do-you-feel-when-something-fails-to-replicate/, for a discussion of the perceived “psychology of the replicators”). I warn readers up front that my story is rather simple and boring.

I have been attending academic conferences since 1998, and one of things I enjoy most is sitting down with colleagues for cocktails and lively discussions. One topic of discussion always seems to be recently published, or about to be published, research. I have often asked, and been asked: “So, what do you think of that study?” That question can chart the course for many hours of debate. During one such discussion in Halifax (2012) I was talking with two colleagues about research on Attachment Theory and preferences for warm food options published in Psychological Science (Vess, 2012). We generally agreed with the notion that activating the attachment concerns of more anxiously attached individuals could make them feel “alone”, motivating them to seek out comfort (or warmth). At the same time I expressed some uncertainty regarding the methodological approach taken in the research. We all expressed our views, moved on to other topics, and eventually went our separate ways.

A little while later, when reflecting on this discussion, I felt that ideally it would be better for me to run the study to see if I could obtain the same pattern of results rather than simply voice my uncertainty regarding the manner in which the study was run. Why? My colleagues and I were essentially raising empirical questions about this research during our discussion, questions that should therefore be answered with data and not simply words. As the person raising many of these questions, I felt obliged to help answer them as best I can. I truly wanted to know if the results would replicate using the methods described by Vess, and the only way to find this out would be to collect additional data using these methods. With this data, I would be able to help answer, instead of only ask, this question: does the effect replicate? Working with Etienne LeBel, and with the input of the original researcher (Vess), we ran two large scale close replication studies of the original study 1 and published the results in Psychological Science (LeBel & Campbell, 2013).

So, what is my motivation as a replicator? When I feel some uncertainty about an idea, or methodological approach used to test an idea, I feel that data trump opinions. So I now devote some research effort to closely replicating published research findings, at the same time attempting to directly replicate my own research findings going forward when feasible.

Oh, and one other thing—somebody needs to do it, so why not me? For that matter, why not you?

References

LeBel, E. P., & Campbell, L. (2013). Heightened sensitivity to temperature cues in highly anxiously attached individuals: Real or elusive phenomenon? Psychological Science, 24, 2128-2120.

Vess, M (2012). Warm thoughts: Attachment anxiety and sensitivity to temperature cues. Psychological Science, 23, 472-474.