Crowdsourced psychological science

Why crowdsource psychological science?

Limits of the traditional vertical model

In psychology, the traditional way of doing research follows a vertical model. In other words, most of the time, research is carried out by small teams from a specific lab or university, and organized around one or two main researchers (often referred to as Principal Investigator, or PI). This small team contributes to all different stages of the research project: conception of the research question, design of the study, data collection, data analysis, discussion of the results, and publication of the manuscript.

However, vertical science has its own limitations that can impede the progress of science. Conducting science within small independent teams makes it difficult to conduct large-scale projects (with large samples, large databases and high statistical power), which may limit the scope of research. In traditional science, researchers have access to less resources, data, and methodologies. Small, independent team projects are also limited in the feedback they can get from other perspectives.

Nowadays, research teams do not necessarily have to make compromises anymore (e.g., small sample size, same stimuli, same contexts and no replication), because much of these limits could be tackled by a more crowdsourced research.

The replication crisis

In the scientific method, replicability is a fundamental criterion to qualify a theory as scientific. The replication crisis (or credibility crisis) is a methodological crisis in science that researchers began to acknowledge around the 2010s. The controversy revolves around the lack of reproducibility of many scientific findings, including those in psychology (e.g., among 100 studies, less than 50% of the findings were replicated).

Some of its main underlying causes are referred to as "questionable research practices". These include p-hacking (i.e., exploiting researcher degrees of freedom until a significant result is obtained), HARKing (i.e., hypothesizing after results are known), publication bias (i.e., the tendency among scholarly journals to only publish significant results), statistical reporting errors, and low statistical power often due to small sample sizes.

Among these various issues, small sample sizes and the lack of diversity within samples can be addressed through crowdsourced science—increasing the generalizability of findings and therefore their replicability as well. Indeed, samples in psychology often rely on college students and Western, Educated, Industrialized, Rich and Democratic (WEIRD) populations.

For these reasons, the replication crisis has contributed to the rise of crowdsourced, large-scale projects, especially replication projects held at an international scale like the Many Labs project, and the Psychological Science Accelerator (PSA). These crowdsourced projects aim at solving some issues raised by the replication crisis, more specifically by assessing the replicability of studies and generalisation of the results to other populations and contexts.

Ambitions of the horizontal model

In contrast to the vertical model of doing research, the horizontal model—an inherent principle of crowdsourced science—mainly relies on variations in terms of inclusiveness and communication. Its core principle is about the non-authority of one or two researchers in terms of resources, ownership and expertise. Following this principle, the different tasks within a project are distributed among many researchers. The whole project is then supported by a team, which ensures the coordination of the contributors.

A perfect horizontal model does not really exist because vertical and horizontal models are more conceptualized as extremes of a same continuum. This distributed model of science work is gaining popularity in the scientific community. In about 40 years and across different scientific disciplines, research teams have roughly doubled in size (from 2 to 4 people on average).

By encouraging larger crowds to contribute to research projects in psychology, the horizontal model aims at reducing some limits of the vertical one. It has three distinct ambitions:

• Carry out wide-ranging works
• Encourage a democratized psychological science
• Establish robust findings

Large-scale research projects

The first ambition of the horizontal model is to enable researchers to conduct more large-scale research projects (i.e., ambitious projects that cannot be conducted by small teams). By aggregating various skills and resources, it is possible to move from a model where research is defined by available means, to a model where research itself defines the necessary means to answer the research question.

Democratizing psychological science

The second ambition of the horizontal model is to compensate for inequalities (e.g., in terms of recognition, status, and success in a researcher's career). Psychology and more generally social sciences show a strong bias of what is called the Matthew effect (i.e., academic advantages going to those who are already the most renowned).

Early career researchers from less renowned institutions, less economically developed countries or underrepresented demographic groups are generally less likely to have access to high-profile projects. Crowdsourcing enables such researchers to contribute to impactful projects and gain recognition for their work.

Robust findings

The third ambition of the horizontal model is to improve the robustness, generalizability, and reliability of findings in order to increase the credibility of psychological science. Horizontal collaboration between research teams facilitates the replication of studies and makes it easier to detect biased effect sizes, p-hacking, and publication bias—different problems raised by the replication crisis (see also #The replication crisis).

Crowdsourcing in practice

Contributions at different stages of research

This section aims at detailing how crowdsourcing can contribute to the different stages of the research process, from the generation of research ideas to the publication of the outcomes.

Ideation

Ideation is the first step of any research project. In psychology, it refers to the process of defining the general idea behind a project—purpose, research question, and hypotheses. This step can be done in collaboration between several researchers to scan a broader spectrum of ideas and select those of broadest interest and impact. Faced with a research question, the different collaborators can bring their expertise in the construction of hypotheses.

Crowds can also help to generate new ideas to solve complex-problems, such as illustrated by the Polymath project.

Assembling resources

When assembling resources, crowdsourcing can be useful, especially for labs with less resources at their disposal. Online platforms such as Science Exchange and Studyswap allow researchers to establish new communication lines and share resources between labs. Matching resources from labs across the globe minimizes waste and facilitates research teams' ability to meet their goals.

Sharing tasks across labs can improve the efficiency of a research project, especially highly time-consuming ones. In biology, for example, studying the entire genome takes a lot of time. By distributing its investigation and combining resources across multiple labs, it is possible to accelerate the research process.

Study design

There can be many ways to design a study. Research teams across the world neither have the same theoretical background, nor are they all equipped with the same materials.

Crowdsourcing can be useful in the case of conceptual replications (i.e., testing a same research question through different operationalizations). When testing a same research question, variations in study designs can lead to strong variations in effect size estimations. Diversifying the methods used to test the same hypothesis across different populations—through collaborative projects—enables better estimations of the true consistency of a scientific claim.

Data collection

In psychology, data collection often relies on samples drawn from Western, Educated, Industrialized, Rich and Democratic (WEIRD) populations, which impedes the overall generalizability of findings. Crowdsourcing the data collection process by relying on multi-lab data collection and online crowdsourcing platforms (e.g., Amazon Mechanical Turk, Prolific) makes it easier to reach a wider audience of participants from different cultural backgrounds and non-WEIRD populations. When the research question makes it possible to rely on internet samples, it is also an easy way to recruit larger samples of participants with minimal financial input and within short amounts of time.

Most of the time, members of the general public are recruited to undergo studies as research participants but they can also be recruited to collect data and observations.

Data analysis

In research, data analysis refers to the process of cleaning, transforming, and modeling data using statistical tools, often with the purpose of answering a research question. Within a research project, this is typically done by a single analyst (or team) and results in a single analysis of a dataset.

Analytic strategies can differ greatly from one team to another. For example, a study found that among 241 published articles on fMRI, 223 different analytic strategies were used. Moreover, there can be many ways to test a single hypothesis from the same dataset. Although defensible, decisions in data analysis remain subjective, which can greatly affect research results. A way to counter this subjectivity is transparency. When data, analytic plans, and analytic decisions are made transparent and open access to the rest of the community, it facilitates criticism and gives the opportunity to explore alternative ways to analyse data.

"Crowdsourcing the analysis of the data reveals the extent to which research conclusions are contingent on the defensible yet subjective decision made by different analysts."—Uhlmann et al., 2019

Writing research reports

A research report is a document reporting the findings of a research project. The overall quality of a research report can benefit from crowdsourcing practices, especially during the writing process. Aggregating a large number of contributors increases the range of expertise and perspectives, which contributes to build more solid arguments. It also makes proofreading easier (e.g., catching grammatical errors, weird phrasing, typos, biases, factual errors, claim-checking).

To facilitate collaborative writing, some researchers have suggested guidelines for writing manuscripts with many authors. There should always be a leading author (or a few, but with individual responsibilities) who takes care of managing the writing process and takes explicit responsibility for any mistake—avoiding diffusion of responsibility in case of errors. It is also recommended that the leading author(s) follow the four general principles mentioned below:

• Care in crediting the coauthors team
• Clear and frequent mass communication
• Make sure materials associated to the manuscript are well-organized
• Early and deliberate decision-making

Peer review

Before getting published in an academic journal, submitted papers undergo peer review (i.e., the process of having an author's academic work reviewed by experts from the same field). Typically, it is performed by a limited number of selected reviewers. Crowdsourcing the peer review process increases chances of getting reviews from a larger number of experts in the relevant domain. This is also a way to significantly increase opportunities for better criticism and faster fact-checking before an article gets published.

Crowdsourced peer-review can be accomplished alongside open access peer-review (e.g., through centralized platforms dedicated to the discussion and criticism of research reports).

Examples in psychology

The replication crisis served as a prelude for the emergence of many large-scale collaborative projects. Some of the most important ones include the ManyLabs project, the ManyBabies project, the #EEGManyLabs project, the Reproducibility Project, the Collaborative Replication and Education Project (CREP), and the Psychological Science Accelerator (PSA).

Projects surrounding the COVID-19 pandemic

In response to the COVID-19 pandemic, several initiatives to improve collaboration between scientists from a variety of fields have been launched.

Studies exploring the impact of the COVID-19 pandemic on behavior and mental health are conducted and widely shared across social media such as Twitter.

In this context, the PSA made a call for studies surrounding the COVID-19 disease and retrieved 66 study proposals from different experts within one week. At the date of May 2020, three studies have been selected and are being conducted worldwide. Two of these studies aim at reaching a better adoption of health behaviors to avoid the spread of COVID-19, and one aims at helping people regulate their negative emotions during the crisis. This project aims to get results that could help all countries to face the pandemic with means adapted to their population.

Challenges and future directions

Confronting the vertical and horizontal models

Although a horizontal model of conducting science seems promising to overcome some limits of the vertical model (see also #Limits of the traditional vertical model and #Ambitions of the horizontal model), it is difficult to empirically assess its benefits. It remains unclear whether two sets of research that study the same question—either through a vertical or horizontal way of doing science—would lead to different outcomes or not.

Financial independence

Currently, to sustain a collaborative project, researchers either have to use money from their own grants or apply for fundings. Such projects are thus limited in the studies they can conduct without financial independence.

Coordinating hundreds of labs to conduct a study requires consequent administrative work. Structures like the Psychological Science Accelerator (PSA) have to ensure each participating lab retrieves ethics approval to conduct a given study (see also Institutional review board). Within the PSA, this background work is currently completed voluntarily by dedicated teams or researchers beside their main occupation. By reaching a financial independence (as it is the case for the CERN), these projects could optimize their functioning through the opening of jobs dedicated to these missions.

Authorship in the era of crowdsourced science

A researcher's career path in the academic world (i.e., job opportunities, grant attributions, etc.) depends on major contributions to research projects, which is often assessed through the amount of publications where one appears as the lead author. In psychology, it is common practice to list authors in order of contribution, with involvement decreasing as we move down the list. Truth is, it is not rare that multiple authors on the same paper contributed equally to the project, but in different ways. In that sense, contributors on a project do not always get the credit they deserve—as the order the authors are listed in does not capture well the contribution of each author. This is especially true for large-scale collaborative projects with many contributors (e.g., the PSA001 project has over 200 contributors).

An alternative to the current authorship system is the CRediT taxonomy, a taxonomy describing 14 distinctive categories (e.g., conceptualization of the project, administration of the project, funding acquisition, investigation) that represent the roles typically played by contributors in a scientific project. Papers relying on this taxonomy allow for a more representative description of the involvement of each contributor on a project.

Developing open-science and crowdsourced practices

The enrolment of students through different collaborative projects could foster open science and crowdsourced practices early in a researcher's career. For instance, in the Collaborative Replication and Education Project (CREP), students are taught the roots and importance of such practices toward the replication of recent major findings in psychology.

Editorial policies of scientific journals also play a role in the adoption of open science and crowdsourced practices, especially by defining new publication criteria. For instance, more than 200 journals now adopt an "in principle acceptance" format of peer-reviewing papers. In this publishing format, articles are accepted for publication prior to data collection, on the basis of the provided theoretical framework, methodology, and analysis plan.

Remaining issues

Flexibility in data analyses and other biases that collaborative projects should address by aggregating experts are not always overcome.

It has also been shown that crowdsourced projects involving low-trained and low-involved actors (e.g., students) can lead to the falsification of data. Linking up a wide array of contributors can thus imply structural problems that may impact research outcomes.

Both issues highlight the importance of educational practices on open science and crowdsourced practices (see also #Developing open-science and crowdsourced practices).

Controversies

Controversies surrounding crowdsourced science do not directly involve a criticism of crowdsourced science itself, but rather its costs—both in terms of financial and time investment. Collaborative practices in research remain very expensive and face an important number of challenges. Solutions to address these challenges require important structural changes within research institutions and have important repercussions on researchers' academic careers (see also #Challenges and future directions).

The shift from a vertical model toward a more horizontal one was partly motivated by the replication crisis in psychology. However, some authors are skeptical about the extent of this crisis in psychology. According to these authors, the failure to replicate most findings is overestimated and mostly due to a lack of fidelity in replication protocols. These claims mitigate whether large collaborative research projects are worth the cost, suggesting that a shift towards a horizontal model of doing science may not be necessary.

Given the cost of crowdsourced projects and the resources they require, crowdsourcing may not always be the most optimal approach. Nonetheless, the crowdsourced science approach helped the development of tools from which any project—either collaborative or not—can benefit. An optimal approach would be a compromise between both vertical and horizontal models, which would depend on the research question at hand and on the constraints of each project.