Web-based experiments
A web-based experiment or Internet-based experiment is an experiment that is conducted over the Internet. In such experiments, the Internet is either "a medium through which to target larger and more diverse samples with reduced administrative and financial costs" or "a field of social science research in its own right." Psychology and Internet studies are probably the disciplines that have used these experiments most widely, although a range of other disciplines including political science and economics also use web-based experiments. Within psychology most web-based experiments are conducted in the areas of cognitive psychology and social psychology. This form of experimental setup has become increasingly popular because researchers can cheaply collect large amounts of data from a wider range of locations and people. A web-based experiment is a type of online research method. Web based experiments have become significantly more widespread since the COVID-19 pandemic, as researchers have been unable to conduct lab-based experiments.
Introduction
Experiments are an integral part of research, however, their integration with the Internet has been gradual. There are three main categories of experiments:- Controlled experiments, done in a laboratory setting, attempt to control for all variables then test for a single effect.
- Natural experiments, conducted after a large-scale event which was prohibitively difficult or impossible to control, collect as many variables as possible then draw correlations.
- Field experiments, observed in a natural setting where less controls can be applied, have the advantage of better external validity.
Benefits
Web-based experiments are significantly less expensive, potentially allowing the researcher to:- Reach more diverse samples, as well as rare or specific sub-populations
- Run experiments more quickly
- Recruit larger subject pools that provide higher statistical power
- Conduct cross-cultural social experiments in real time
Criticisms and limitations
Web-based experiments may have weaker experimental controls compared to laboratory-based experiments, and may face greater difficulties coming up with procedures that ensure reliability and internal validity. Online natural and field experiments may also face challenges generalizing findings beyond the online context in which they were conducted. Some potential difficulties faced by web-based experiments include:- Difficulty verifying the identity of subjects participating in the experiment
- Experimental instructions ignored or read too carelessly, leading to lower quality data
- Shorter decision times of online participants triggering instinctive and emotional reasoning processes rather than cognitive and rational ones, which could cause subjects to make more pro-social decisions on average
- Significant distractions occurring during the course of the experiment unbeknownst to the researchers
- Subjects selectively dropping out of the experiment, especially if drop-out is correlated with the independent variable
- Variance in the data due to network connection speed and reliability, browser and computer types, screen size and resolution, etc.
- Subjects taking the experiment less seriously and behaving with less risk-aversion
- Subjects not believing that they are interacting with real human partners
- Subject concerns about compensation at the end of the experiment or anonymity of payment processing
- The non-representative nature of the mostly English-speaking computer users who participate
Studies have been conducted to test the internal validity of web-based experiments, comparing across experimental conditions and successfully replicating findings. For example, Schoeffler et al. compared laboratory- and web-based results of an auditory experiment and found no significant differences. A paired experiment in behavioral economics split into online and traditional lab environments produced substantively similar results. Uncompensated and unsupervised subjects on have been shown to replicate previous in-lab study results with comparable data quality.
Methodologies
Experimental protocols have been suggested to prevent or control difficulties associated with web-based experimentation. Methods like sequential subject matching, background timing and mouse use tracking, and instantaneous compensation through PayPal have the potential to address many of the concerns about the internal validity of web-based experiments. These methods control for differences in response times, address issues of selective attrition, concentration, and distraction, minimize subject concern about compensation, improve subject confidence that they have a real human partner in the experiment, and ensure that subjects have an appropriate understanding of the instructions and the decision problems in the experiment.Scholars have also formulated techniques to decrease or account for drop-outs, including the high-hurdle technique, the seriousness check, and the warm-up phase.
Examples
Use in psychology
A wide range of psychology experiments are conducted on the web. The Web Experiment List provides a way to recruit participants and archives past experiments. A good resource for designing a web experiment is the free Wextor tool, which "dynamically creates the customized Web pages needed for the experimental procedure" and is remarkably easy to use. Web experiments have been used to validate results from laboratory research and field research and to conduct new experiments that are only feasible if done online. Further, the materials created for web experiments can be used in a traditional laboratory setting if later desired.Interdisciplinary research using web experiments is rising. For example, a number of psychology and law researchers have used the web to collect data. Lora Levett and Margaret Bull Kovera examined whether opposing expert witnesses are effective in educating jurors about unreliable expert evidence. Rather than sensitizing jurors to flaws in the other expert's testimony, the researchers found that jurors became more skeptical of all expert testimony. In her experiment, this led to more guilty verdicts. Levett and Kovera's research used a written transcript of a trial, which participants then read before making their decision. This type of stimulus has been criticized by some researchers as lacking ecological validity—that is, it does not closely approximate a real-life trial. Many recommend the use of video where possible. Researchers at New York University are currently conducting a psychology and law study that uses video of a criminal trial.
Researchers at University of Salford are currently conducting a number of studies online to explore sound perception. Sound experiments over the web are particularly difficult due to lack of control over sound reproduction equipment.
Salganik, Dodds, and Watts conducted an experiment to measure social influence, specifically in the popularity rating of songs. Their use of the Internet allowed them to collect over 14,000 participants and examine the relationship between individual and collective behavior.