Diary studies

Background

Diary studies originate from the fields of psychology and anthropology. In the field of human–computer interaction (HCI), diary studies have been adopted as one method of learning about user needs towards designing more appropriate technologies.

Temporal processes in diary studies

A key characteristic of diary studies is their ability to track daily events over time. Researchers have begun conducting studies that allow them to explore the connection between a previous day's events and a current day's outcome, or to what extent prior events make people responsive to current events. Researchers Robert E. Wickham and C. Raymond Knee have concluded that future research studies would benefit from evaluating temporal processes, or processes related to time, in diary studies. This would serve as a way for researchers to test unique questions and hypotheses.

Evaluating in-person change in diary

Researchers have been able to use diary studies to evaluate how people can change over time. Traditional diary studies have evaluated change between individuals, but new studies have been conducted to evaluate within-person changes using diary studies. Through a framework of the generalizability theory, researchers have used a condensed version of the Profile of Mood States (POMS) to study within-person emotional changes via diaries.

Types of diary studies

Feedback studies

Feedback studies involve participants answering predefined questions about the phenomenon of interest in a natural setting, with the answers acting as a diary entry. This is usually at assigned times, frequencies, or occurrences of the phenomenon, stated by the researcher. The most common method is using ‘paper and pencil’, although there are some studies that both utilise and suggest other technologies such as mobile phones or Psion Organisers. As such, feedback studies involve asynchronous communication between the participants and the researchers, as the participants’ data is recorded in their diary first, and then passed on to the researchers once complete.

Feedback studies are scalable - that is, a large-scale sample can be used, since it is mainly the participants themselves who are responsible for collecting and recording data.

Elicitation studies

In elicitation studies, participants capture media as soon as the phenomenon occurs - the media is usually in the form of a photograph, but can be in other different forms as well, and so the recording is generally quick and less effortful than feedback studies. These media are then used as prompts and memory cues - to elicit memories and discussion - in interviews that take place much later. As such, elicitation studies involve synchronous communication between the participants and the researchers, usually through interviews.

In these later interviews, the media and other memory cues (such as what activities were done before and after the event) can improve participants’ episodic memory. In particular, photos were found to elicit more specific recall than all other media types.

Comparisons

There are two prominent trade-offs between each type of study. Feedback studies involve answering questions more frequently and in situ, therefore enabling more accurate recall but more effortful recording. In contrast, elicitation studies involve quickly capturing media in situ but answering questions much later, therefore enabling less effortful recording but potentially inaccurate recall.

When to use diary studies

Diary studies are most often used when observing behavior over time in a natural environment. They can be beneficial when one is looking to find new qualitative and quantitative data. Diary studies aim to measure people's behavior over an extended period. They provide the opportunity for exploratory research, collecting large quantities of precise data that are both in-depth and contextual. What makes diary studies particularly unique is that these substantial amounts of data are collected at the micro-level. When the subject of research undergoes a change, utilizing diary studies becomes interesting because they allow for the measurement of change over an extended period and the observation of its effects on the individual; this is referred to as within-subject analysis. Additionally, it is possible to conduct between-subjects analysis to observe the different effects among respondents. A diary study offers the advantage over a traditional survey study in that it allows for the collection of data on a daily basis or even multiple times a day. In contrast, a survey study typically gathers data at a single point in time, or in the case of a longitudinal study, with time lags spanning months or years.

Advantages

Advantages of diary studies are numerous. They allow:

• collecting longitudinal and temporal information;
• reporting events and experiences in context and in-the-moment;
• participants to diary their behaviours, thoughts and feelings in-the-moment thereby minimising the potential for post rationalisation;
• determining the antecedents, correlations, and consequences of daily experiences and behaviors.
• respondents are not coerced
• regular reporting leads to a rich data collection
• an autonomous report is created, with no influence of social desirability
• it is unobtrusive: there is little distortion due to setting or investigations
• it is suitable for Small-n, medium-n, or even large-n (depending on the purpose and analysis method)
• it is a unique window on human phenomenology

Limitations

There are some limitations of diary studies, mainly due to their characteristics of reliance on memory and self-report measures.

There is low control, low participation and there is a risk of disturbing the action. In feedback studies, it can be troubling and disturbing to write everything down. This is called a respondent burden.

Inaccurate recall

The validity of diary studies rests on the assumption that participants will accurately recall and record their experiences. This is somewhat more easily enabled by the fact that diaries are completed & media is captured in a natural environment and closer, in real-time, to any occurrences of the phenomenon of interest. However, there are multiple barriers to obtaining accurate data, such as:

• Social desirability bias - where participants may answer in a way that makes them appear more socially desirable. This may be more prominent in longitudinal studies where participants frequently communicate with researchers.
• Recall bias - where participants recall events or feelings inaccurately due to systematic errors. Human memory is unreliable, as it is an active reconstruction which is susceptible to errors and biases. For example, after recording their emotions in daily diaries, older adults tended to recall more positive emotions than younger adults, while younger adults tended to recall more negative emotions than older adults.
• The phenomenon of interest itself - participants may find it difficult to both record and experience the phenomenon accurately at the same time. For example, a diary study assessing the drinking urges of recently treated alcoholics found that, if/when participants began to drink alcohol, they tended to stop recording in their diary.
• Reactive bias - As people are required to keep a diary, they may start thinking differently. This can lead to the Hawthorne effect.

Non-compliance and retrospective entries

Diary studies tend to dictate specific times or a frequency at which the participant should complete their diary entries or capture media. This is usually a time that is close to the event or phenomenon of interest to the researcher, and it is assumed that participants will comply with these instructions. However, researchers often find it difficult to verify the extent to which this assumption is met, and have observed various forms of non-compliance. One example of this is hoarding, where previously missed entries are 'backfilled' or completed all at once, therefore being completed retrospectively rather than at the assigned time. A further example of this is where entries are 'forward-filled', i.e. recorded for a future date.

Researchers have found significant rates of non-compliance and entries written retrospectively in feedback studies. A study by Hyland and colleagues (1993) estimated that the percentage of errors in paper and pencil diaries could be anywhere from 2-24%, due to diaries written retrospectively and/or due to inaccuracies in recording. Another study by Stone and colleagues (2003) compared paper diaries versus compliance-enhancing electronic diaries. The paper diaries contained a hidden instrument which detected when the diary was opened - from this, actual compliance rate was found to be only 11%, while 79% of entries were faked, i.e. not written within 30 minutes of the assigned time. In contrast, the electronic diaries created timestamps for each entry, which prevented the possibility of fake timestamped entries, and therefore yielded a compliance rate of 94%.

Notable diary studies

In 2015, a diary study was conducted at a Dutch University of Applied Sciences to evaluate how learning spaces affect students' learning activities. 52 business management students kept records of the learning activities that they worked on, where they worked on them, and why they worked there for a week. Through evaluating the diary entries of this study, researchers found a significant correlation between the spaces in which students chose to work and their learning activities.

Tools

PACO is an open source mobile platform for behavioral science.

StudyRun is a commercial platform that offers diary-study capability through scheduled ESM/EMA surveys plus photo and audio diary tasks on iOS and Android, combined with background smartphone sensor data.