Real-time Delphi

History

The basic idea of a real-time, therefore computer-based (usually web-based), Delphi approach originates in a paper published by Turoff back in 1972 about an online Delphi conference conducted in the United States. The conference was characterized by remote locations of participants, an online tool to access and give judgments, anonymity of the participants, continuous operations and analysis of results (i.e. participants were able to see given answers of the other participants in real-time), as well as asynchronous participation (i.e. participants could independently login and logout as often and when they desired). The stated aspects are some of the key characteristics of Real-Time Delphi studies, which shows that the original idea of conducting such studies can be traced back to the respective year. Today, nevertheless, technological innovations and advanced computer aided design possibilities (e.g. high-speed internet connections, high definition graphic, and advanced processor performance) facilitate more sophisticated studies in this context. The general idea to develop a faster advanced form of Delphi studies by using ideas and basic concepts of Turoff, was initiated by the U.S. Defense Advanced Research Projects Agency (DARPA), which awarded a grant in 2004 to develop an approach to improve "speed and efficiency of collecting judgments in tactical situations". A small software company named Articulate Software in San Francisco was awarded an innovation research grant to develop what DARPA was asking for. Adam Pease, principal consultant and CEO of Articulate Software, published the findings and methodology together with Theodore Gordon in 2006.

Based on the findings in this seminal paper, Heiko von der Gracht developed a second Real-time Delphi platform until 2008. In subsequent years, von der Gracht and colleagues used the platform extensively for prospective research studies. In 2011, von der Gracht and Gordon presented a joint paper at WFS Annual Summit 2011, which summarizes the lessons learned of 40 RTD studies across the two platforms.

In 2017, Angenheyster and colleagues published a study, which compares various Real-time Delphi platforms. Analysis criteria include among others visual appearance, data output, user friendliness, administration, and types of questions. A cross-disciplinary study by Beiderbeck et al. focused on new directions and advancements of the Delphi method, including real-time formats. The authors provide a methodological toolbox for designing Delphi surveys including among others sentiment analyses of the field of psychology.

Differences between conventional and real-time Delphi method

The question arises how a Real-Time Delphi study differs from a Conventional Delphi study. The basic framework is to think of a Delphi study which is conducted in the form of an online questionnaire. However, a Conventional round-based Delphi study conducted via the internet is called "Internet Delphi". The basic difference to Internet Delphi is that the process of a Real-Time Delphi is not characterised by single iterated rounds. In fact, real-time calculation and provision of responses are the key characteristics of Real-Time Delphis. Various other labels for Real-Time Delphi can be found in literature and many authors are not completely aware of the differences: "Electronic Delphi", "Computer Delphi", "Computer-aided Delphi", and "Technology Delphi". However, it is important to truly understand the design and process a researcher has chosen to find out whether real-time calculations and provisions have been applied or not.

The typical Real-Time Delphi process can be described in the way that participants get access to an online questionnaire portal for a certain time frame, within which they are allowed to log in and log out as often as they want. Whenever they login, they will see all their quantitative and qualitative answers of previous sessions and they can change all answers as desired within the given period of time. Besides their own answers they will see the ongoing – hence, real-time – responses of other participants, and with regard to metric assessments the group as a whole will be visualised in terms of median, average, and interquartile range (IQR). In modern RTD platforms, the group's distribution is conveniently presented visually in the form of a histogram, sometimes overlay on the rating slider. It has to be pointed out that the numerical visualisations as well as the qualitative inputs change in the course of other participants changing their responses. Consequently, a participant can find out to what extent his own responses from an earlier point of time are still within the group opinion (i.e. IQR). The core innovation, then, of Real-Time Delphi studies is the real-time calculation and provision of results.

Methodological advancements

The core methodological innovation of Real-Time Delphi studies are the absence of iterated rounds and the real-time calculation and provision of group responses. Whereas Conventional Delphi studies are characterised by repeating sequential rounds, the Real-Time Delphi approach is characterised by a continuous round-less procedure leading to a reduced time frame needed to conduct such studies. Consequently, conducting large-scale studies of huge complexity in a relatively short period of time becomes possible. Another core methodological innovation is the fact that experts may not only judge once or twice, depending on the number of rounds, as it has been usual in a Conventional Delphi study. During a Real-Time Delphi, experts can independently reassess their responses as often as they want.

Hartman and Baldwin discuss further advantages of the Real-Time Delphi approach: First, the number of experts participating in the real-time study can be increased due to a higher degree of automation during and improved possibilities for analysis after the study. Additionally, the Internet provides the possibility to invite a worldwide expert panel to participate in the study. Second, the degree of interaction among the experts can be increased due to the fact that they can immediately react on others' comments. Additionally, the time frame between giving own answers and getting insights into others' responses is very short, which encourages stronger cognitive examination with the respective issue in question. Hartman and Baldwin argue that with the help of this procedure the validity of results is maximised.

In order to conduct a Real-Time Delphi study, computer software – usually web-based – is needed for facilitating real-time calculations and visualisation of results. It is generally proposed in the existing literature that the experts participating in the study see not only their own answers but also the median and interquartile range of all given responses immediately after answering a quantitative question. Besides the quantitative assessment a qualitative judgment of participants can be shown which serves as a justification for their numerical assessment of the question. Additionally, it can be shown to the expert how many respondents have already given their answers. To examine the qualitative arguments of others participants can click on a button, and a "reasons window" opens, which shows the statements of other participants to underline their point of view. So, the respective legitimisations given by others may cause a respondent to recapture his own point of view. In the next step the expert can change his own answer, add new arguments to underline his point of view, or leave his answer unchanged. In addition, the respondent will be shown an attention indicator, a so-called "flag", if his answer is within or outside the interquartile range or significantly different from the median. This application helps to see and understand immediately the own assessment and to think about reasons for the deviation from group opinions or else a high degree of consent. The respondent's attention will be called by highlighting questions with a high degree of deviation with a different colour and by asking him to give further reasons for his deviation from the group opinion.

After operating a question in the described procedure, the participant can continue to the next question or press a "save"-button in the program, which leads to an immediate update of the median, interquartile range, and given arguments, and then leave the program. In modern platforms, the action of pressing a "save" -button may also not be required, to lessen then number of clicks/interactions with the platform which contribute to participant fatigue. A second advantage of the round-less approach is the fact that, in order to take part in the study, participants can login and logout with their personalised account as often as they want during the time frame provided. Their already given answers will be saved and recalled when they login the next time. So, by design of the study, there are no explicit single rounds to answer the questions. Updating and playing back the information to the other participants follow immediately in succession to the process of answering. Here, it becomes clear that the process of answering can be synchronous or asynchronous and a worldwide expert panel can be reached, which is one of the major advantages of web-based tools. Turoff and Hiltz argue that the issue of asynchronous interaction is probably one of the least understood characteristics of Real-Time Delphis. Zipfinger points out two advantages of asynchronous participation of the experts: First, they can login to the portal whenever they want; therefore, one could argue that the degree of convenience of taking part is increased for the participant due to a 24-hour availability of the portal. Second, panellists can contribute to whatever aspects in the questionnaire they want, especially when having gone through each question at least once. Here, a substantial aspect of Real-Time Delphis becomes obvious: Turoff and Hiltz explain that a Real-Time Delphi study offers a design of structured communication which allows every individual to choose the sequence and speed to contribute to the problem solution process. So, in comparison to face-to-face discussions, the Real-Time Delphi approach gives room for individuality and different cognitive abilities of the participants.

A further advantage is the fact that the administrator of the study can set an arbitrary time frame in which participants have to log in and take part in the questionnaire. So, whenever the researcher or administrator of the study is satisfied with the existing answers (i.e. in terms of quantity and quality), he can declare the study to be ended and close the online tool (i.e. "freeze" the responses).

The key features of Delphi studies, such as anonymity, controlled feedback and group response, are also met in the context of Real-Time Delphi studies. However, the issue of iteration is, by design of the technique, not valid for Real-Time Delphi studies anymore. Instead of answering each question a first time and getting a second sheet with the group responses in the second round, the Real-Time Delphi already shows the second screen (i.e. group responses) immediately after answering each question. Having answered each question at least once, the participant can usually control which question to reassess from a "consensus portal", which serves as a kind of control panel to access single questions again. So, on the one hand, the procedure differs from a Conventional Delphi and, on the other hand, the iteration into single rounds is missing.

Having asked the question how the accomplishment of a Real-Time Delphi study differs from conducting a usual Delphi study, Gordon and Pease point out that a Real-Time Delphi study can be implemented via a site on the Internet or in any other network (e.g. intra-company network, local area network) and is, therefore, not conducted in paper-and-pencil form any more.

As with all Delphi studies, the process of defining and selecting experts is still extremely important. The Conventional Delphi study is then divided into several steps of response round, analysis through the facilitator, playing back the information, next response round, and so on. However, the Real-Time Delphi study is, after granting access to the online tool, rather a self-running process.

The basic strengths of a Real-Time Delphi study are its efficiency and applicability to all Delphi topics (i.e. common problem sets, decision making issues, cross impact studies, etc.). Figure 2 illustrates that the process of a Real-Time Delphi differs. Important is to point out that the number of interventions of the facilitator needed during the response phases (i.e. after opening the online tool) are usually less. Having developed the online tool in advance, the intermediate analysis done by the facilitator of the study is rather uncomplicated in comparison to the Conventional Delphi. The overall shortened time period needed to conduct a real-time study underlines that the approach can be regarded as generally more efficient.

Gordon and Pease point out that a Real-Time Delphi study is applicable for a wide range of possible circumstances under which the consultation of experts is necessary. On the one hand, the authors give the example of a "small group operating synchronously in a conference room with laptop computers connected wirelessly to the web site where the software resides, with anticipated completion of the exercise in say 20 min.". On the other hand, it can be thought of a larger panel of experts operating asynchronously from remote locations within a longer period of time.

The greatest weakness of the real-time approach is that it is missing a wholly integrated, scientifically founded concept. The real-time Delphi idea is still a very new concept, which requires further research and application to become a tool for full-scale operations. Especially the editing of the alpha (i.e. the first) inputs of respondents, the real-time presentation of group results, and the tracking of progress over time should be integrated in a kind of administrator package to make the accomplishment of a Real-Time Delphi less difficult.

Examples of real-time Delphi applications

Numerous examples for real-time Delphi applications can be found. Among them, The Millennium Project conducted by Glenn, Gordon, & Florescu in 2009 provides a context for global thinking and improved understanding of global issues, opportunities, challenges, and strategies. The Millennium Project originally used a real-time Delphi platform developed by Gordon and has now shifted to a new generation platform, designed to integrate qualitative and quantitative data, and facilitate robust dialogue minimizing participant fatigue.

Another stream of projects based on real-time Delphi studies was conducted by the former Institute for Futures Studies and Knowledge Management of EBS University of Business and Law in Germany, among them the T&L2030 Series of five different studies together with PricewaterhouseCoopers: Transportation & Logistics 2030 Series. Further illustrative examples in academic journals include cases from logistics, social business, professional services, retail, and automotive.