Practices:Delphi survey

From FORwiki

Revision as of 14:40, 26 April 2010 by Dan Grosu (Talk | contribs)
Jump to: navigation, search

Welcome to Foresight Wiki. Although everyone is welcome to make constructive contributions to Foresight Wiki, at least one of your recent edits did not appear to be constructive and has been reverted. Please use the sandbox for any test edits you would like to make. Thank you.

Delphi method is a version of survey analysis that involves repetitive questioning of respondents. Delphi researchers aim to predict and explore alternative future developments, the probability of their occurrence and their desirability. Delphi studies are action-oriented meant to affect actions or thoughts of decision makers. Herman Kann developed the Delphi Survey Method within RAND Corporation in early 1960s.


The FOR-LEARN Guide to Expert Panels

This is a summary of the article on the Delphi Method from the FOR-LEARN guide. To read the full article go here.

Overall description

The Delphi method is based on structural surveys and makes use of information from the experience and knowledge of the participants, who are mainly experts. It therefore yields both qualitative and quantitative results and draws on exploratory, predictive even normative elements. Delphi is an expert survey in two or more 'rounds' in which, in the second and later rounds of the survey the results of the previous round are given as feedback.

Delphi studies tackle issues formulated in statements about which uncertain and incomplete knowledge exists. They involve making judgments in the face of uncertainty by large number of experts selected based on knowledge and experience. The assumption is that self-fulfilling and self-destroying prophecies will be thus developed, thus shaping or even 'creating' the future. A Delphi study usually involves experts from business, government, research associations and any other stakeholders of a specific field under debate.

When is this method appropriate?

A Delphi study is usually organized when there is a need to organize a debate, to collect and synthesize opinions and to achieve a degree of convergence. This is the case when there is not a lot of evidence about possible developments, or when long-term issues are involved. Also, common incentives for using the method are the intention to produce statistical significant results, or the will to involve a large number of people in processes.

Step by Step guide

Developing a classical Delphi study usually involves running the following steps:

  • Selection of the subject to forecast (one or more thematic fields);
  • Definition of the procedure;
  • Formulation of the statements and questions;
  • Administration of the questionnaire;
  • Analysis of responses. In the analysis of data are used descriptive statistics (median, inter-quartile range, etc.) in order to quantitatively summarize the set data and to anticipate possible developments of the characteristics / variables measured.

Resources needed

The costs depends on the number of experts, the length of questionnaire, type of technology used. A Delphi can take between three weeks and 3-4 months. The organizers need management skills, neutrality and to be open to creativity.

Pros and cons

Supporters of the Delphi study underline certain advantages in using the method:

  • It’s a credible and popular approach for policy-makers.
  • It forces people to think about long term issues.
  • It highlights clearly whether there is a consensus on an issue or not.
  • The judgment allows for analyses, rankings and priority–settings.

At the same time, there are limits to using the method. Certain statements or forecasts cannot be assessed even by accomplished experts, making the Delphi study pointless. At the same time, if the Delphi is not well designed it will produce poor quality information and might compromise the entire Foresight activity. Other criticisms are aimed at the fact that Delphi studies are time-consuming, labour intensive and require expert preparation, or that many participants might drop-out during the process.

Complementary methods

The Delphi method implies identifying statements (topics) that are relevant for the future. Therefore, creativity methods (e.g. brainstorming, 6-3-5 or others), scenarios or key technology can be used in the preparatory phase to define these statements. Data from desktop studies: literature research, patent analysis or bibliometrics can be added. In the analytical phase, different modelling or statistical methods (calculation, rankings, correlations) or the re-building of scenarios as well as pseudo-roadmaps can be used. For comments or additional explanations, qualitative analyses are necessary. A SWOT analysis can be based on the results.

Delphi Method


Delphi is a particular collaborative process that is designed to improve group communications about a complex problem or topic. In the past, it was largely done by paper and pencil communications and is now often done on the Web. The Delphi Method has the following properties:

  • The gathering of what might be a very large group of participants to consider a complex problem (tens, hundreds, or thousands).
  • The participants usually number about five people in each area of special knowledge or expertise needed to present and share information about the problem and various solutions to it.
  • A knowledge structure allows the participants to place their comments, insights, and concerns in the appropriate location so a large involved discussion is easy to follow.
  • There is also the ability to vote on contributions so the group can determine what specific things they agree or disagree on.
  • Individual participants are usually anonymous when authoring items and when voting.
  • Since the computer process or paper process keeps track of the contributions, what each individual has contributes, what they have read or seen, every participant can participate asynchronously at a time and place convenient for them.

There are three major objectives of a Delphi Process:

  • Gathering the information which is needed to deal with the problem or topic and fill in the resulting knowledge structure.
  • Making sure this information can be understood by the many different backgrounds of the contributors.
  • Exposing agreements and disagreements and trying to come up with various recommendations for actions of various types.

There is a rich Practices:Delphi survey#History of the Delphi Method and the authors understanding of it. There is also Practices:Delphi survey#References for much more specific information and examples of the method.

Some Definitional Aspects

The basic Delphi concept is the design of a collaborative communication structure and process that is tailored to the nature of the problem and the nature of the group (Linstone and Turoff, 1975). Although it was used largely in the early days for predicting future technical breakthroughs, it has been used to address a wide range of complex problems that are often current and it has been used subsequently to try to understand the past as well as the future. There have, for example, been a number of examples of experts in a given field using a Delphi to establish the most significant contributions to their field. Anonymity of the responses is one fundamental property so that people will feel free to express themselves and to be able to expose ideas that could turn out to be stupid as well as brilliant. However, in some current online approaches it is possible to allow the participants, if they chose, to put in a comment with their true name, or when they want to be anonymous, or when they want to use a pen name. An advantage of the pen name is that they can develop a series of comments to express a coherent viewpoint. In some cases, the respondents are told who is participating so they will feel they are part of a peer group of people they would like to communicate with about the particular topic. Usually those acting as the design team will commit to the fact that who said what would never be divulged to the other participants or to the sponsor. Delphis that are well done usually try to capture and seed the process with the material that can be found in the literature on the subject, so that those participating realize that they are not being asked to educate the design team on what should be obvious. The material to be asked of the respondents is what would be difficult to find in the literature and what is not obvious. Too many poor Delphis have attempted to give people a blank piece of paper that says "tell me what I should know about this problem!" Associated with the above is that people have to be motivated to put in the effort to participate in a Delphi exercise. The sort of motivation factors that have to be considered and made clear to the participants are:

  • Is this an important problem that should be addressed by a larger group of experts who will all have an equal opportunity to contribute?
  • Is this the right group to undertake this effort?
  • Is someone or some organization going to make real use of the results of this effort?
  • Is it worth it for me to spend the necessary time to make a good contribution?
  • Will I learn things I should learn from those in other professional areas that are participating?
  • Is it clear to me what the process is and what I will be committing to in time and effort?
  • If some of the above is not true, am I going to be paid, and what is my time is worth to participate?

The typical view of Delphi is that it has a round structure and goes through at least three phases:

  • Exploring the problem and exposing new insights and additional relevant material.
  • Gaining a collective understanding of the material generated.
  • Evaluating the material and hopefully reaching a consensus.

This is usually what leads to a three round exercise for Delphi processes done via pencil and paper. Sometimes it does suffer because the design may lead to a premature consensus when there is not an adequate structure to expose hidden disagreements. Sometimes the pressure is to get just quantitative subjective estimates of variables such as costs, likelihood of success, effectiveness, etc. without a sufficient design in the structure to expose hidden or underlying disagreements. Voting is often used as a conclusion rather than for its real purpose, which is to expose potential disagreements and get rid of possible ambiguities so that true uncertainties can be dealt with. This leads to a number of other requirements that when done with paper and pencil can require five rounds for the complete process. They add the following phases after phase one above.

  • Initial voting on generated material to expose disagreements.
  • Exploration of the underlying reasons for disagreements.

Underlying the above is the requirement to have a morphological structure for the information that is contributed that allows the participants to input their knowledge into appropriate categories that will organize and cluster information. Today this is referred to as a knowledge structure and it is exhibited in many of the Delphis that deal with complex problems. Many of the concepts underlying the Delphi Process have been adopted in other related methods: Prediction Markets, Recommender Systems, Collaborative Tagging or Folksonomies, and other Collaborative Systems Practices:Delphi survey#Examples of Delphi like Processes. Over the past forty years, a number of specific Delphi Structures have been designed and are very popular in terms of successful usage. This includes the conditional forecasting of trends where the emphasis is generating the conditions that affect the trend forecast. A second is a problem solving Delphi structure to come up with an evaluated list of alternatives or options. The third is the Policy Delphi which is devoted to determining the alternative and complementary policy options to a policy issue and the arguments supporting each one. The fourth is the example of Cross Impact Analysis for building individual and group models of interaction among future events and scenarios. Practices:Delphi survey#Example Delphi Structures The specific area of cross impact analysis is a foundation for the creation of a Delphi based Planning process The foundation of Cross Impact Analysis.

Examples of Delphi like Processes

"Treason doth never prosper: what's the reason? Why if it prosper, none dare call it treason." -- Sir John Harington 1561-1612 The name of Delphi was not chosen by the inventors of the method at RAND (Olaf Helmer and Norman Dalkey) but by their fellow professionals, since it was commonly used for future predictions. The strange name affixed to the Delphi process has not been favorable for the spread of this method. What has happened as a result is that many of the premises of Delphi have been rediscovered or renamed under other methods to use group processes to try to obtain some level of collective intelligence. This is the concept that the group can reach a higher quality result than any individual in the group would have acting alone (Hiltz and Turoff, 1978). The most common Delphi Derivatives today are:

  • Prediction Markets
  • Recommender Systems
  • Collaborative Tagging or Folksonomies
  • Wikis, and Collaborative Systems for humans

The idea of using a consensus on creating index tags for various objects by anonymous agreement on words to represent the object has become quite popular and goes under the name of collaborative tagging or Folksonomies. The PhD thesis that created the field of collaborative tagging was "" which is still on the web and a wonderful experience to appreciate fully a two-person Delphi structure. A design for a recommender system for professionals in online "communities of practice" incorporating a dynamic Delphi structure for obtaining group preferences and using collaborative tagging was published in 2009 (Turoff and Hiltz). In many Delphis, it is desirable to break down the votes into subgroups by the characteristics of professionals or knowledgeable people involved. In the design presented in this recommender paper, we specifically call for rating the documents of interest to the given community through collaboratively tagging by the community as to the special topics the documents represent. The users should also be self-tagged by same index so that it is easier to make participants aware of what is a significant information and evaluation vote of interest to them on a personal basis. This is a desirable feature for any online continuous Delphi/Recommender system. In recommender systems like the product review system used for Amazon the product of concern is the tag that links purchasers and those who might purchase a given product. Norm Dalkey emphasized the concept that Delphis could ultimately produce very concise results that because of quantification would not suffer from ambiguity and the other problems facing face-to-face verbal communications. He would have viewed the prediction market as an excellent example of a type of Delphi process. Certainly, the prediction market has roots in the concept of financial markets, but what people tend to forget is that those markets are only as good as the people who are making the investments. In addition, there is a tremendous amount of qualitative material used by people who recommend to others what to do in such markets. They are anything but concise in their output of material, nor are they always accurate, especially in forecasting negative financial events of any size. Delphi processes are no better than the group that participates. The movie financial success prediction market is quite good since most of the thousands who play that market are extremely knowledgeable about movies. Another example is that NETFLIX can identify very small groups of individuals who make very similar movie choices, and use that to make recommendations among the members of that small group. This is a complete Delphi process with anonymity within the larger NETFLIX recommender system. They might someday decide to allow social networks to form within that context for those willing to reveal their identity to each other. Wikis started out to be completely free for anyone to rewrite anyone else's material. Clearly, this did not work for any subject where disagreements existed. They have now evolved to have strong editing approval procedures like journal editors. They have evolved to be much more Delphi-like than their original conception. One of the most interesting examples is Wikimapia, which allows anyone to place information into a geographical database. There are number of examples where local governments allow citizens to place information directly into a database for the local community. Most local governments do not have the resources or funds to do this when the data has to be maintained. In this case, the citizens can update entries when needed. This is used to create GPS databases relevant to emergency management and provide information on sites vulnerable to certain disasters and the locations of equipment that might be shared among the community in emergencies such as a contractor's earth moving equipment or possible shelter locations. The recent emergence of Social Networks that allow the users to form their own groups to share information of common interest to the group has lead to a large number of local community activities including sharing information relevant to an expected or ongoing emergency (Palen, Hiltz, and Liu, 2007, Vieweg et al, 2008, White, et al, 2008). The idea of groups within online Web systems goes back to the earliest days of Group Decision Support Systems on the Web (Hiltz and Turoff, 1978, 1993). The unfortunate situation with respect to Social Networks is that most of these systems are designed to serve a commercial objective and the functionality does not really include what could be designed to facilitate collaborative goals for applications like emergency management information systems, as has been demonstrated in the professional literature of that field (Turoff et al 2004).

Example Delphi Structures

There are a number of "classic" structures that have been used very successfully many times in the past forty years and have been the basis of a number of proprietary organizational studies. They can each be used on a wide range of similar problems. Some of them have been utilized in online exercises using bulletin boards and auxiliary software such as survey packages. A few have been fully implemented in software. These are the original Policy Delphi structure (Turoff, 1972) and most recently the Problem Solving Delphi (White et al, 2007). An online Delphi is extremely dynamic and what used to take months using paper and pencil processes can now be done in a few days or a few weeks. Some particular urgent real time problems such as those in the area of Emergency Management can be dealt with in less than an hour by small groups of 10 to 20 dispersed participants. In an online continuous Delphi process, any participant at any time may:

  • Focus on the particular topic they want to deal with
  • Be informed of changes since he or she was last using the system
  • Be able to create new material, new comments, new options to vote on, etc.
  • Be able to vote or change their vote
  • Defer voting until there is more information about an item to make a judgment
  • Allow lists of items to which additions can be made or current items edited
  • Arrange such lists dynamically according the attached voting scale results
  • View vote results to show differences by voters with different backgrounds
  • Provide dynamic collaborative tagging to create new classifications of items and lists
  • Show the status of voting with respect to number voting and vote changes for each item as well as the vote distribution.

The following are the general types of Delphi processes that apply to a large number of applications:

  • Trend Delphi: produces a forecast of a trend along with the mental model of the group making the extrapolation of the trend curve into the future.
  • Problem Solving Delphi: Collects solutions to the problem which are rescaled to a group interval scale based upon individuals ranking or paired comparisons. Use voting to focus discussion on items that need it.
  • Policy Delphi: seeks policy resolutions and the strongest pro and con evidence or arguments to support each policy resolution.
  • Cross Impact Modeling: Collaborative building of a model of the future possible outcomes of a set of unique events.

Each of the above has specific characteristics that are summarized in the following tables:

Table 1: Trend Delphi
System Functions Participants Responses System Actions on Responses
Present a historical trend to be extrapolated by the participants Draw a future curve or redraw a new one when a change has occurred in viewpoint. Present summary of 50% median and 0%, 25%, 75% and 100% boundaries
Request assumptions and uncertainties used to make above estimate; Turn all these into potential assumptions Vote on validity scale for each potential assumption. Scale is from completely true to completely false. Reorder assumptions from true to false. Focus on middle range (maybe) and ask which can be influenced or measured for occurrence
Assume these can reduce the future uncertainty in the curve; Ask for a redrawing of curve extrapolation based upon assumption list for each trend curve in the study Supply suggestions on how to influence or measure the "maybe" assumptions causing significant uncertainty in the projected curve. Summarize important findings at any time: trend, true and false assumptions, assumptions that cause uncertainty, and their potential actions, and measurements
Table 2: Problem Solving Delphi
System Functions Participants Responses System Actions on Responses
State the problem and request solution options Provide options to solve the problem Present options in order of occurrence
Request paired comparisons to measure individual preferences for options Make comparisons for option pairs that a participant feels confident about judging at any time. Use Thurstone's law of comparative judgment (using incomplete information) to derive a single group interval scale. Calculate uncertainty due to those who have not yet voted with same type of scale.
Show interval scale: this indicates disagreements when two or more items are close together. This also shows clustering. Ask for comments about items where people disagree from vote. Make comments about items you want to see others change their votes about. Present discussions about items for review. As more people vote or change votes, scales will reflect decreasing uncertainty and often more separation between options.
Table 3: Policy Delphi
System Functions Participants Responses System Actions on Responses
State a policy issue to be examined. Ask for specific policy solutions Add resolution options or specific policies Request vote for Desirability and Feasibility scales of each solution
Plot two dimensional distribution of policy resolutions; Exploring desirable but infeasible solutions often important Request comments especially about those showing disagreement Request comments about policy resolutions. Indicate if comment is pro, con, or neutral.
Request vote on comments for importance and validity (It might be considered important because others believe it to be true) A person may think a comment is important because others think it is valid. Provide updated two dimensional plots and summarize discussions
Table 4: Cross Impact Modeling
System Functions Participants Responses System Actions on Responses
Use problem solving Delphi to produce a set of future unique events focused on a given situation Evaluate those events for their relative importance to the future objective guiding the choice of events Place the final most important events into a cross impact model
Ask each individual to answer: what are the probabilities of each event occurring in some future time frame? Tell them for each event that they should assume it will or will not occur and ask them to express any changes in the probabilities of the other events due to that certain knowledge about the future. Show them the expected outcome of their model, which will have differences from their predictions. Allow them to vary initial probabilities to see how the future changes. Allow them to go back and modify some of their estimates Create the cross impact model using the approach by Turoff (logistic, Fermi Dirac equations). This provides a scale that changes nonlinear probabilities (0 to 1) to a linear influence factor (Cij) between each pair of events (plus to minus infinity). When participants are satisfied with their individual model, utilize the internal linear influencing factors (Cij) to create a group model.

Cross Impact is the one of the most challenging areas of interface design today as it is the concept of allowing users to build their own models without having to program the model The Foundations of Cross Impact Analysis.

Table 5: Cross Impact Modeling Creating Scenarios
System Functions Participants Responses System Actions on Responses
Analyze the internal parameters to show people which of the relationships between which events show the most disagreement among the group. Ask for comments on these combinations from those who have inconsistent or extreme views. View these comments and changes to some of their original estimations. Create a model of interacting scenarios by voting on where to stop the integration of the events in process that can turn all the events into one scenario. When no more changes are being made, use Interpretive Structural Modeling to generate a set of macro scenarios collecting individual events that are tightly coupled into a set of scenarios that interact. Requires human monitor to know when to trigger the scenario creation part.

Delphi has been in active use since its invention in the 1960's. Unfortunately, there are probably more examples of unsuccessful Delphi exercises than successful ones in that it sounds like a simple process – it is anything but that. There is a lot of effort and careful planning to do a successful one with a quality group of participants. Some of the best Delphis over the years were unfortunately proprietary and were never published. There have also been some very interesting controlled experiments. One experiment shows that online Delphi exercises are more productive with respect to improvisation and creativity that exhibits novelty is than unstructured online discussions without voting (Chao et al, 2003, 2004). Another shows that the only time middle managers in a major corporation are willing to discuss prior past decisions as possible mistakes in planning the future of the company is when they can be anonymous in an online discussion rather than using their real name (Hiltz, Turoff and Johnson, 1989).


I entered the Delphi field in a circuitous way via astrophysics and software engineering when I became, in 1968, a civil servant as an operations research analyst at the Office of Emergency Preparedness (OEP) in the Executive Offices of the President of the U.S. I designed and executed an early paper and pencil Delphi on the Future of the Steel and Ferroalloy industry in the United States (Goldstein, 1975). When our new agency director General Lincoln (Eisenhower's logistic general in World War II and a West Point Professor) saw that result, he brought to me the following issue (paraphrased from a verbal discussion): "When I ask the bureaucracy to investigate a policy issue, they seem to try to guess what my decision will be and then deliver me something justifying that assumed decision. What I need to know, even if I know what the decision is I have to make, is what will be the worst counter positions against that decision, and what are the possible answers to those." This observation is what led to the Policy Delphi, which was designed as a Hegelian type debate through pencil and paper to expose all the different policy options and the most important rationales that were pro or con for each option. This was designed not to promote a consensus, as it requires choosing a respondent group representing many different viewpoints and encouraging a debate on those viewpoints. Each policy resolution was voted on for both desirability and feasibility since it was important to learn about desirable solutions that were felt by some significant sized group to be undesirable. Similarly, rationales presented to justify opposing positions were voted on for both importance and validity as some people might feel a rationale is important because others believed it to be true. The hope is that out of these conflicting worldviews a synthesis might result, by proposing new resolutions evolving out of rationales that had some common agreement among different groups. The communication structure for the original policy Delphi via paper and pencil was the online system design for an online Delphi process (Turoff, 1970). That system did not have a round structure when it was implemented online.

The Future of Delphi

The future of the concepts of Delphi is quite good even if they may not appear under the same name. Improving large group processes and capturing the collective knowledge that can aid in addressing complex problems is still an open ended goal. The development of large scale continuous planning for organizations is one excellent example of an area needing much more in the way of both field trails and case studies. The active area today that is tied to Delphi is that of creating knowledge structures. All successful Delphis usually have a designed knowledge structure to capture and categorize the information provided by the participants. Defining the morphology for the complex problem the Delphi is dealing with is extremely important to easing the potential for cognitive overload among the participants and making clear how different contributions relate to one another (Turoff, 1991). This has always been a factor of concern in the area of Computer Mediated Communications (CMC) and related areas such as online learning using CMC facilities. Typically, one attempts to develop the morphology to make it easy for the participants to arrive at a common understanding of the contributed information and the voting scales. It requires examining the literature in the areas represented by the problem area and the various jargon used by the different experts participating to arrive at an understandable structure. For example, dealing with policy issues the following is a typical structure that could be used.

 Goals and objectives
     Underlying assumptions
          Environmental Considerations
          Regulations and Constraints
     Policy Issues
          Policy Resolutions
          Management and Administrative Practices
          Potential Resource Allocations
          Decision Options

When entering a new item the participant chooses the appropriate heading or key subject phrases. While the above appears linear, the relationships among these different categories can be from any one line to any other line even though there are some assumed major and sub categories. The various application examples in the Delphi Method book (Linstone and Turoff, 1975) all present highly tailored categorical constructions associated with the topic addressed. The principal objects of discourse in typical Delphis can be illustrated in the following table:

Table 6: Example Objects of Discourse in Delphi Designs
Objects Examples Type
Problems, issues, questions Main concerns
Goals, Objectives, plans Normative formulations
Strategies, policies, agendas, approaches Solution management
Concerns, criteria, arguments, assumptions, viewpoints, options, values, interests, constraints Underlying factors
Consequences, scenarios, impacts Evaluation factors
Tradeoffs, compromises, proposals, solutions, allocations, decisions, projects, tasks, actions, options Possibilities

In addition, one tries to impose ordered categorical scales for voting that are anchored. One objective is to give meaning to each point on the scale or a majority of points. Another objective is to provide a meaningful results and anchoring points on a scale, which tends to remove any real possibility of Arrow's Paradox (1963). Another point is that you would like the middle value on the scale to be similar to a concept of "no judgment" so the scale is two sided and will capture any differences of view. For example, potential assumptions might have the following validity scale:

Table 6: Example Validity Scale in Delphi
Scale Term Definitions
1 Certain to be true No decision based upon this assumption will be wrong
2 Likely to be true Only a small chance the decision will be wrong if based upon this assumption
3 Might be true Some risk a decision will be wrong if based upon this assumption
4 Uncertain (Maybe) Should not make a decision based only on this assumption
5 Might be false A real risk that decision will be wrong if based upon this assumption
6 Likely to be false Considerable risk the decision based upon this assumption will be wrong
7 Certain to be false This assumption should not be used to justify any decision

Clearly, we want to include pragmatic explanations of what a given point on the scale means. Therefore, in the context of this particular example we have tied the validity of a potential assumption to its appropriateness to support the making of a decision. Since we are not trying to measure an abstract physiological state of a respondent we do not use the sort of scales that are meant to not change or modify what is being solicited. We are actually trying to stimulate the person to think more carefully about the estimates they are making and maybe to recall information that will change their first impressions of the item being evaluated. The success of a complex Delphi is often measured by how much change of views and votes occur in the process. We want to encourage people to think about their responses and not to make quick response with out thought that is behind the design of many consumer type surveys and polls. Today, a number of the bulletin board systems allow authors a choice to use their regular names or anonymous entry of a comment. Coupling this with the use of survey systems to allow the monitor or facilitator to create feedback surveys on the ideas and views as they occur, one has a straightforward way of doing simple Delphi discussions on the Web. However, there are growing threats to the ability of individuals to remain anonymous on the Web, especially on an international scale. Today there are significant attacks on the concept of anonymity that attempt to take away the right of individuals to engage in pen name ("handle") or anonymous contributions to these systems. There are continual attacks by organizations seeking to expose the authors of anonymous comments so they can discover if they are employees who are belittling their organization, and who they would like to fire or sue. So far, the resulting court cases have defended anonymity but in the United States, it has not yet reached the Supreme Court. The gathering of data about individuals that is useful for commercial, political, insurance, dictatorial, or charitable purposes is so immense, both nationally and internationally on the Web that it presents considerable dangers that anonymity will disappear as an option for large collaborative systems. A system like Facebook is attempting to create an environment where they can trace whatever their users are doing elsewhere on the Web so to be able to make money on the resulting information. The protections we used to attribute to communications over the phone have never been adequately extended to the Web. The Web has become as important to the individual citizen just as the phone once was, but our law makers do not yet comprehend that truth. We are facing in the decades ahead severe challenges to maintain and improve the current state of our societies at any level. Whether it is at the community, organizational, national, or international level, we need the ability of individuals to contribute their knowledge of problems that exist, and which need to be solved, as well as their expertise in many different areas. We need people who can point out problems that need to be addressed to be able to do so without incurring retribution from those causing the problem and we need people from many different backgrounds to be able to contribute to their solution. We also need those that can validate problems or solutions to be able to contribute as well, without fear of retribution.


The treatment of cross impact analysis by utilizing the transitions of probabilities to a linear scale has a very wide variety of potential application areas. It also provides a very appropriate methodology for observing the consistency of the human judgment of those creating the overall model. Of no less importance is the ability to allow individuals to first develop and evolve their personal models before they are contributed to the resulting collective model. This is a very significant step for the future improvement of collaborative model building. It has the merit of eliminating ahead of time potential ambiguities in the communication process if everyone is utilizing the same event set. The result is that the differences that occur are more likely to be due to uncertainties that can be resolved by a discussion process that focuses on discussions of the differences of viewpoints that have emerged. Because all this can be put on computer system as an integrated planning processes where new knowledge leads to new events and resulting scenarios that can be treated as macro events, it is thus possible to develop both long and short term plans as a continuous process by a large collaborative group. In fact, the distinction between short and long term plans becomes meaningless and the interval of time that is required is whatever is sufficient to gather the appropriate contributions for and from an evolving database of events, scenarios, policies, potential investments, objectives, and resulting models. This will not come easily because it represents not only a new methodology for most organizations but also a belief in the benefits that could result from continuous collaborative planning using the ability to incorporate all the knowledgeable individuals in the organization that should be contributing to the future of their organization. This also requires a personal dedication of the participants to the future of the organization they represent.


Some of the references below may be useful to follow up for more information even if they are not explicitly referenced in the paper.

  • Arrow K. J., (1963), Social choice and individual values, 2nd edition, Yale University Press.
  • Bañuls, V. A. B., Turoff, M., and Silva, J. L., Clustering Scenarios using Cross-Impact Analysis, accepted ISCRAM 2010, Seattle, Washington, May. Contact an author for a copy.
  • Bañuls, V.A. and Salmeron, J.L. (2007) Benchmarking the Information Society in the Long Range, Futures, 39, 1, 83–95.
  • Bañuls, V.A., and Salmeron, J.L. (2007), A Scenario-based Assessment Model - SBAM, Technological Forecasting and Social Change, 74, 6, 750-762
  • Chermack, T.J., (2004), Improving Decision-making with Scenario Planning, Futures, 36, 3, 295-309.
  • Chermack T.J. (2007) Disciplined imagination: Building scenarios and building theories, Futures, 39, 1, 1-15.
  • Cho, H.K. & Turoff, M., “Delphi Structure and Group Size in Asynchronous Computer-Mediated Communications,” Proceedings of the Americas Conference on Information Systems, Tampa, August 2003.
  • Cho, H. K. (2004) The effect of Delphi structure on small and medium-sized asynchronous groups, Ph. D. Dissertation, January 2004, New Jersey Institute of Technology, Information Systems Department. [1]
  • Cho, H. K., Murray Turoff, and Starr Roxanne Hiltz, The Impact of Delphi Communication on Small and Medium Sized Asynchronous Groups: Preliminary Results, HICSS 36, January 2003, IEEE Computer Society Press.
  • Cho, K.T., and Kwom, C.S. (2004) Hierarchies with dependence of technological alternatives: a cross impact hierarchy process, European Journal of Operational Research, 156, 2, 420–432.
  • Choi, C., Kim, S. and Park, Y. (2007), A patent-based cross impact analysis for quantitative estimation of technological impact: The case of information and communication technology, Technological Forecasting and Social Change, 74, 8, 1296 -1314.
  • Dalkey, N. C., Brown, H., and Cochran, S. W., The Delphi Method IV: Effect of Percentile Feedback and Feed-In or Relevant Facts, RM-6118 PR, March 1970.
  • Dalkey, N. (1975) Toward a Theory of Group Estimation, in the Delphi Method: Techniques and Applications, eds. H. A. Linstone and M. Turoff, Addison Wesley, page 236-261.
  • Duval, A., Fontela, E., and Gabus, A. (1974) Cross Impact: A Handbook of Concepts and Applications, in Portraits of Complexity, Application of Systems Methodologies to Societal Problems, Battelle-Geneva, Geneva.
  • Futures Group, (1994), Scenarios, UNU´s Millennium Project Feasibility Study.
  • Godet, M. (1994) From Anticipation to Action: a Hand Book of Strategic Prospective, UNESCO Publishing, Paris.
  • Goldstein, N., A Delphi on the future of the steel and Ferroalloy Industries, Part III, C3, in the Delphi Method, edited by Lindstone and Turoff, 1975.
  • Harries, C. (2003) Correspondence to What, Coherence to What? What is Good Scenario-based Decision Making? Technological Forecasting and Social Change, 70, 8, 797-817.
  • Hendela, A., Turoff, M., Hiltz, S. R. Cross Impact Security Analysis using the HACKING Game, accepted for ISCRAM 2010, Seattle Washington, May.
  • Hiltz, S. R., Turoff, Murray, & Johnson, K., (1989), Experiments in Group Decision Making, 3: Disinhibition, Deindividuation, and Group Process in Pen Name and Real Name Computer Conferences, Decision Support Systems, 5, 217-232. Full report and other related experimental reports on the NJIT Library archive website at the end of the reference list.
  • Hiltz, S.R. and Turoff, M., The Network Nation: Human Communication via Computer, 1978 Addison Wesley, revised edition reprinted 1993 by MIT Press
  • Hopkins, R.H., K.B. Cambell, and N.S. Peterson, Representations of Perceived Relations among the Properties and Variables of a Complex System, IEEE Transactions on Systems, Man and Cybernetics, (SMC-17:1), January/February 1987, 52-60.
  • Hsu, E. Y. P., Hiltz, S. R., and Turoff, M. (1992). Computer-Mediated Conferencing System as Applied to a Business Curriculum: A Research Update. In V. S. Jacob and H. Pirkul, eds., The Impact of Information Technology on Business Schools: Research, Teaching and Administration, Proceedings of the 20th Annual North American Conference of the International Business School Computer Users Group, pp. 214- 227. Awarded "Best Paper- Teaching."
  • Kane, J., A Primer for a New Cross-Impact Language—KSIM (with Examples Shown from Transportation Policy), in the Delphi Method Book, editors Linstone and Turoff.
  • Lendaris, G. G. (1980), "Structural Modeling - A Tutorial Guide," IEEE Transactions on Systems, Man, and Cybernetics SMC-10(12): 34.
  • Linstone, H. and Turoff, M., the Delphi Method: Techniques and Applications by Harold Linstone and Murray Turoff, 1975. [2]
  • Palen, L. Hiltz, S.R., and Liu, S. (2007) Citizen Participation in Emergency Preparedness and Response, Communications of the ACM special issue, 50, 3, 54-58.
  • Plotnick, Linda, Elizabeth Avey Gomez, Connie White, Furthering Development of a Unified Emergency Scale Using Thurstone's Law of comparative Judgment: A progress Report, Proceedings of ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management, Delft, the Netherlands, May 13-16, Brussels University Press
  • Turoff, Murray (2009), The Past, Present, and Future of Delphi, In: FUTURA, The Quarterly Journal of the Finnish Society for Futures Studies, ISSN 0785-5494 Helsinki, 32-44.
  • Turoff, M., Hiltz, S.R.: The Future of Professional Communities of Practice. In: Weinhardt, C., Luckner, S., Stößer, J. (eds.) WeB 2008. LNBIP, vol. 22, pp. 144-158. Springer-Verlag, Berlin Heidelberg (2009).
  • Turoff, M., and S. R. H., Information Seeking Behavior and Viewpoints of Emergency Preparedness and management professionals concerned with Health and Medicine, Final Report for the National Library of Medicine, March 6, 2008, available at [3].
  • Turoff, M., S. R. Hiltz, X. Yao, Z. Li, Y. Wang, and H. K. Cho, Online Collaborative Learning Enhancement Through the Delphi Method, Turkish Online Journal of Distance Education-TOJDE April 2006 ISSN 1302-6488 Volume: 7 Number: 2 Article: 6, Publisher: Anadolu University, Eskisehir, Turkey, [4]
  • Turoff, M., Hiltz, R., Bieber, M., Rana, A., Fjermestad, J., Collaborative Discourse Structures in Computer Mediated Group Communications, HICSS 32, 1999. Reprinted in Special Issue of Journal of Computer Mediated Communications on Persistent Conversation, Volume 4, Number 4, 1999, [5]
  • Turoff, Murray and S. R. Hiltz, (1995), Computer Based Delphi Processes, in Michael Adler and Erio Ziglio, editors., Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, London, Kingsley Publishers, pp. 56-88.
  • Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H. "The Delphi Process as a Collaborative Learning Method." In (edited by J. C. Moore) Elements of Quality Online Education: Into the Mainstream: Wisdom from the Sloan Consortium, 121-134. Needham, MA: Sloan-C, September 2004
  • Turoff, M. (1972) An Alternative Approach to Cross Impact Analysis, Technological Forecasting and Social Change 3, 309-339, also in the Delphi Method book.
  • Turoff, M., The Design of a Policy Delphi, Technological Forecasting and Social Change 2, No. 2 (1970), also in the Delphi Method book, 1975.
  • Turoff, Murray, (1970) Delphi Conferencing: Computer Based Conferencing with Anonymity, Journal of Technological Forecasting and Social Change 3(2), 1970, 159-204.
  • Turoff, M. (1991), Computer-mediated communication requirements for group support, Journal of Organizational Computing, vol. 1, number 1, pp 85-113.
  • Vieweg, S, Palen, L., Liuk, S., Hughes, A., and Sutton, J., Collective Intelligence in Disaster: Examination of the Phenomenon in the Aftermath of the 2007 Virginia Tech Shooting, ISCRAM, Washington, D.C. 2008.
  • Wang, Y., Li, Z., Turoff, M. and Hiltz, S.R. (2003), Using a social decision support system toolkit to evaluate achieved course objectives, Proceedings of the Americas Conference on Information Systems, Tampa, August. (Nominated as a “best paper.”)
  • Warfield, J. N., Societal Systems, Wiley, New York, 1976.
  • Warfield, J. N., Annotated Mathematical Bibliography for Interpretive Structural Modeling, May, 1992, [6]
  • Weimer-Jehle, W. (2006) Cross-impact balances: A system-theoretical approach to cross impact analysis, Technological Forecasting and Social Change, 73, 334-336.
  • White, C., Plotnick, L., Adams-Moring, R., Turoff, M., and Hiltz, S.R. Leveraging A Wiki to Enhance Collaboration in the Emergency Domain, 41st Hawaii International Conference on System Sciences, (HICSS) 2008.
  • White, Connie, Murray Turoff, Bartel Van de Walle, A Dynamic Delphi Process Utilizing a Modified Thurstone Scaling Method: Collaborative Judgment in Emergency Response, Proceedings of ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management, Delft, the Netherlands, May 13-16, Brussels University Press
  • Worrell, W., Hiltz, S. R., Turoff, M. and Fjermestad, J., (1995), An experiment in collaborative learning using a game and a computer-mediated conference in accounting games, Proceedings of the 28th Annual Hawaii International Conference on System Sciences, Vol. IV, pp. 63-71. Los Alamitos, CA: IEEE Computer Society Press, 1995.
  • Yao, X., Turoff, M., and Chumer, M. J. (2009), Designing Collario for continuous reviewing and practicing of emergency plans to ensure complex system safety, Proceedings of the 6th Conference on Information Systems for Crisis Response and Management (ISCRAM'09), Gothenburg, Sweden.
  • Yao, X., Turoff, M., and Hiltz, S. R., (2010), A field trial of a Collaborative Online Scenario Creation System for Emergency Management, accepted for ISCRAM 2010, Seattle, May, []

See also

Environmental Scanning & Monitoring
System Dynamics
Structural Analysis
Agent Modelling
SWOT Analysis
Trend Intra & Extrapolation
Modelling & Simulation
Creativity Methods
S&T Roadmapping
Critical & Key Technology Study
Scenario Building
Morphological Analysis & Relevance Trees
Cross-Impact Analysis
Multi-Criteria Analysis

Personal tools