Practices:Delphi survey

From FORwiki

(Difference between revisions)
Jump to: navigation, search
(Undo revision 7640 by Iulia Maries (Talk))
(→Lessons learned)
 
(14 intermediate revisions not shown)
Line 1: Line 1:
-
{{stub}}
+
The '''Delphi Survey''' is a particular collaborative process that is designed to improve group communications about a complex problem or topic. The objectives of a Delphi process are three-fold: (1) to gather the information which is needed to deal with the problem or topic and fill in the resulting knowledge structure; (2) to make sure this information can be understood by the many different backgrounds of the contributors; (3) to expose agreements and disagreements and trying to come up with various recommendations for actions of various types.
-
Delphi method is a version of survey analysis that involves repetitive questioning of respondents. Delphi researchers aim to predict and explore alternative future developments, the probability of their occurrence and their desirability. Delphi studies are action-oriented meant to affect actions or thoughts of decision makers. Herman Kann developed the Delphi Survey Method within RAND Corporation in early 1960s.  
+
-
==The FOR-LEARN Guide to Expert Panels==
+
The Delphi method involves gathering of what might be a very large group of participants to consider a complex problem, usually about five people in each area of special knowledge or expertise needed to present and share information about the problem and various solutions to it. A knowledge structure allows the participants to place their comments, insights, and concerns in the appropriate location so a large involved discussion is easy to follow. Individual participants, usually anonymous when authoring items and when voting, have the ability to vote on contributions so the group can determine what specific things they agree or disagree on. In the past, the Delphi Survey was largely done by paper and pencil communications and is now often done on the Web. Since the computer process or paper process keeps track of the contributions, what each individual has contributes, what they have read or seen, every participant can participate asynchronously at a time and place convenient for them.  
-
''This is a summary of the article on the Delphi Method from the FOR-LEARN guide. To read the full article go [http://forlearn.jrc.ec.europa.eu/guide/4_methodology/meth_delphi.htm here].''
+
-
===Overall description===
+
== Definitional aspects ==
-
The Delphi method is based on structural surveys and makes use of information from the experience and knowledge of the participants, who are mainly experts. It therefore yields both qualitative and quantitative results and draws on exploratory, predictive even normative elements. Delphi is an expert survey in two or more 'rounds' in which, in the second and later rounds of the survey the results of the previous round are given as feedback.
+
''Main article: [[Definitional aspects of the Delphi method]]''
-
Delphi studies tackle issues formulated in statements about which uncertain and incomplete knowledge exists. They involve making judgments in the face of uncertainty by large number of experts selected based on knowledge and experience. The assumption is that self-fulfilling and self-destroying prophecies will be thus developed, thus shaping or even 'creating' the future. A Delphi study usually involves experts from business, government, research associations and any other stakeholders of a specific field under debate.  
+
The basic Delphi concept is the design of a collaborative communication structure and process that is tailored to the nature of the problem and the nature of the group. Anonymity of the responses is one fundamental property so that people will feel free to express themselves and to be able to expose ideas that could turn out to be stupid as well as brilliant. The typical view of Delphi is that it has a round structure and goes through at least three phases:
 +
<ol>
 +
<li> Exploring the problem and exposing new insights and additional relevant material.
 +
<li> Gaining a collective understanding of the material generated.
 +
<li> Evaluating the material and hopefully reaching a consensus.
 +
</ol>
-
===When is this method appropriate?===
+
Over the past forty years, a number of specific Delphi Structures have been designed and are very popular in terms of successful usage. This includes the conditional forecasting of trends where the emphasis is generating the conditions that affect the trend forecast.  A second is a problem solving Delphi structure to come up with an evaluated list of alternatives or options. The third is the Policy Delphi which is devoted to determining the alternative and complementary policy options to a policy issue and the arguments supporting each one.  The fourth is the example of Cross Impact Analysis for building individual and group models of interaction among future events and scenarios.  The specific area of cross impact analysis is a foundation for the creation of a Delphi based Planning process.  
-
A Delphi study is usually organized when there is a need to organize a debate, to collect and synthesize opinions and to achieve a degree of convergence. This is the case when there is not a lot of evidence about possible developments, or when long-term issues are involved. Also, common incentives for using the method are the intention to produce statistical significant results, or the will to involve a large number of people in processes.  
+
-
===Step by Step guide===
+
=== Delphi like processes ===
-
Developing a classical Delphi study usually involves running the following steps:
+
''Main article: [[Delphi like processes]]''
-
* Selection of the subject to forecast (one or more thematic fields);
+
-
* Definition of the procedure;
+
-
* Formulation of the statements and questions;
+
-
* Administration of the questionnaire;
+
-
* Analysis of responses. In the analysis of data are used descriptive statistics (median, inter-quartile range, etc.) in order to quantitatively summarize the set data and to anticipate possible developments of the characteristics / variables measured.
+
-
===Resources needed===
+
The name of Delphi was not chosen by the inventors of the method at RAND (Olaf Helmer and Norman Dalkey) but by their fellow professionals, since it was commonly used for future predictions. The strange name affixed to the Delphi process has not been favorable for the spread of this method.  What has happened as a result is that many of the premises of Delphi have been rediscovered or renamed under other methods to use group processes to try to obtain some level of collective intelligence.  This is the concept that the group can reach a higher quality result than any individual in the group would have acting alone<ref>Hiltz, S.R. and Turoff, M., The Network Nation: Human Communication via Computer, 1978 Addison Wesley, revised edition reprinted 1993 by MIT Press </ref>.  The most common Delphi derivatives today are: collaborative tagging or folksonomies and recommender systems, prediction markets, wikis and collaborative systems for humans.
-
The costs depends on the number of experts, the length of questionnaire, type of technology used. A Delphi can take between three weeks and 3-4 months. The organizers need management skills, neutrality and to be open to creativity.
+
-
===Pros and cons===
+
=== Examples of Delphi structures ===
-
Supporters of the Delphi study underline certain advantages in using the method:
+
''Main article: [[Examples of Delphi structures]]''
-
* It’s a credible and popular approach for policy-makers.
+
-
* It forces people to think about long term issues.
+
-
* It highlights clearly whether there is a consensus on an issue or not.
+
-
* The judgment allows for analyses, rankings and priority–settings.
+
-
At the same time, there are limits to using the method. Certain statements or forecasts cannot be assessed even by accomplished experts, making the Delphi study pointless. At the same time, if the Delphi is not well designed it will produce poor quality information and might compromise the entire Foresight activity. Other criticisms are aimed at the fact that Delphi studies are time-consuming, labour intensive and require expert preparation, or that many participants might drop-out during the process.
+
-
===Complementary methods===
+
There are a number of "classic" structures that have been used very successfully many times in the past forty years and have been the basis of a number of proprietary organizational studies.  They can each be used on a wide range of similar problems. Some of them have been utilized in online exercises using bulletin boards and auxiliary software such as survey packages. A few have been fully implemented in software.
-
The Delphi method implies identifying statements (topics) that are relevant for the future. Therefore, creativity methods (e.g. brainstorming, 6-3-5 or others), scenarios or key technology can be used in the preparatory phase to define these statements. Data from desktop studies: literature research, patent analysis or bibliometrics can be added. In the analytical phase, different modelling or statistical methods (calculation, rankings, correlations) or the re-building of scenarios as well as pseudo-roadmaps can be used. For comments or additional explanations, qualitative analyses are necessary. A SWOT analysis can be based on the results.
+
-
==Delphi Method==
+
The following are the general types of Delphi processes that apply to a large number of applications:
 +
* Trend Delphi: produces a forecast of a trend along with the mental model of the group making the extrapolation of the trend curve into the future.
 +
* Problem Solving Delphi: Collects solutions to the problem which are rescaled to a group interval scale based upon individuals ranking or paired comparisons.  Use voting to focus discussion on items that need it.
 +
* Policy Delphi: seeks policy resolutions and the strongest pro and con evidence or arguments to support each policy resolution.
 +
* Cross Impact Modeling: Collaborative building of a model of the future possible outcomes of a set of unique events.
-
===Introduction===
+
== EUFORIA Delphi Survey ==
-
Delphi is a particular collaborative process that is designed to improve group communications about a complex problem or topic.  In the past, it was largely done by paper and pencil communications and is now often done on the Web. The Delphi Method has the following properties:
+
''More about the [[Narrative:The_EUFORIA_project|EUFORIA Project]].''
-
* The gathering of what might be a very large group of participants to consider a complex problem (tens, hundreds, or thousands).
+
-
* The participants usually number about five people in each area of special knowledge or expertise needed to present and share information about the problem and various solutions to it.
+
-
* A knowledge structure allows the participants to place their comments, insights, and concerns in the appropriate location so a large involved discussion is easy to follow.
+
-
* There is also the ability to vote on contributions so the group can determine what specific things they agree or disagree on.
+
-
* Individual participants are usually anonymous when authoring items and when voting.
+
-
* Since the computer process or paper process keeps track of the contributions, what each individual has contributes, what they have read or seen, every participant can participate asynchronously at a time and place convenient for them.
+
-
There are three major objectives of a Delphi Process:
+
-
* Gathering the information which is needed to deal with the problem or topic and fill in the resulting knowledge structure.
+
-
* Making sure this information can be understood by the many different backgrounds of the contributors.
+
-
* Exposing agreements and disagreements and trying to come up with various recommendations for actions of various types.
+
-
There is a rich [[History]] of the Delphi Method and the authors understanding of it.  There is also [[References]] for much more specific information and examples of the method.
+
-
===Some Definitional Aspects===
+
The Euforia project was commissioned by the ''European Foundation for the Improvement of Living and Working Conditions'' in the context of its programme aimed at ''Analyzing and anticipating change to support socio-economic progress 2001-2004''. The project’s goal was to create a structured process by which questions related to living conditions, working conditions and industrial relations in a Knowledge Society might be illuminated. Euforia was launched as a pilot project taking three EU countries - Finland, Greece and Germany - as its test cases, designed to see how far foresight methods could be used to throw light on the relevance of KS concepts.  
-
The basic Delphi concept is the design of a collaborative communication structure and process that is tailored to the nature of the problem and the nature of the group (Linstone and Turoff, 1975). Although it was used largely in the early days for predicting future technical breakthroughs, it has been used to address a wide range of complex problems that are often current and it has been used subsequently to try to understand the past as well as the future. There have, for example, been a number of examples of experts in a given field using a Delphi to establish the most significant contributions to their field.
+
-
Anonymity of the responses is one fundamental property so that people will feel free to express themselves and to be able to expose ideas that could turn out to be stupid as well as brilliant. However, in some current online approaches it is possible to allow the participants, if they chose, to put in a comment with their true name, or when they want to be anonymous, or when they want to use a pen name.  An advantage of the pen name is that they can develop a series of comments to express a coherent viewpoint. In some cases, the respondents are told who is participating so they will feel they are part of a peer group of people they would like to communicate with about the particular topic. Usually those acting as the design team will commit to the fact that who said what would never be divulged to the other participants or to the sponsor. 
+
-
Delphis that are well done usually try to capture and seed the process with the material that can be found in the literature on the subject, so that those participating realize that they are not being asked to educate the design team on what should be obvious. The material to be asked of the respondents is what would be difficult to find in the literature and what is not obvious. Too many poor Delphis have attempted to give people a blank piece of paper that says "tell me what I should know about this problem!" 
+
-
Associated with the above is that people have to be motivated to put in the effort to participate in a Delphi exercise. The sort of motivation factors that have to be considered and made clear to the participants are:
+
-
* Is this an important problem that should be addressed by a larger group of experts who will all have an equal opportunity to contribute?
+
-
* Is this the right group to undertake this effort?
+
-
* Is someone or some organization going to make real use of the results of this effort?
+
-
* Is it worth it for me to spend the necessary time to make a good contribution?
+
-
* Will I learn things I should learn from those in other professional areas that are participating?
+
-
* Is it clear to me what the process is and what I will be committing to in time and effort?
+
-
* If some of the above is not true, am I going to be paid, and what is my time is worth to participate?
+
-
The typical view of Delphi is that it has a round structure and goes through at least three phases:
+
-
1. Exploring the problem and exposing new insights and additional relevant material.
+
-
2. Gaining a collective understanding of the material generated.
+
-
3. Evaluating the material and hopefully reaching a consensus.
+
-
This is usually what leads to a three round exercise for Delphi processes done via pencil and paper. Sometimes it does suffer because the design may lead to a premature consensus when there is not an adequate structure to expose hidden disagreements.  Sometimes the pressure is to get just quantitative subjective estimates of variables such as costs, likelihood of success, effectiveness, etc. without a sufficient design in the structure to expose hidden or underlying disagreements. Voting is often used as a conclusion rather than for its real purpose, which is to expose potential disagreements and get rid of possible ambiguities so that true uncertainties can be dealt with. This leads to a number of other requirements that when done with paper and pencil can require five rounds for the complete process. They add the following phases after phase one above.
+
-
1.1 Initial voting on generated material to expose disagreements.
+
-
1.2 Exploration of the underlying reasons for disagreements.
+
-
Underlying the above is the requirement to have a morphological structure for the information that is contributed that allows the participants to input their knowledge into appropriate categories that will organize and cluster information. Today this is referred to as a knowledge structure and it is exhibited in many of the Delphis that deal with complex problems.
+
-
Many of the concepts underlying the Delphi Process have been adopted in other related methods: Prediction Markets, Recommender Systems, Collaborative Tagging or Folksonomies, and other Collaborative Systems [[Examples of Delphi like processes]].
+
-
Over the past forty years, a number of specific Delphi Structures have been designed and are very popular in terms of successful usage. This includes the conditional forecasting of trends where the emphasis is generating the conditions that affect the trend forecast. A second is a problem solving Delphi structure to come up with an evaluated list of alternatives or options. The third is the Policy Delphi which is devoted to determining the alternative and complementary policy options to a policy issue and the arguments supporting each one. The fourth is the example of Cross Impact Analysis for building individual and group models of interaction among future events and scenarios. [[Example Delphi Structures]]
+
-
The specific area of cross impact analysis is a foundation for the creation of a Delphi based Planning process [[The foundation of Cross Impact Analysis]].
+
-
===Examples of Delphi like Processes===
+
A “Knowledge Society Delphi” Survey was run during the EUFORIA project. The survey covered a large range of topics concerning the Knowledge Society, emerging from national workshops or presenting a particular interest for the project’s sponsor. These topics covered the social, technological, economic, environmental, political and values-related aspects (STEEPV), and were clustered into six categories: governance and mobility, health and privacy, industrial relations, living conditions; sustainable development, and working conditions. The outputs of the Delphi Survey were used as inputs for national scenario workshops, as well as for preparing a full Delphi-based scenario.
-
"Treason doth never prosper: what's the reason? Why if it prosper, none dare call it treason." -- Sir John Harington 1561-1612
+
-
The name of Delphi was not chosen by the inventors of the method at RAND (Olaf Helmer and Norman Dalkey) but by their fellow professionals, since it was commonly used for future predictions. The strange name affixed to the Delphi process has not been favorable for the spread of this method. What has happened as a result is that many of the premises of Delphi have been rediscovered or renamed under other methods to use group processes to try to obtain some level of collective intelligence. This is the concept that the group can reach a higher quality result than any individual in the group would have acting alone (Hiltz and Turoff, 1978). The most common Delphi Derivatives today are:
+
-
* Prediction Markets
+
-
* Recommender Systems
+
-
* Collaborative Tagging or Folksonomies
+
-
* Wikis, and Collaborative Systems for humans
+
-
The idea of using a consensus on creating index tags for various objects by anonymous agreement on words to represent the object has become quite popular and goes under the name of collaborative tagging or Folksonomies. The PhD thesis that created the field of collaborative tagging was "espgame.org" which is still on the web and a wonderful experience to appreciate fully a two-person Delphi structure. A design for a recommender system for professionals in online "communities of practice" incorporating a dynamic Delphi structure for obtaining group preferences and using collaborative tagging was published in 2009 (Turoff and Hiltz). In many Delphis, it is desirable to break down the votes into subgroups by the characteristics of professionals or knowledgeable people involved. In the design presented in this recommender paper, we specifically call for rating the documents of interest to the given community through collaboratively tagging by the community as to the special topics the documents represent. The users should also be self-tagged by same index so that it is easier to make participants aware of what is a significant information and evaluation vote of interest to them on a personal basis. This is a desirable feature for any online continuous Delphi/Recommender system. In recommender systems like the product review system used for Amazon the product of concern is the tag that links purchasers and those who might purchase a given product. 
+
-
Norm Dalkey emphasized the concept that Delphis could ultimately produce very concise results that because of quantification would not suffer from ambiguity and the other problems facing face-to-face verbal communications. He would have viewed the prediction market as an excellent example of a type of Delphi process. Certainly, the prediction market has roots in the concept of financial markets, but what people tend to forget is that those markets are only as good as the people who are making the investments. In addition, there is a tremendous amount of qualitative material used by people who recommend to others what to do in such markets. They are anything but concise in their output of material, nor are they always accurate, especially in forecasting negative financial events of any size. Delphi processes are no better than the group that participates. The movie financial success prediction market is quite good since most of the thousands who play that market are extremely knowledgeable about movies. Another example is that NETFLIX can identify very small groups of individuals who make very similar movie choices, and use that to make recommendations among the members of that small group. This is a complete Delphi process with anonymity within the larger NETFLIX recommender system. They might someday decide to allow social networks to form within that context for those willing to reveal their identity to each other.
+
-
Wikis started out to be completely free for anyone to rewrite anyone else's material. Clearly, this did not work for any subject where disagreements existed. They have now evolved to have strong editing approval procedures like journal editors. They have evolved to be much more Delphi-like than their original conception. One of the most interesting examples is Wikimapia, which allows anyone to place information into a geographical database. There are number of examples where local governments allow citizens to place information directly into a database for the local community.  Most local governments do not have the resources or funds to do this when the data has to be maintained. In this case, the citizens can update entries when needed. This is used to create GPS databases relevant to emergency management and provide information on sites vulnerable to certain disasters and the locations of equipment that might be shared among the community in emergencies such as a contractor's earth moving equipment or possible shelter locations. 
+
-
The recent emergence of Social Networks that allow the users to form their own groups to share information of common interest to the group has lead to a large number of local community activities including sharing information relevant to an expected or ongoing emergency (Palen, Hiltz, and Liu, 2007, Vieweg et al, 2008, White, et al, 2008). The idea of groups within online Web systems goes back to the earliest days of Group Decision Support Systems on the Web (Hiltz and Turoff, 1978, 1993). The unfortunate situation with respect to Social Networks is that most of these systems are designed to serve a commercial objective and the functionality does not really include what could be designed to facilitate collaborative goals for applications like emergency management information systems, as has been demonstrated in the professional literature of that field (Turoff et al 2004).
+
 +
=== Statement-building methodology ===
 +
The process of building  Delphi statements followed a complex methodology.
-
==See also==
+
'''Step 1''': initial lists of topics produced during national workshops were consolidated into a single database with 172 entries;
 +
 
 +
'''Step 2''': 30 groups of topics with similar characteristics were identified;
 +
 
 +
'''Step 3''': individual topics within these groups were then annotated with the country of origin and whether or not they were among the recommended set from its country of origin, and thus the list was reduced a total of 136 topics;
 +
 
 +
'''Step 4''': sets of topics were then created, and those that were repetitions or were seen to be sufficiently similar were written as one topic, which further reduced the number of topics to 77;
 +
 
 +
'''Step 5''': the topics were then edited using conventional rules for writing Delphi statements, and the total number of statements was reduced again;
 +
 +
'''Step 6''': remaining topics were examined for their relevance, reasonableness and robustness, and a number of them were eliminated because they were considered to be well advanced already or lacking clarity;
 +
 
 +
'''Step 7''': the PREST team added five more to make the set up to the target of 30 topics;
 +
 
 +
'''Step 8''': each national centre was asked to name five topics from its original list, which they consider vital to their national location, thus raising the number of topics that appeared in the web-based ‘questionnaire’ to 35;
 +
 
 +
'''Step 9''': two more statements proposed by the project sponsor;
 +
 
 +
'''Step 10''': the statements were clustered into six categories, allowing participants to access only the questions related to one specific category and send their partial questionnaire.
 +
<ol>
 +
<li> ''governance and mobility:'' interactions between EU governments, citizens, and labor organizations; working force immigration and emigration; the use of information technologies by governments; etc;
 +
<li> ''health and privacy:'' EU policies on genetic engineering; civil liberties, health monitoring; DNA screening; creation of genetic databanks; etc.;
 +
<li> ''industrial relations'': the way in which EU industries manage their networks; monitoring and supervision through electronic means; employment contracts; role of trade unions; decision-making practices; etc.;
 +
<li> ''living conditions'': EU citizens’ behavior and live; ethics; justice; education; social isolation and loneliness; lifelong learning; work-life balance and family relations; role of ICT in everyday life; etc.;
 +
<li> ''sustainability and development'': EU governments’ policies on sustainable development; regional employment; business management practices; environmental technologies; wealth creation and quality of life; effects of EU enlargement; etc.;
 +
<li> ''working conditions'': EU governments’ policies on gender-related work inequalities; working time; forms of employment; organizational learning; violence and harassment at work; etc.
 +
</ol>
 +
 
 +
=== Assessment options ===
 +
The Delphi statements were exploratory option was chosen, asking how far a development will have progressed by the time horizon of 2015, a rough approximation to the time horizon of 2010 set in the recommendations of the Lisbon Council, rather than the more common form of by when a development will transpire. This would allow comparison between countries at different stages of development to be made more readily.
 +
 
 +
When responding to the Delphi statements, the respondents were offered a choice from the following five options<ref>European knowledge society foresight: The Euforia project synthesis, European Foundation for the Improvement of Living and Working Conditions, http://www.eurofound.europa.eu/</ref>:
 +
<ol>
 +
<li> the statement underestimates the situation by 2015;
 +
<li> the statement is about right by 2015;
 +
<li> the statement overestimates the situation by 2015;
 +
<li> the statement will not follow this path; and,
 +
<li> the statement is too uncertain (participant does not know or cannot provide any judgment on the development of the statement).
 +
</ol>
 +
 
 +
Respondents were asked to indicate the influence of each statement on each of nine factors related to the conclusions of the Lisbon Council and the Foundation’s mission. The specified factors were<ref>European knowledge society foresight: The Euforia project synthesis, European Foundation for the Improvement of Living and Working Conditions, http://www.eurofound.europa.eu/</ref>:
 +
<ol>
 +
<li> '''social cohesion''' - those features of society that relate to social integration and the reduction of conflict between or within social groupings;
 +
<li> '''social exclusion or divides''' - any matters that create or exacerbate inequality and inequity between or within social groupings including the ‘digital divide,’ access to education, gender and other equality related issues;
 +
<li> '''sustainability/environmental quality''' - those matters that influence the development of the natural and built environments in which future generations will live;
 +
<li> '''employer-employee relations''' - includes the role of trade unions, management, and employees including employment standards as set by regulations and directives;
 +
<li> '''economic growth/wealth creation''' - those matters that increase national income and strengthen the industrial base;
 +
<li> '''entrepreneurship and innovativeness''' - those matters that enable and promote new products, processes and services in existing businesses and the formation of new businesses in novel fields;
 +
<li> '''employee exercise of autonomy and responsibility at work''' - including the advancement of the quality of working life through freedom to make decisions, to exercise management of time and to embark on retraining;
 +
<li> '''work-life balance''' - those matters that enable people to manage the stresses caused by for example the pressures arising from longer and unsocial working conditions;
 +
<li> '''job creation''' - refers to the expansion of employment opportunities at all skill levels irrespective of gender.
 +
</ol>
 +
 
 +
For each factor, the influence of the statements were rated by selecting one of the following criteria:
 +
<ol>
 +
<li> the statement would strongly increase/improve the factor;
 +
<li> the statement would increase/improve the factor;
 +
<li> the statement would have no effect on the factor;
 +
<li> the statement would decrease/deteriorate the factor;
 +
<li> the statement would strongly decrease/deteriorate the factor.
 +
</ol>
 +
 
 +
=== The online survey ===
 +
The Delphi was run online. Difficulties in achieving a workable online Delphi resulted from its multilingual nature. A reasonable level of involvement in the survey was obtained from Greek and Finnish participants, while Germany had a disappointing level of about 20 participants. Many people took part from other countries (United Kingdom, Netherlands) bringing the total response rate to over 180. Organizers consider this number to be lower than desirable. According to them, a Delphi is not expected to involve a large enough sample to be representative of a population, but rather to tap into expert knowledge. However, the range of topics considered was such that a lot of people may be required to ensure that this condition was achieved.
 +
 
 +
=== Lessons learned ===
 +
The main concern was the lack of a committed body of respondents. It was a mistake to assume that a sufficient number of respondents would emerge from a light marketing approach. In any larger study it will be necessary to ensure that in each country there is an existing body of interested and willing respondents. A low percentage response rate resulted in a lack of appreciation of the viability of survey, particularly by people from a scientific background who expect statistically high response rates for the outcome to be of any value. Organizers believe a reasonable response rate should be in the region of 10% of the likely population.
 +
 
 +
Experience with the online Delphi did not confirm all initial expectations. One reason for the low overall response rate might have something to do with the fact that users were experiencing an upsurge of spam and viruses. Another factor may have been that the need to log in to a website to fill in a survey was just too much of a break from normal routines. Organizers believe that greater attention needs to be paid to the respondents’ level of skill with online working. More effort needs to be put into the preparations for an online Delphi, not just on the technical side, but in the management of the survey.
 +
 
 +
Translating the Delphi questionnaire proved to be a daunting task, and the particular Delphi format employed was a source of aggravation for some respondents. Translation can affect the quality and credibility of the Delphi process, since the original English nuances can easily be lost. This produces uncertainties in regard to the ways in which respondents interpret and answer the questions.
 +
 
 +
The results themselves were fairly typical and told an interesting story. However, organizers believe there is still a good deal of persistent misunderstanding about the interpretation of a Delphi survey’s output. Panels usually complain that they do not get unusual ideas and new information from the survey. Fortunately, that was not the case in the Euforia survey, where the workshops did not pose questions to seek confirmation of their established ideas and opinions.
 +
 
 +
== References ==
 +
<references/>
 +
 
 +
== Sea also ==
[[Practices:Environmental Scanning & Monitoring|Environmental Scanning & Monitoring]]<br>
[[Practices:Environmental Scanning & Monitoring|Environmental Scanning & Monitoring]]<br>
[[Practices:System Dynamics|System Dynamics]]<br>
[[Practices:System Dynamics|System Dynamics]]<br>

Current revision as of 04:37, 29 June 2010

The Delphi Survey is a particular collaborative process that is designed to improve group communications about a complex problem or topic. The objectives of a Delphi process are three-fold: (1) to gather the information which is needed to deal with the problem or topic and fill in the resulting knowledge structure; (2) to make sure this information can be understood by the many different backgrounds of the contributors; (3) to expose agreements and disagreements and trying to come up with various recommendations for actions of various types.

The Delphi method involves gathering of what might be a very large group of participants to consider a complex problem, usually about five people in each area of special knowledge or expertise needed to present and share information about the problem and various solutions to it. A knowledge structure allows the participants to place their comments, insights, and concerns in the appropriate location so a large involved discussion is easy to follow. Individual participants, usually anonymous when authoring items and when voting, have the ability to vote on contributions so the group can determine what specific things they agree or disagree on. In the past, the Delphi Survey was largely done by paper and pencil communications and is now often done on the Web. Since the computer process or paper process keeps track of the contributions, what each individual has contributes, what they have read or seen, every participant can participate asynchronously at a time and place convenient for them.

Contents

Definitional aspects

Main article: Definitional aspects of the Delphi method

The basic Delphi concept is the design of a collaborative communication structure and process that is tailored to the nature of the problem and the nature of the group. Anonymity of the responses is one fundamental property so that people will feel free to express themselves and to be able to expose ideas that could turn out to be stupid as well as brilliant. The typical view of Delphi is that it has a round structure and goes through at least three phases:

  1. Exploring the problem and exposing new insights and additional relevant material.
  2. Gaining a collective understanding of the material generated.
  3. Evaluating the material and hopefully reaching a consensus.

Over the past forty years, a number of specific Delphi Structures have been designed and are very popular in terms of successful usage. This includes the conditional forecasting of trends where the emphasis is generating the conditions that affect the trend forecast. A second is a problem solving Delphi structure to come up with an evaluated list of alternatives or options. The third is the Policy Delphi which is devoted to determining the alternative and complementary policy options to a policy issue and the arguments supporting each one. The fourth is the example of Cross Impact Analysis for building individual and group models of interaction among future events and scenarios. The specific area of cross impact analysis is a foundation for the creation of a Delphi based Planning process.

Delphi like processes

Main article: Delphi like processes

The name of Delphi was not chosen by the inventors of the method at RAND (Olaf Helmer and Norman Dalkey) but by their fellow professionals, since it was commonly used for future predictions. The strange name affixed to the Delphi process has not been favorable for the spread of this method. What has happened as a result is that many of the premises of Delphi have been rediscovered or renamed under other methods to use group processes to try to obtain some level of collective intelligence. This is the concept that the group can reach a higher quality result than any individual in the group would have acting alone[1]. The most common Delphi derivatives today are: collaborative tagging or folksonomies and recommender systems, prediction markets, wikis and collaborative systems for humans.

Examples of Delphi structures

Main article: Examples of Delphi structures

There are a number of "classic" structures that have been used very successfully many times in the past forty years and have been the basis of a number of proprietary organizational studies. They can each be used on a wide range of similar problems. Some of them have been utilized in online exercises using bulletin boards and auxiliary software such as survey packages. A few have been fully implemented in software.

The following are the general types of Delphi processes that apply to a large number of applications:

  • Trend Delphi: produces a forecast of a trend along with the mental model of the group making the extrapolation of the trend curve into the future.
  • Problem Solving Delphi: Collects solutions to the problem which are rescaled to a group interval scale based upon individuals ranking or paired comparisons. Use voting to focus discussion on items that need it.
  • Policy Delphi: seeks policy resolutions and the strongest pro and con evidence or arguments to support each policy resolution.
  • Cross Impact Modeling: Collaborative building of a model of the future possible outcomes of a set of unique events.

EUFORIA Delphi Survey

More about the EUFORIA Project.

The Euforia project was commissioned by the European Foundation for the Improvement of Living and Working Conditions in the context of its programme aimed at Analyzing and anticipating change to support socio-economic progress 2001-2004. The project’s goal was to create a structured process by which questions related to living conditions, working conditions and industrial relations in a Knowledge Society might be illuminated. Euforia was launched as a pilot project taking three EU countries - Finland, Greece and Germany - as its test cases, designed to see how far foresight methods could be used to throw light on the relevance of KS concepts.

A “Knowledge Society Delphi” Survey was run during the EUFORIA project. The survey covered a large range of topics concerning the Knowledge Society, emerging from national workshops or presenting a particular interest for the project’s sponsor. These topics covered the social, technological, economic, environmental, political and values-related aspects (STEEPV), and were clustered into six categories: governance and mobility, health and privacy, industrial relations, living conditions; sustainable development, and working conditions. The outputs of the Delphi Survey were used as inputs for national scenario workshops, as well as for preparing a full Delphi-based scenario.

Statement-building methodology

The process of building Delphi statements followed a complex methodology.

Step 1: initial lists of topics produced during national workshops were consolidated into a single database with 172 entries;

Step 2: 30 groups of topics with similar characteristics were identified;

Step 3: individual topics within these groups were then annotated with the country of origin and whether or not they were among the recommended set from its country of origin, and thus the list was reduced a total of 136 topics;

Step 4: sets of topics were then created, and those that were repetitions or were seen to be sufficiently similar were written as one topic, which further reduced the number of topics to 77;

Step 5: the topics were then edited using conventional rules for writing Delphi statements, and the total number of statements was reduced again;

Step 6: remaining topics were examined for their relevance, reasonableness and robustness, and a number of them were eliminated because they were considered to be well advanced already or lacking clarity;

Step 7: the PREST team added five more to make the set up to the target of 30 topics;

Step 8: each national centre was asked to name five topics from its original list, which they consider vital to their national location, thus raising the number of topics that appeared in the web-based ‘questionnaire’ to 35;

Step 9: two more statements proposed by the project sponsor;

Step 10: the statements were clustered into six categories, allowing participants to access only the questions related to one specific category and send their partial questionnaire.

  1. governance and mobility: interactions between EU governments, citizens, and labor organizations; working force immigration and emigration; the use of information technologies by governments; etc;
  2. health and privacy: EU policies on genetic engineering; civil liberties, health monitoring; DNA screening; creation of genetic databanks; etc.;
  3. industrial relations: the way in which EU industries manage their networks; monitoring and supervision through electronic means; employment contracts; role of trade unions; decision-making practices; etc.;
  4. living conditions: EU citizens’ behavior and live; ethics; justice; education; social isolation and loneliness; lifelong learning; work-life balance and family relations; role of ICT in everyday life; etc.;
  5. sustainability and development: EU governments’ policies on sustainable development; regional employment; business management practices; environmental technologies; wealth creation and quality of life; effects of EU enlargement; etc.;
  6. working conditions: EU governments’ policies on gender-related work inequalities; working time; forms of employment; organizational learning; violence and harassment at work; etc.

Assessment options

The Delphi statements were exploratory option was chosen, asking how far a development will have progressed by the time horizon of 2015, a rough approximation to the time horizon of 2010 set in the recommendations of the Lisbon Council, rather than the more common form of by when a development will transpire. This would allow comparison between countries at different stages of development to be made more readily.

When responding to the Delphi statements, the respondents were offered a choice from the following five options[2]:

  1. the statement underestimates the situation by 2015;
  2. the statement is about right by 2015;
  3. the statement overestimates the situation by 2015;
  4. the statement will not follow this path; and,
  5. the statement is too uncertain (participant does not know or cannot provide any judgment on the development of the statement).

Respondents were asked to indicate the influence of each statement on each of nine factors related to the conclusions of the Lisbon Council and the Foundation’s mission. The specified factors were[3]:

  1. social cohesion - those features of society that relate to social integration and the reduction of conflict between or within social groupings;
  2. social exclusion or divides - any matters that create or exacerbate inequality and inequity between or within social groupings including the ‘digital divide,’ access to education, gender and other equality related issues;
  3. sustainability/environmental quality - those matters that influence the development of the natural and built environments in which future generations will live;
  4. employer-employee relations - includes the role of trade unions, management, and employees including employment standards as set by regulations and directives;
  5. economic growth/wealth creation - those matters that increase national income and strengthen the industrial base;
  6. entrepreneurship and innovativeness - those matters that enable and promote new products, processes and services in existing businesses and the formation of new businesses in novel fields;
  7. employee exercise of autonomy and responsibility at work - including the advancement of the quality of working life through freedom to make decisions, to exercise management of time and to embark on retraining;
  8. work-life balance - those matters that enable people to manage the stresses caused by for example the pressures arising from longer and unsocial working conditions;
  9. job creation - refers to the expansion of employment opportunities at all skill levels irrespective of gender.

For each factor, the influence of the statements were rated by selecting one of the following criteria:

  1. the statement would strongly increase/improve the factor;
  2. the statement would increase/improve the factor;
  3. the statement would have no effect on the factor;
  4. the statement would decrease/deteriorate the factor;
  5. the statement would strongly decrease/deteriorate the factor.

The online survey

The Delphi was run online. Difficulties in achieving a workable online Delphi resulted from its multilingual nature. A reasonable level of involvement in the survey was obtained from Greek and Finnish participants, while Germany had a disappointing level of about 20 participants. Many people took part from other countries (United Kingdom, Netherlands) bringing the total response rate to over 180. Organizers consider this number to be lower than desirable. According to them, a Delphi is not expected to involve a large enough sample to be representative of a population, but rather to tap into expert knowledge. However, the range of topics considered was such that a lot of people may be required to ensure that this condition was achieved.

Lessons learned

The main concern was the lack of a committed body of respondents. It was a mistake to assume that a sufficient number of respondents would emerge from a light marketing approach. In any larger study it will be necessary to ensure that in each country there is an existing body of interested and willing respondents. A low percentage response rate resulted in a lack of appreciation of the viability of survey, particularly by people from a scientific background who expect statistically high response rates for the outcome to be of any value. Organizers believe a reasonable response rate should be in the region of 10% of the likely population.

Experience with the online Delphi did not confirm all initial expectations. One reason for the low overall response rate might have something to do with the fact that users were experiencing an upsurge of spam and viruses. Another factor may have been that the need to log in to a website to fill in a survey was just too much of a break from normal routines. Organizers believe that greater attention needs to be paid to the respondents’ level of skill with online working. More effort needs to be put into the preparations for an online Delphi, not just on the technical side, but in the management of the survey.

Translating the Delphi questionnaire proved to be a daunting task, and the particular Delphi format employed was a source of aggravation for some respondents. Translation can affect the quality and credibility of the Delphi process, since the original English nuances can easily be lost. This produces uncertainties in regard to the ways in which respondents interpret and answer the questions.

The results themselves were fairly typical and told an interesting story. However, organizers believe there is still a good deal of persistent misunderstanding about the interpretation of a Delphi survey’s output. Panels usually complain that they do not get unusual ideas and new information from the survey. Fortunately, that was not the case in the Euforia survey, where the workshops did not pose questions to seek confirmation of their established ideas and opinions.

References

  1. ↑ Hiltz, S.R. and Turoff, M., The Network Nation: Human Communication via Computer, 1978 Addison Wesley, revised edition reprinted 1993 by MIT Press
  2. ↑ European knowledge society foresight: The Euforia project synthesis, European Foundation for the Improvement of Living and Working Conditions, http://www.eurofound.europa.eu/
  3. ↑ European knowledge society foresight: The Euforia project synthesis, European Foundation for the Improvement of Living and Working Conditions, http://www.eurofound.europa.eu/

Sea also

Environmental Scanning & Monitoring
System Dynamics
Structural Analysis
Agent Modelling
SWOT Analysis
Trend Intra & Extrapolation
Modelling & Simulation
Gaming
Creativity Methods
Backcasting
S&T Roadmapping
Critical & Key Technology Study
Scenario Building
Morphological Analysis & Relevance Trees
Cross-Impact Analysis
Multi-Criteria Analysis

Retrieved from "http://forwiki.ro/wiki/Practices:Delphi_survey"
Personal tools