CENTER FOR COLLECTIVE INTELLIGENCE

Working Paper Series

How can people and computers be connected so that-collectively-they act more intelligently than any individuals, groups, or computers have ever done before?

ACHIEVING COLLECTIVE INTELLIGENCE VIA LARGESCALE ON-LINE ARGUMENTATION Mark Klein CCI WORKING PAPER 2007-001 MIT SLOAN SCHOOL OF MANAGEMENT WORKING PAPER 4647-07 http://ssrn.com/abstract=1040881

Sloan School of Management 3 Cambridge Center, NE20-336 © 2007 Klein

Cambridge, MA 02139-4307 U.S.A. http://cci.mit.edu/

Achieving Collective Intelligence via Large-Scale On-line Argumentation Mark Klein Center for Collective Intelligence Massachusetts Institute of Technology Cambridge MA [email protected] Abstract—While argumentation tools represent a promising avenue for exploiting the Internet’s vast potential for enabling collective intelligence, they have to date been applied almost exclusively to small-scale, usually centrally facilitated, groups. This paper explores some of the issues and design options involved in supporting large-scale on-line argumentation. Argumentation; scalability; collective intelligence

I.

THE CHALLENGE: APPLYING COLLECTIVE INTELLIGENCE TO SYSTEMIC PROBLEMS

Humankind now finds itself faced with a range of what we can call systemic problems, i.e. vastly complex challenges like climate change that affect every one of us and are affected by every one of our actions. Such problems call for us to be able to engage in effective deliberations on a global scale. The spectacular emergence of the Internet has enabled unprecedented opportunities for such interactions - via email, instant messaging, news groups, chat rooms, blogs, wikis, podcasts, and the like - on a scale that was impossible a few short years ago. To date, however, such large-scale interactions have been incoherent and dispersed, contributions vary widely in quality, and there has been no clear way to converge on well-supported decisions concerning what actions humanity should take to solve their most pressing problems. Can we do a better job of harnessing the vast collective intelligence now potentially available to us? This paper explores this question, reviewing the weaknesses of current technologies, and exploring the challenges involved in developing more effective solutions based on argumentation support technology. II.

LIMITATIONS OF CURRENT TECHNOLOGIES

Let us define “collective intelligence”, for this context, as the synergistic and cumulative channeling of the efforts of many minds towards selecting actions in response to some challenge, i.e. as large-scale deliberation [1]. How well does current technology enable this? Generate Ideas Evaluate Ideas Expert markets Prediction markets GDSS (brainstorming) e-Voting Conversation tools (email, IM …) Open forums (wikis, blogs, …) Argumentation tools

By far the most commonly used technologies include synchronous and asynchronous conversation tools (e.g. email and instant messaging) as well as open forums (e.g. wikis and blogs). While such tools allow interactions on a global scale, they face serious shortcomings from the standpoint of enhancing collective intelligence. The content captured by such tools is notorious for often being unsystematic, highly repetitive, and of highly variable quality. At its best (as with Wikipedia), a carefully nurtured community process can be effective at capturing short descriptive articles about noncontroversial topics, but all these approaches tend to break down when faced with the need to come up with coherent responses to complex problems that involve many competing perspectives. In such cases, discussions can be hijacked by a narrow set of “hot” issues, small voices can be lost, and achieving or identifying consensus becomes almost impossible. Other technologies have emerged to enable more specialized kinds of collective intelligence. “Expert markets” (such as elance.com and guru.com) enable interested parties to gather ideas from experts worldwide [2]. The ideas themselves are not necessarily generated by a collective process, however. Prediction markets [3] allow large numbers of people to collectively come up with often surprisingly accurate evaluations about the truth of a given hypothesis. In this case too, however, the technology collates a body of (what should be) independently-derived individual predictions, rather than enabling a collaborative evaluation process. Group Decision Support Systems (GDSS) [4] [5] have been applied effectively to enable brainstorming, but only for small physically colocated groups. e-Voting can be effective in revealing consensus with very large and highly distributed groups if there is a relatively small number of mutually exclusive options, but voting schemes break down for complex problems which involve many interdependent decisions. Argumentation (also known as rationale capture) tools [6] [7] have emerged to address many of these concerns. Such tools ask users to structure their interactions into a network consisting of three kinds of entities: issues (questions to be answered), options (alternative answers for a question), and arguments (claims that support or detract some other statement). These entities can be interleaved recursively to produce a system of issues and options surrounded by a cloud of arguments for and against these different options, e.g.

argumentation system when the direct benefits to them may be unclear. Moving from small-scale argumentation to large-scale collective intelligence introduces additional challenges, arising from the following key differences: •

The sheer volume of posters and entries increases dramatically in large-scale systems.

•

The set of participants is open and may change over time.

•

Most participant interactions will not be face-to-face, due to geographic dispersion and sheer numbers, and will thus be mediated by the argumentation system.

This in turn has many important consequences on system design, which we can frame in the form of design issues: Such tools help make deliberations, even complex ones, more systematic and complete. The results are captured in a compact form that makes it easy to understand what has been discussed to date and, if desired, add to it without needless duplication, enabling synergy across group members as well as cumulativeness across time. These tools have been used successfully in contexts ranging from software design to education to legal argumentation to policy reviews [8] [6]. Argumentation systems have been used predominantly in physically co-located meetings where participants engage in a free-form discussion while a single facilitator captures these deliberations in the form of an commonly-viewable argumentation map [9] [10] [11] [12]. Argumentation systems have also been used, to a lesser extent, to enable nonfacilitated deliberations, over the Internet, with physically distributed participants [13] [14] [15] [16] [17] [18] [19]. With only one exception1, the scale of use has however been small, with on the order of 10 participants or so working together on any given task, far less than what is implied by the vision of large-scale collective intelligence introduced in this paper. III.

How do we avoid needless duplication? When there are many contributors working concurrently, and the sheer volume of entries grows, it is no longer possible to capture an argument structure within a single screen, as is typical for many current applications of argumentation systems. It therefore is increasingly likely that someone will introduce an issue, option, or argument that has already been posted by someone else, leading to a self-reinforcing reduction in the signal-to-noise ratio in the database. This suggests that tools and/or procedures should be put in place to make it as easy as possible for potential authors to see if their post has already been captured, and also to make it easy to consolidate redundant entries post-hoc.

•

How do participants converge on the key issues? The issues raised in a database represent in effect a way of indexing the ideas (and associated arguments) in the database. The existence of a single, widely-accepted and well-understood set of issues thus substantially reduces the likelihood of duplication, since people can trace down the issue tree to find the part of the discussion they are interested in. But different people may divide up a problem differently, leading to the possibility of multiple competing issue trees that cover the same territory in different ways, and thus leading to duplication and/or fragmentation of the content. Converging on key issues is unlikely to occur prior to posting into the system, as it might in a small-scale setting, especially a facilitated one. This suggests that tools and/or procedures should be made available to enable deliberations about the structure of the argumentation, and not just the content.

•

How do we ensure wide participation in entering/editing content? One of the key benefits of collaboration comes from the fact that, if we have more “eyes” involved, we are more likely to fully cover the space of important issues, options and arguments, and also more likely to detect and correct errors therein. Large groups are, however, notoriously prone to phenomena wherein people who might gladly make a significant contribution decline to do so. People, for example, are often reluctant to replace, or even significantly modify, a piece of work authored by someone else, even if that posting has serious failings. They may be reluctant to offer diverging opinions if the bulk of the existing arguments all seem to point in another direction. If confronted with a large body of pre-existing content, they may conclude that their ideas have probably

TOWARDS LARGE-SCALE ARGUMENTATION

Scaling argumentations systems up so that they can productively mediate the interactions of hundreds or thousands of participants will require, we believe, considering design issues and options substantially different from those involved in the small-scale applications attempted to date. Argumentation in the large may also prove to be well-suited to different kinds of collaboration problems. We will consider both of these points in the sections below. A. Design Issues Small-scale argumentation systems face a range of wellknown issues, most of them revolving around the question of how we incent people to entering their reasoning into an 1

•

This exception (the Open Meeting project’s mediation of the 1994 National Policy Review [15]) was effectively a comment collection system rather than a deliberation system, since the participants were predominantly engaged in offering reactions to a large set of pre-existing policy documents, rather than interacting with each other to create new policy options.

already been captured somewhere, or soon will be, and thus choose not to enter it themselves. Addressing this will require careful thinking about incentives (e.g. do anonymous postings encourage better content?), in addition to defining tools and procedures that make it easy for people to find gaps and offer alternatives. •

•

•

•

How do we ensure that the argument is structured ‘correctly’? In an open system, we can expect that many of the participants will not be experts in how to structure argument maps effectively. This seems to be a skill that requires significant experience to master. People may fail to properly “unbundle” their arguments into the constituent issues options and arguments, they may attach them to the wrong postings, they may fail to give their posts accurate titles, and so on. Some people may find the argumentation structure unfamiliar enough that they simply opt out of participating altogether. This suggests that a large-scale argumentation system needs to support a continuum of formalization, allowing people to enter content in the form that they are comfortable with, be it extended prose or fully-structured argument maps, and also making it easy for content to be [re-]structured as appropriate, over time. How do we ensure succinct argumentation about options? In a large-scale settings, in the absence of face-to-face interactions, the participant’s discussions are mediated, and not just (in final form) recorded, by the argumentation system. That means that long discussion threads may take place whose final import, in terms of their relevance to the issues under discussion, is actually easy to capture in a compact way. This suggests that we need some tools and/or procedures to summarize and even replace discussion threads with more succinct forms. How do we separate the ‘wheat from the chaff’? Participants in an open system will range from content experts to people who are unqualified to comment on a topic. How do we highlight which entries are high-quality, and minimize the impact of entries which are not? Blog sites like digg.com have relied on “goodness” scores entered by readers to identify the posts which seem most valuable. One can also use review-and-certify procedures, as in academic publishing, reputation systems to identify well-respected authors, endorsements from thought leaders, and so on. These well-known techniques may take on new forms, however, in an argumentation context. A reputation score, for example, could be based on information that would be difficult for software to unravel from unrestricted natural language text but is readily available from an argumentation map, e.g. whether a given author’s arguments tend to cohere in their polarity (pro or con) with those entered by known experts. How do we mediate attention sharing? In a small-scale face-to-face setting, it is relatively straightforward to guide the group en masse through a systematic consideration of all the issues. The facilitator often plays a key role in this. In a large-scale (and therefore un-facilitated) system, however, people will follow their own agendas and there is a real risk that important issues will go neglected, or the discussion of them will become balkanized, with sub-

groups each attending to distinct issue sets without interacting with each other. This suggests that a large-scale system needs to provide aggregate information about the activity in the system so people can make more informed choices about where to turn their attention. Some ideas include maintaining lists of active or controversial issues, providing email notification when changes/additions occur in an area of interest, creating newsletters that periodically summarize activity in a given content area, and so on. •

How do we encourage/recognize consensus? Merely identifying the pros and cons for different options does not ensure, of course, that everyone will eventually agree on what the best options are. In small-scale argumentation systems, this consensus emerges off-line via the face-toface interactions amongst the participants, but in a largescale system, this consensus-making needs to be mediated or at least be made discernible in some way by the system itself. This is a challenging problem because option choices are often inter-dependent (i.e. the value of an option for one issue will often be deeply affected by what options were selected for other issues). Consensusidentification approaches like voting for options issue-byissue fail to account for these dependencies, and we need to turn to techniques suited for large combinatorics and nonlinear utility functions [20] [21].

These issues are all, of course, deeply intertwined. B. Application Domains Another critical issue concerns what kinds of problems large-scale argumentation systems should be applied to. They can be used in many ways, ranging from the collaborative authoring of artifacts (such as software or policies or textbooks), to collecting comments on existing policies, to facilitating a group learning experience. A key factor is whether large-scale systems are able to effectively mediate the consensus-finding that occurs “off-line” in small systems. Finally, we need to consider whether - given the surprising slow pace of adoption of small-scale argumentation systems – we will find that successful large-scale systems are even more elusive. This is an open question for now because, as noted earlier, only one attempt has been made to date to create a (weak form of) large-scale argumentation system. One could argue, however, that in some ways large-scale systems may have more potential than small-scale systems. There is widespread disaffection with using forums and conversation tools for mediating large-scale deliberations because of the sheer volume, low value density, and mixed quality of the postings in these contexts. A recent digg.com thread (on the “airplane on a treadmill” question), for example, generated hundreds of postings even though there was only one issue, two possible options, one correct supporting argument (for the correct option), and a small handful of faulty arguments (supporting the incorrect option). Generally speaking. It seems clear that the number of distinct issues options and arguments will grow, after a certain point, much more slowly than the number of participants in a discussion. The qualitative increase in succinctness offered by using an argumentation system at large scales may thus prove quite compelling.

IV.

NEXT STEPS

In this paper, we have considered some of the design issues raised by developing a collective intelligence system based on large-scale internet-enabled argumentation. We have implemented a web-based argument capture system we call the Collaboratorium (see figure 1), and begun implementing and evaluating enhancements aimed at addressing the issues identified above. Our initial investigations will explore the critical question of whether we can generate, through software tools and incentives, a user community dynamic wherein at least some of the contributors add content informally (i.e. as extended comments on high-level issues) and other participants add value by structuring this content into properly organized issues options and arguments. This is analogous to what often happens in Wikipedia and it’s offshoots: some people focus on generating new content, and others specialize on editing and fact-checking existing content.

6.

7. 8.

9.

10. 11. 12. 13.

14.

Figure 1. A snapshot of the Collaboratorium.

15.

ACKNOWLEDGMENT I’d like to thank Thomas Malone and Marco Cioffi for many useful discussions about the ideas underlying this paper.

16.

REFERENCES

17.

1. 2. 3. 4. 5.

Walton, D.N. and E.C.W. Krabbe, Commitment in dialogue: Basic concepts of interpersonal reasoning. 1995, Albany, NY: State University of New York Press. Denning, P.J. and R. Hayes-Roth, Decision making in very large networks. Communications of the ACM, 2006. 49(11): p. 19–23. Wolfers, J. and E. Zitzewitz, Prediction Markets. Journal of Economic Perspectives, 2004. 18(2): p. 107-126. Pervan, G.P. and D.J. Atkinson, GDSS research: An overview and historical analysis. Group Decision and Negotiation, 1995. 4(6): p. 475-483. Gopal, A. and P. Prasad, Understanding GDSS in Symbolic Context: Shifting the Focus from Technology to Interaction. MIS Quarterly, 2000. 24(3): p. 509-546.

18. 19.

20.

Kirschner, P.A., S.J.B. Shum, and C.S.C. Eds, Visualizing Argumentation: Software tools for collaborative and educational sense-making. Information Visualization, 2005. 4: p. 59-60. Moor, A.d. and M. Aakhus, Argumentation Support: From Technologies to Tools. Communications of the ACM, 2006. 49(3): p. 93. Moran, T.P. and J.M. Carroll, eds. Design Rationale: Concepts, Techniques, and Use. Computers. Cognition, and Work, ed. G.M. Olson, J.S. Olson, and B. Curtis. 1996, Lawrence Erlbaum Associates: Mahwah NJ USA. Shum, S.J.B., et al., Hypermedia Support for Argumentation-Based Rationale: 15 Years on from gIBIS and QOC, in Rationale Management in Software Engineering, A.H. Dutoit, et al., Editors. 2006, SpringerVerlag. Cluxton, D., S.G. Eick, and J. Yun, H y p o t h e s i s Visualization. Information Visualization, 2004. INFOVIS 2004. IEEE Symposium on, 2004: p. p4-p4. Verheij, B., Dialectical Argumentation with Argumentation Schemes: An Approach to Legal Logic. Artificial Intelligence and Law, 2003. 11(2): p. 167-195. Suthers, D., et al., Belvedere: Engaging students in critical discussion of science and public policy issues. Proceedings of AI-ED, 1995. 95: p. 266-273. Jonassen, D. and H.R. Jr, Mapping alternative discourse structures onto computer conferences. International Journal of Knowledge and Learning, 2005. 1(1/2): p. 113129. Chklovski, T., V. Ratnakar, and Y. Gil, User interfaces with semi-formal representations: a study of designing argumentation structures. Proceedings of the 10th international conference on Intelligent user interfaces, 2005: p. 130-136. Hurwitz, R. and J.C. Mallery, The Open Meeting: A WebBased System for Conferencing and Collaboration. World Wide Web Journal: The Fourth International WWW Conference Proceedings, 1996. 1(1): p. 19-46. Lowrance, J.D., I.W. Harrison, and A.C. Rodriguez, Capturing Analytic Thought, in First International Conference on Knowledge Capture. 2001. p. 84-91. Karacapilidis, N., E. Loukis, and S. Dimopoulos, A WebBased System for Supporting Structured Collaboration in the Public Sector. LECTURE NOTES IN COMPUTER SCIENCE, 2004: p. 218-225. Heng, M.S.H. and A. de Moor, From Habermas' s communicative theory to practice on the internet. Information Systems Journal, 2003. 13(4): p. 331-352. Li, G., et al., ClaiMaker: Weaving a Semantic Web of Research Papers. Proceedings of the First International Semantic Web Conference on The Semantic Web, 2002: p. 436-441. Ito, T., M. Klein, and H. Hattori. Multi-issue Negotiation Protocol for Agents: Exploring Nonlinear Utility Spaces. in Twentieth International Joint Conference on Artificial Intelligence. 2007. Hyderabad, India.

21. Klein, M., et al., Protocols for Negotiating Complex Contracts. IEEE Intelligent Systems, 2003. 18(6): p. 32 38.