Section 1: Narrative Overview


SECTION 1: NARRATIVE OVERVIEW


I. Executive Summary

In February 1997, over 40 researchers in the computing, social, behavioral, organizational, information, and engineering sciences gathered for a workshop sponsored by the National Science Foundation. The topic of the workshop was "Human-Centered Systems: Information, Interactivity, and Intelligence", and the goal of the workshop was to define this emerging multidisciplinary field and articulate research, educational, and infrastructure needs to support work in this area. In this Executive Summary, the definition, research directions, and some debates in Human-Centered Systems are summarized.

Motivation: Why Support Human-Centered Systems Research?

The concept of "human-centered systems", as elaborated below, represents a significant shift in thinking about information technology: A shift that embraces human activity, technological advances, and the interplay between human activity and technological systems as inextricably linked and equally important aspects of analysis, design, and evaluation. Human-centered systems have vast potential to alleviate problems of information overload and complexity in computer software, to increase the effectiveness of computer technology in communities and the public sector by making computers easier to use by ordinary people, and to enhance the ability of distant individuals and groups to work together using computer support. Research in human-centered systems also advances basic scientific knowledge in such areas as distributed cognition, speech, and social systems, in disciplines ranging from linguistics to psychology and computer science. In an era of unprecedented technological change and growth, basic scientific research is crucial to design appropriate interventions into complex human social systems and to analyze and evaluate the effects of such interventions.

Definition and Research Directions

A system is defined as an agglomeration of interacting, interdependent components which used in combination accomplish an activity that no one component can perform alone. In this report, we focus primarily on information, communication, and distributed knowledge systems. A human-centered system aims to serve human activity. It is one that incorporates explicitly human (e.g., perceptual, motor, cognitive, and social) ramifications as components of design.

Advances in information technology, computing, knowledge representation, learning, communications, and the behavioral and social sciences, taken together, offer unprecedented new opportunities for the design of human-centered systems that can support creative knowledge work (e.g., planning, decision making, knowledge creation and dissemination). Such work is often collaborative, with participants physically separated. It often depends upon distributed resources of computing and data, making networked communications (with dynamic control and allocation of capacities) an essential central infrastructure in distributed computing.

Human-centered systems employ computing technology as a tool for the human user, not as a substitute; the human is the ultimate authority for control and the technology is employed to expand human capabilities and intellect. But to accomplish desired knowledge-intensive tasks, the human must interact with machine components of the system and with other humans. The computing environment and the network become the mediator and facilitator for this interaction. The opportunity arises for extending human intellect through capabilities of the technological system. This system must be tolerant and adaptive enough to accommodate users having a wide range of skills and competence, and to use human-machine communication technologies that are, as yet, imperfect.

The human-machine interface enables users to acquire information, explore alternatives, execute plans, and monitor results. Making a high bandwidth interface that presents data rapidly in a form that facilitates human decision making is the central challenge. The sensory modalities of sight, sound, and touch are major channels for the human (senses of smell and taste being less utilized in most information management tasks). Integration of these modalities can support human judgment, but the technologies for sight (visual presentation, spatial organization, gesture, gaze tracking, image recognition), sound (speech recognition, text-to-speech synthesis, speech store-and-forward, non-speech audio), and touch (manual gesture, two-handed input, grasp, force feedback) are incompletely developed. Development of multimodal interfaces is therefore a central concern of human-centered systems.

Today, we have the possibility to reach far beyond traditional notions of communication and interaction as commonly practiced among humans. We are moving towards an era of ubiquitous computing (anyone, anytime, any place, anywhere). We must provide effective means for humans to generate new creative environments and knowledge/communication infrastructures that support activities that were not possible before (e.g., Web-based education, electronic commerce, virtual travel). Thus, key topics for research include knowledge representation and exchange, interaction with a sea of unstructured information, ability to cope with ambiguity and uncertainty, adaptive environments, learning, user and organizational models, collaborative environments, data visualization, summarization and presentation, universal access to complex hybrid digital libraries, and distributed knowledge networks. Communication network issues, including standards, accessibility, and dynamic resource allocation methods, also are a part of the picture.

Advancing the understanding of human-centered systems requires several other things. A science of design is needed; that is, design methodologies in which the unit of analysis is the joint human-machine system, where particular attention is paid to the ways in which technological change transforms cognitive and collaborative activities in a field of practice. It is insufficient to study and model people in isolation from technology or technology disconnected from a field of human activity. Both perspectives are needed in a fundamentally integrated way. An implication of this view is the centrality of field work to provide real data on real activity in real contexts. A related issue is metrics: how can we measure what is happening in a distributed cognitive system in a meaningful way? Simple quantitative measures such as time to solution, cost, and so on can provide some insights but are insufficient alone. A worthy alternative is rich cognitive simulation studies that are grounded in context and relevant to real problem-solving tasks.

Collaborative knowledge work pervades most sectors of national concern. Included are health care, environmental sciences, education (both knowledge propagation [teaching] and knowledge creation [research]), transportation, communication, and basic human needs of food, clothing, and shelter. In each of these sectors, human creativity is advancing the frontiers of understanding and implementation. And, in each sector, human-centered systems have contributions to make. The benefits are not only immediate economic ones, with productivity gains in traditional human activities, but they include significant advances in the quality of life for our nation, and ultimately the world. Sustained support of programmatic research in human-centered systems will stimulate these advances.

Four Organizing Themes

At their November 1996 meeting, the HCS Workshop Steering Committee identified four themes that became the foci of discussion for the Break-Out Groups (BOGs). These themes are:

These themes are highlighted in Figure 1 below.




Figure 1. Organizing themes of the Human-Centered Systems Workshop.

Differences and Debates

During the workshop itself, and as also evidenced in individual position papers and BOG reports, workshop participants had some differences of terminology and opinions on issues. Some of these differences are summarized here:


1. The term "system" was used in different ways: the technological system of software and hardware, the human-machine system , the social system of relationships and commitments among people.

2. "Human-centered systems" was used to refer both a process of design (taking into account human activity, the context of use) and to qualities of the technological products of that design process (e.g., flexible, intelligible software systems). To some, "human-centered" was also an ethical and philosophical position.


3. In the workshop, there were multiple interpretations of what human-centered means. Break-Out Group (BOG) 3 articulated the various currents as several wide interpretations versus a strong view of human-centered systems. In the strong view of human-centered systems, design is grounded in the goals and activities of people in real world situations: human-centered design is problem-driven, activity-centered, and context-bound. These criteria would seem to be a part of all design activity: all designers make assumptions about user problems, activities, and the context of use. The strong view says that human-centered research and design makes these explicit, based on empirical results, and models how technological artifacts influence them.


4. Intelligent systems was also the subject of vigorous discussion. Intelligent systems provide several potentially useful technologies for human-centered systems. For example, they might serve human operators by reducing effort in entering and receiving information from machines (e.g., using text to speech transcription and interpreting natural language communication), perform information filtering from unstructured databases, facilitate information extraction, fuse multimodal data streams, execute high-level instructions (e.g., using agent technologies for computer-assisted medical procedures), monitor and track errors in operating complex systems, provide decision tools to cope with uncertainty and interpretation of noisy or poorly understood scientific and technical data, provide advanced modeling capabilities for user and environment modeling (e.g., Bayesian networks), help achieve more effective resource utilization in networked environments, and provide ontological knowledge bases to support information retrieval, education, and other knowledge intensive tasks.

In the framework of a human-centered methodology, intelligent systems technology must be properly "tamed" and designed to serve the human operator(s) in the most effective and safe manner that incorporates a deeper understanding of the needs, capabilities, and limitations of human operators when using advanced technology in specific contexts and applications.

Other items for discussion included (1) conceptualizing intelligent systems as "team players" in joint human-machine interaction, (2) incorporating affective components into models of intelligence in addition to perceptual, cognitive, and motor processes, and (3) issues of trust between humans and intelligent machines (also see Point (6) below on anthropomorphizing technology).

5. A set of questions and tradeoffs arose about modeling. Positions included (1) the goal of modeling should be to provide a valid (and not necessarily quantitative) explanation of the phenomena in question, (2) models of activity, rather than generative mechanisms of cognition and performance, are needed, (3) models should be generalizable and reusable across contexts, but yet must stay grounded in the contexts in which they were built, and (4) modeling and representation activities are always political and have political/ideological consequences for the technologies that depend upon these models.

6. Anthropomorphizing computational technology: Some researchers argued against this because of empirical research results and consistent historical rejection by users, while others explore building believable agents that incorporate affective components.

Recommendations: Partial List


A summary of recommendations to the National Science Foundation are as follows (also see the full list of recommendations later in this report). By "collaboratory" we mean a network of people and computing technology that is organized around a substantive problem area and includes digital library, communication, and domain-specific tool technologies.

1. Sustained programmatic effort in Human-Centered Systems

2. Establishment of a Human-Centered Systems Collaboratory to drive research and education and provide infrastructure for the HCS community. For example, such a collaboratory could support HCS researchers and educators by providing (1) a rich digital library of case studies and data , (2) information brokerage services to support distributed courseware development, and (3) an organized set of downloadable software demonstrations that illustrate HCS principles in context.

3. Establishment of HCS collaboratories organized around critical societal issues (e.g., health care, education, and other applications described later in this report)

4. An HCS Visiting Scholar Program that brings together diverse disciplines for a common purpose in HCS research

5. More focused HCS workshops (e.g., information in context, collaboration, HCS design, and social informatics)

6. Establishment of HCS testbeds (e.g., standard packages of images for image search experiments)

7. HCS competitions, modeled after TREC efforts on retrieval systems

8. Critical Research Initiative on metrics and evaluation of HCS, with particular emphasis on longitudinal, multidisciplinary studies of co-evolution of the social and technological systems over time

9. HCS research grants therefore need to be of relatively long duration; e.g. 5-8 years.

10. Capitalize upon and work with emerging technologies (e.g., high speed networking and digital multimedia (e.g., Internet 2, VBNS, PACIs))

II. Overview of Workshop

The Committee on Computing, Information, and Communication of the National Science and Technology Council has identified five components for a High Performance Computing Program, of which one is Human-Centered Systems. Discussions between members of the NSF CISE Directorate, in particular, Y. T. Chien, John Cherniavsky, Gary Strong, and Howard Moraff, and Thomas Huang of the University of Illinois at Urbana-Champaign gave rise to the idea of organizing a Workshop on this topic. Thomas Huang from the University of Illinois at Urbana-Champaign and James Flanagan from Rutgers University co-organized and co-chaired the Workshop. The Workshop Steering Committee met in November 1996 to organize the workshop.

A. The HCS Steering Committee was:

The workshop was hosted by the Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign at the Crystal Gateway Mariott Hotel, Arlington, VA, on February 17-19, 1997.

B. Participants

The workshop was organized into four breakout groups (BOGs) with invited participants as follows. Each participant wrote a position paper that was distributed electronically to all other Workshop attendees before the meeting.

BOG1: Information Organization and Context

Co-Leaders: Michael Lesk, Bellcore and Gio Wiederhold, Stanford University


BOG2: Communication and Collaboration

Co-Leaders: Patricia Jones, University of Illinois at Urbana-Champaign and Simon Kasif, University of Illinois at Chicago

BOG3: Human-Centered Design

Co-Leaders: Terry Winograd, Stanford University and David Woods, Ohio State University

BOG4: Organization and Social Analysis (Social Informatics)

Co-Leaders: Rob Kling, Indiana University and S. Leigh Star, University of Illinois at Urbana-Champaign

Five invited plenary talks were as follows:

Charles E. Billings, Ohio State University

"Issues Concerning Human-Centered Intelligent Systems:

What's 'human-centered' and what's the problem?"

Bernard M. Corona, Army Research Laboratory

"Army Research Efforts in Human-Centered Design"

Joseph Mariani, Limsi-CNRS, France

"Spoken Language Processing and Multimodal Communication:

A View from Europe"

Ryohei Nakatsu, ATR, Japan

"Integration of Art and Technology for Realizing Human-like Computer Agents"

Lawrence Rabiner, AT&T

"The Role of Speech Processing in Human-Computer Intelligent Interactions"

Government Observers
Also attending the workshop were a number of observers from government agencies:

National Science Foundation: John Cherniavsky, Y. T. Chien, Les Gasser, Steve Griffin, Juris Hartmanis, Rachelle Hollander, Howard Moraff, Larry Reeker, Nora Sabelli, Larry Scadden, Gary Strong, Maria Zemankova

Defense Advanced Research Projects Agency: Ron Larson, Kevin Mills, Allen Sears

Army Research Laboratory: Bernard Corona, Carolyn Dunmire, Mark Kindl

Office of Naval Research: Helen Gigley

NASA Ames Research Center: Kevin Corker

Air Force Office of Scientific Research: John Tangney and Abraham Waksman

NASA Johnson Space Center: Jane T. Mailin

Army Research Office: Ming Lin

General Services Administration: Susan Brummel

C. Agenda

The workshop agenda was as follows:

Sunday, February 16, 1997


Monday, February 17, 1997


Tuesday, February 18, 1997

Wednesday, February 19, 1997

III. The Challenge of Human-Centered Systems

This section summarizes discussions, BOG reports, and other interactions between Workshop participants. It is organized as (1) importance and benefits of human-centered approaches and systems, (2) definitions of human-centered systems, (3) application domains and cross-cutting issues for research, and (4) recommendations for research directions, educational initiatives, and infrastructure needs for a strong national human-centered systems programmatic effort.

A. Importance and Benefits

We are experiencing unprecedented leaps in technological power which is manifested as computer speed, memory, disk capacity, minituarization, and universally accessible networks. These advances present unparalleled opportunities for expanding the ubiquitous use of computers in fundamental human activities such as communication, interaction, collaboration, decision making, knowledge creation and dissemination, and creative work.

However, powerful, uncommunicative technologies that cannot take interactive direction from humans make the joint human-machine system vulnerable to a variety of miscommunications, misassessments, and miscoordinations that can and have lead to failures. Thus, human-centered systems need to fit into the context of use, uphold human authority, and be open, inspectable, and intelligible. The number of variables that contribute to the design of human-centered systems is large, and thus controlled and measurable experiments must be performed in order to improve our ability to assess the performance of systems in the context of use. The implications of technology in different applications must also be studied and better understood.

Therefore, while expanding the technological capability, availability, and accessibility of computer systems is essential, we must create carefully constructed infrastructures for studying and experimenting with human-centered environments. This approach will allow us to develop a methodology of designing and engineering useful systems for individuals, groups, organizations, and society. We see human-centered systems as an emerging discipline that combines principles from cognitive science, social science, computer science, and engineering to study the design space of systems that support and expand fundamental human activities.

B. Characterizations of Human-Centered Systems

Each of the break-out groups discussed definitions of "human-centered" systems, research, and design and evaluation practices. In brief, these definitions were:

(1) To be human-centered, a [computer] system should be based on an analysis of the human tasks that the system is aiding, monitored for performance in terms of human benefits, built to take account of human skills, and adaptable easily to changing human needs. Relevance and feedback are core issues. Research should produce principles of how people deal with information, and how information systems can be comprehensible, predictable, reliable, and controllable. Technological systems should act as tools and amplify the power and force of practitioners.

(2) To be human-centered, a technological system should support actual practice effectively, be flexible, adaptive, context-sensitive, open, inspectable, engaging, enjoyable, and should be designed in a iterative and longitudinal manner. Relevance, context, and co-evolution are core issues. In collaborative systems, issues related to the ease with which participants can share information, engage in coordinative 'articulation work', and allocate tasks also define part of the agenda for human-centered systems research.


(3) "Human-centered" research can be widely interpreted as being driven by human needs, keeping people "in the loop", building technology that interacts somehow with people, or justified by predicted improvements in human performance, cognition, or collaboration. However, a strong interpretation is that human-centered research and design is problem-driven, activity-centered, and context-bound. Human-centered analysis looks in detail at situated action in context yet also provides generalizations that are useful for other contexts.


(4) Human-centered design recognizes that technology structures social relationships and takes into account the various ways in which actors and organizations are interconnected via social relationships, information flow, and decision making authority. Human-centered research and design must address the complexity, interdependence, and social embeddedness of modern computing systems. As such, it is necessarily holistic and ecological and is concerned with usefulness, usability, sustainability, cultural and political factors, infrastructure, and standards. A human-centered analysis addresses the variety of concrete social situations that exist in the field of practice.

In synthesizing these collective positions, we propose the following:

Human-centered analysis, modeling, design, and evaluation is a process that is

Technological systems (e.g., software, hardware) are outcomes of, and embedded in, this process. Some design criteria for effective human-centered technological systems are:

Human-centered systems research should be driven by problems and organized around context and activity and relies also on fundamental advances in technology, design, and behavioral and social sciences.

C. Domains and Issues

The subtitle of this workshop is Information, Interactivity, and Intelligence. These three themes are explored briefly below in remarks that complement the remainder of this section, which summarizes issues as organized in the Break-Out Groups in the context of critical application areas.


Information in Context: Data Overload

Coping with "information overload" is an issue because what is informative depends on context. We have not solved the problem of helping people interpret or find relevant information in a large data set. A human-centered approach to this question relies on understanding problem, activity, and context in the task at hand. Privacy and security are key issues as well.

Interactivity

Interaction involving humans and technology, whether two or more humans interacting through technology or human interaction with machines as an end in itself, is a defining feature of human-centered systems. Interaction brings up many issues, including communication, common ground, shared information, synchronization, shared focus of attention, and awareness of others.

Intelligent Systems

A human-centered approach to intelligent systems focuses on the utilization of specific frameworks such as learning, speech and language technology, visual interfaces, and intelligent decision aids in order to create a richer, more versatile, and effective virtual environment that supports human activity. Thus, the emphasis in this research is not on building autonomous systems that mimic humans but rather supporting human activity using intelligent system tools subject to the constraints, goals, and principles of human-centered design. One approach to a human-centered use of intelligent system technology focuses on how to make such systems "team players" in the context of human activity. Another approach focuses on building effective computational tools for modeling, interpreting, fusing and analyzing (mining) cognitive or social interactions such as speech, vision, gesture, language, or collaboration. These tools can be used to facilitate, enrich, and improve the state of the art in human-computer interaction.


Organizing Around the Four Themes of Information, Collaboration, Design, and Social Informatics

Because human-centered approaches to analysis, design, and evaluation are context-bound, organizing research efforts around domains of practice (e.g., health care, natural science, manufacturing) is necessary. Also, we strive to have generalizable results across contexts, and thus we articulate cross-cutting classes of issues that are based on the organization of the workshop itself. That is, these cross-cutting issues are defined as Information Science, Collaboration Science, Design Science, and Social Informatics. Information Science includes issues of multimedia representation, search and retrieval, visualization, and data mining. Collaboration Science includes issues of shared information, workflow, mutual awareness, and social information processing. Design Science includes methodologies and methods of inquiry. Social Informatics encompasses a range of social, organizational, cultural, and political issues. A matrix that represents these domains and issues, with just a very few examples of research questions, is shown below in Table 1.

Table 1. A sketch of representative research issues in the context of critical application domains, organized around the four themes of the HCS workshop.

Application Domains Information Science Collaboration Science Design Science Social Informatics
Health Care medical records; multimedia databases; visualization tele-surgery;

virtual support groups for patients

clumsy automation in the operating room and how to avoid it privacy;

politics of disease classification

Education organization of information to support effective learning; including learning by discovery cooperative learning; distance learning; intelligent tutoring supporting active, authentic learning in context classroom culture and how it changes with computing technologies
Natural Sciences

(e.g., environmental science, earth science)

digital libraries of complex spatiotemporal data; visualization to support scientific reasoning collaboratories for communication, remote control of instrumentation;

virtual communities;

electronic journals

effective debate and interpretation of data supporting debate;

negotiation of meaning ;

how technologies change the nature of scientific practice

Manufacturing heterogeneous information concurrent engineering computer-human integrated manufacturing sustainability; standards; re-use
Government, law enforcement, and public policy heterogeneous information for public decision making (e.g., budgetary, scientific, regulations) heterogeneous groups with different agendas and languages managing constraints, conflicting opinions, and information to make effective decisions negotiation among different value systems; power and authority; privacy and security
Large-scale operations (e.g., disaster relief) spatiotemporal data, terrain, weather, political boundaries, courses of action, diplomatic protocols distributed ad-hoc teams for coordinating activity remotely;

supporting rapid socialization and

the "relevant common picture"

time-critical information, naturalistic decision making negotiating among diverse media (physical maps, computer systems, etc.);

authority, permissions, security, impacts (economic, health, quality of life)
Virtual organizations coping with ill-defined, emergent goals and information needs; distributed databases; electronic commerce mutual awareness; rapid socialization coping with activity and context in a dynamic, fluid organization emergence of culture and community; power and authority, trust, ambiguity, competition

D. Recommendations


Human-centered systems are by nature multidisciplinary and thus a variety of perspectives are appropriate to make scientific advances. Such advances rely in varying degrees on fundamental advances in technology and engineering, design, and the social and behavioral sciences. A new science of Human-Centered Systems emerges that takes as the object of study the interaction among human, technological, material, social, and cultural systems. In consequence, we believe the National Science Foundation should commit major investments in the following dimensions:


1. Research


Human-centered systems define a new area of research that is grounded in fundamental interactions between computing, engineering, and social sciences. In particular, interaction technology draws upon cognitive science, perception, speech, language, and visual communication; collaboration technology draws upon organizational science and collaboration science; and societal benefits can be understood with economics and social informatics. Overarching research issues are how to cope with context and relevance and how to do appropriate and meaningful evaluation.

A sample of more specific research issues is provided below. Many of these are described further in the BOG reports and individual position papers.
Also recommended for further reading is the Survey of the State of the Art in Human Language Technology.

Information, Modeling, Visualization, Multimodal Interaction

Communication and Collaboration

Design Science

Social Informatics

2. Education

Multidisciplinary educational initiatives are critical to bring together students and faculty in the computing, social, behavioral, and engineering sciences. Besides conventional approaches such as workshops, educational task force reports, and pilot multidisciplinary programs at universities, new learning technologies for telementoring, teleapprenticeship, and collaboration can leverage the creation of new communities of HCS researchers and learners.

3. Infrastructure

By "collaboratory" we mean a network of people and computing technology that is organized around a substantive problem area and includes digital library, communication, and domain-specific tool technologies.

last updated July 2, 1997 by Patricia M. Jones (send me mail at p-jones5@uiuc.edu)