Imagine a battle where commanders use computers to receive video, audio, and graphical information about weather, terrain, and troop movement. The computers then filter this information and present field commanders with several good courses of action, clearly displayed with certain colors coding specific types of information. The commanders, all scattered but receiving the same information, are able to discuss details and plan their best course of action.

That plan is quickly communicated to soldiers through helmet-mounted displays, carefully designed to present key information without being distracting. Soldiers acknowledge receiving information by squeezing a button on their rifle.

Researchers at the Beckman Institute, in conjunction with the U.S. Army, are helping make these types of scenarios a reality for the Army.

In the Fall/Winter 1995-96 issue of the Beckman Institute newsletter, the Army Research Laboratory's Federated Laboratory (Fed Lab) grant was announced. Since then, Institute researchers have worked hard learning what the Army needs and how it operates, as well as conducting research that will lead to future applications-both military and commercial. The following is an update of that progress.

The goal of the entire Fed Lab project, which comprises several dozen institutions working on three different overarching topics (Advanced Displays, Telecommunications, and Sensors), is to develop computer technology to help Army personnel receive and understand information as quickly and clearly as possible-creating an "Army for the digital age." Beckman Institute researchers are involved in the Advanced Displays and Interactive Displays consortium, which comprises six institutions: two academic, three industrial, and the Army itself. The University of Illinois at Urbana-Champaign is the major university partner and Rockwell International, the major industrial partner. These partners are joined by Sytronics, Microelectronics Computing and Networking Center, and North Carolina Agricultural and Technical State University.

"What all of us are doing in one way or another is trying to identify how to present a large amount of information in such a way that a person can grasp, keep track of, and then quickly access the information they particularly need," says George McConkie, co-Principle Investigator (PI) on the project. "We're focusing on the commander in the battlefield, but I'm sure that many of the same things can apply to other people who are involved in complex situations, such as a person who's flying an airplane or commanding a ship. I'm in Educational Psychology and I keep thinking: 'How can some of these methods be used for helping a teacher or a principal?' Imagine being able to keep track of the progress of every student in your school, detecting when a particular child is failing to understand some aspect of long division or cell structure, and being able to provide assistance in some way before the child's progress is held up."

Advanced Displays focuses on improving human-computer interaction, which is one of the three major research themes at the Beckman Institute. "Advanced display is not just about what you see on your computer screen, but all the aspects of how the displays get there," says Thomas Huang, PI of the project. "That includes data processing, human-computer interaction, and much more."

Creating new interactive displays requires figuring out how to present complex information as quickly and clearly as possible; determining the human limitations to receiving that information; and developing communication with the computer that doesn't require a keyboard or a mouse. These goals require managing enormous, multimedia databases, complete with video, audio, and graphics data; creating intelligent information processing, in which the computer sifts through these large databases quickly and accurately; and determining how to input information in such a way that a user can then query the computer effectively or be notified of the availability of potentially useful information.

Many of these issues already have been addressed to some extent by Beckman Institute researchers in the Human-Computer Intelligent Interaction area, but the Army's needs are different from other commercial or educational applications. Much of the first year of the project was spent learning exactly what the Army's needs are and how it operates. "A year ago, people had only ideas of where we wanted to go," says Wendy Harris, project coordinator. "While the majority of our effort is still in basic research, we are constantly making discoveries about whether what we originally envisioned is feasible and, if so, how to create it."

"The Fed Lab is an interdisciplinary project and the Beckman Institute is already set up for interdisciplinary research," notes Harris. "We have mechanical engineers talking to computer scientists every day and they sit in the same room with supercomputing display specialists. Each of these fields has a very specialized language but, by talking with each other, we're starting to share a common language. That's not a small or trivial task and it's what the Institute is all about."

Huang notes that it is rare to find this type of cross-disciplinary expertise at a single university. To illustrate this, he points out that of the three consortia, Advanced Displays is the only one with a single major university partner. The others required five or six universities to bring together the divergent expertise.

The Advanced Displays project consists of two equally important and closely coupled parts: the "human elements" side and the technical side. Researchers on the human side investigate the elements that affect how well humans receive information and how to present information that can be easily grasped by commanders and by soldiers in the midst of battle. Researchers on the technical side investigate how to create computer techniques that can support dynamic databases involving uncertainty and imprecision, both in incoming information and user queries; how to intelligently filter the vast amount of information; and how to present information to the users in the best possible way.

Human factors researchers are in the midst of determining how humans process information and how to translate those findings into clear displays. The Beckman Institute's team of human factors researchers is led by psychologists McConkie, Chris Wickens, and Art Kramer.

McConkie's group is building on earlier research indicating that the human eye processes detailed information from a relatively small part of the visual field. This means, perhaps, that the computer does not have to provide a top-grade picture for the whole screen, as long as it can keep track of where exactly the eye is looking and provide good quality in that area. McConkie's group is studying how large that part of the visual field must be. The group is also asking such questions as "How degraded can the image outside the immediate visual field be and not be distracting?" McConkie is beginning to explore the use of the eye as a pointing device. Ultimately eye movement will be integrated with the rest of the system that will use gestural and verbal commands to communicate with the computer.

Wickens and his graduate students have examined the pros and cons of 2-D and 3-D displays, as well as "immersed" displays. They have found that display type is "task dependent"-in other words, depending on the task, sometimes a 2-D display is a more effective tool than a 3-D or immersed display. In the case of maps, for example, they have found that a 2-D view is more effective when someone needs to accurately measure distance between two objects. This finding suggests that more sophistication is not always better.

"The more immersed you are in the display, the more technologically demanding and expensive it is and the more vulnerable to breaking down. So if we're going to use these fancy displays, we'd better be darn sure that they are the best choice for a given application," says Wickens.

Wickens is also looking at how best to design helmet-mounted displays. He has found that a head-up display with information projected onto the visor (sort of like "bugs on a windshield") works better than a conventional head-down display, given certain caveats: You can't have too much information on a head-up or helmet-mounted display and, if something unexpected happens in the environment, it may take longer for the soldier to notice it. In an example taken from aviation, Wickens explains that when a pilot uses a head -up display in landing, if another plane pulls out on the runway in front of him, the pilot using the head-up display may take longer to notice and take corrective action than he would were he using conventional head-down instruments.

Art Kramer's group is also examining elements of visual perception that will have an impact on display design. His graduate students are trying to learn how people extract and perceive visual information in a 3-D environment. One aspect they are examining is the effects of intrusive visual information on the ability to locate needed information. Kramer has found, for example, that the eye will move to a new object 50 percent of the time if it appears suddenly, even when the subject didn't intend to look at it. This means that any new information, whether relevant or not, can have a bigger negative impact than the researchers had anticipated.

Kramer's group also is examining the physiological indicators of cognition. By using data on heart rate and brain waves, researchers can track a person's attention level. This becomes important if one is sending a message that is critical to another's survival. "If you decide that you want to communicate with somebody, maybe you should know before you try to grab their attention whether they are awake, asleep, or under great stress at the moment," says Kramer. "Brain waves may be able to tell you that."

On the technical side of the project, researchers are designing algorithms and software to accomplish three broad tasks: managing enormous amounts of multi-dimensional data (i.e. graphic, visual, and audio); coping with uncertainty and ambiguity; and filtering and processing information.

In order to manage these large graphic, visual, and audio databases, Huang is applying the relevance feedback technique, a common tool within computer science. Imagine, for example, that someone is looking for all barns in an aerial photograph. The operator inputs parameters that describe the physical characteristics of a barn: type of material used, color, and so on. The computer gives the operator some choices of images. Then, based on what the operator selects, the computer is triggered to return with more choices that more closely fit the original selection. Using this method, images can be indexed on the basis of visual content, so users can retrieve items that contain a particular object or a specified texture. The ability to index visual or audio images will prove enormously helpful in a wide range of fields, from clothing design to the recording industry, notes Huang.

In a battle situation, not every detail will be known for certain and information changes quickly and constantly. Planners have to factor in the possibility that intelligence information might not be completely accurate. Among others, Sharad Mehrotra is using the concept of relevance feedback to compensate for imprecise information, as well as for a vague query from the operator.

His group is also developing a structure to handle a dynamic database. In other words, information is coming in all the time during a battle and the database is constantly being updated. It is critical, however, that the computer doesn't garble the data if, for example, new data comes in at the same time that a query is made. Once these two tasks (sorting data and factoring in uncertainty) are accomplished, users need an intelligent information processor to quickly and accurately sift through all the information and present it in a coherent manner. This is where Battlefield Reasoning System will come in.

As computer Deep Blue's defeat of chess master Gary Kasparov illustrated, computers can be expert at quickly and accurately playing out an enormous number of possible scenarios. The group headed by Caroline Hayes has developed software called "Battlefield Reasoning System" (BRS), that takes advantage of that strength. The software identifies various strategies that commanders can use in their tactical planning. BRS will filter a complex web of data to identify the best possible courses of action (COA). The goal is to develop a hierarchy and a process whereby all the planning divisions of the Army work together with the same information and can easily share the information among units.

The computer will generate a diverse set of alternative COAs based on specific guidance from the commander, such as "use tanks as reserve only," or "leave no avenues of approach unattacked." The computer program will also advise about how much army strength to commit to each avenue of approach. Using a genetic algorithm called FOX (as in to "out-fox" the enemy), developed by Hayes and students Major Jerry Schlabach and Carolyn Fiebig, the computer can help the commander to identify the best COAs from the set of alternatives.

Currently, Army war gaming is often done moving Post-It notes around on a large map, or moving miniature units around in a sandbox, says Hayes, having observed such exercises. Soon, with the combination of BRS and Battleview (a complex visual display), commanders will be able to war-game with the help of computer-filtered information, making the process much more effective.

"Battleview," a computer program designed by Polly Baker's group, enables commanders to see actual battle scenes and displays accurate terrain information and information about the location of both enemy and friendly forces. This information can be displayed in either 3-D or 2-D (or both) on an "immersascreen," a wall-sized computer display. Battleview also recognizes voice commands. A person can query the computer for information about friendly troops and get such information as the name of the battalion commander or the unit's ammunition status. This fall, Baker's group is creating the software that enables users to conduct multiple-object queries, such as "show all tanks whose ammunition status is low."

Although still in the research and testing stage, ultimately Battleview, combined with BRS and other applications, will integrate a complex and dynamic level of information about terrain, weather, and troop movement, and users will be able to communicate with it using voice and gesture.

Much of Battleview's achievement will be applicable to any situation that has a large-scale environment based on geography. Baker is applying much of the same technology to a project studying the effect of weather, water flow, and farm-chemical runoff on Chesapeake Bay's ecosystem. This project is being run in conjunction with another branch of the Army.

One of the goals of the Fed Lab is to make it possible for people to communicate with a computer display system quickly and naturally without mouse and keyboard. To achieve this, computers need to understand both verbal and gestural commands.

Unfortunately, verbal commands are often given in the midst of high-stress settings with numerous distractions and background noise. To compensate for this, a group of students working with Huang is creating systems for communicating by speech that include computer lip reading.

Another natural way of communicating with the computer is through the use of hand gestures. Based on other research on real hand movements, Huang and his students have established 27 parameters that describe hand movement in three dimensions. Ultimately, human-computer communication will take place using lip reading, audio speech, eye gaze, and gestures. This is also an aspect of research being pursued in the Yamaha project (see article in this issue), with the hope that the more natural human-computer interactions are, the more people will use and benefit from computers.

Not only do soldiers and commanders have to be able to give verbal commands, but also the computers or the people receiving the commands need to be able to hear them clearly. The group led by Robin Bargar is working on enhancing the listener's ability to hear messages in a noisy environment. Their work is based on a unique system they developed for "sonification," a method of transforming scientific data into computer-generated sounds. The system has a wide range of applications, not only for the sonification of scientific data but also for the synchronization of interactive audio for multimedia and web browsers. This group's findings will be integrated into a prototype auditory head-mounted display that includes enhanced voice signals, synthesized sound symbols, intelligent mixing of sounds, and localization.

Another communication system being developed relies on the tactile sense. David Beebe (see profile in this issue) and graduate student Hui Tang have developed a rifle-mounted communication system. One use for this communication system, says Beebe, might be found when a group of soldiers is moving, for example, single file through a jungle at night. With tactile sensors and vibrators on the rifle for receiving and sending information, soldiers can keep in touch without talking and while maintaining audio and visual surveillance of their surroundings.

The system has three pressure sensors, each of which can register three levels of pressure; and three vibrators, each of which can provide three levels of vibration. Tests are now underway to see how best to train soldiers to communicate this way and to determine the psychophysical factors that affect this communication system.

The trick will be to integrate each of these research achievements into a single, coherent system. As Mehrotra puts it, "if the techniques we come up with work with FOX, then we've succeeded."

Another measure of success, less tangible but equally important, is the new relationship that the Fed Lab is helping to forge between academia, industry, and the military.

"The military today wants to have closer working relationships with industry and academia," says Harris. "The whole idea of this Fed Lab, which is a cooperative agreement, is a new concept for the Army. They want the developments from academia and industry to come to them faster and in a more usable form."

The Army holds an annual two-day symposium in Washington, DC, at which faculty and students present papers and posters and run demonstrations for top Army personnel involved in the project. Harris estimates that either someone from the Army or another consortium member visits the Beckman Institute at least monthly to confer on ongoing research or plan for future research. Other visitors also come, usually researchers interested in collaborating on some aspect of the project.

Wickens has sent a graduate student to West Point to run experiments on Army personnel. The Army has sent its members to the university on a semi-permanent basis, to pursue graduate training, to collaborate with researchers, and to provide an intimate knowledge of the Army; and Institute researchers have observed training sessions to learn about tactical operations command. "There is some video teleconferencing, and much of our work gets done via e-mail," says Harris. "But face-to-face meetings are also essential."

The collaboration benefits all parties. By including the Beckman Institute in the Fed Lab consortium, the Army leverages the University's expertise, notes Harris. For example, several pieces of equipment being used in the Fed Lab project were purchased through a National Science Foundation infrastructure grant awarded to Huang. That grant examines, among other things, 3-D modeling of the hand. This lays the groundwork for computer communication via gesture-an essential element in the Fed Lab project. The Fed Lab project also gains access to Beckman facilities, the university facilities, and an "army" of graduate students.

The Army, on its side, is learning how universities work. For example, the Army has nothing comparable to the large number of graduate students who do much of the work for the Fed Lab project. "The Army personnel appreciate very much all the input from our students," says Harris. "The students are really invaluable. We sent one student per every faculty member to the annual symposium and, for the most part, it was the students who presented papers and ran demos for top Army personnel." And Beckman Institute researchers benefit by being involved in a major collaboration that enables them to develop some cutting-edge technology and to create new applications for much of their work.

-by Debby Aronson, freelance writer