Crossing the Longest Yard: Comments on Challenges and Opportunities for Level-5 Multi-sensor Data Fusion

Dr. David Hall

Even before the 9/11 attack, the U. S. military and intelligence communities had expended huge amounts of resources to develop new types of sensors and surveillance methods. Advanced sensors range from the development of nano-scale, smart sensors in distributed sensor networks to national level sensors involving image and signal collection. The trend has been an ever increasing ability to collect data in a huge “vacuum cleaner” type of approach.

The bulk of the literature on multi-sensor data fusion focuses on the automation of target tracking and automatic target recognition (level-1 fusion). While such research is needed, current problems involve complexities such as identifying and tracking individual people and groups of people, monitoring global activities and recognition of events that may be a precursor to terrorist activities. The requisite data includes sensor data, textual information, and utilization of models. This analysis process is human intensive and requires teams of analysts to search for data, interpret the results, develop alternative hypotheses, and assess the consequences of such hypotheses. Our perception is that researchers have started at “the wrong end” of the data fusion process.

Researchers have started at the input side and sought to address methods for processing sensor data to automatically develop a situation data base and display. We argue that research on the user-side of the data fusion process (level-5) is still relatively immature. Intelligence analysts are immersed in a sea of data (drowning in data), but thirsting for knowledge. A key issue is how to cross the longest yard – that is, how to transform data within a computer to knowledge within a human analyst.

This presentation describes innovative techniques for improved understanding and retrieval of data associated with intelligence analysis, in a multi-sensor environment. We are developing new concepts for assisting analysts in information retrieval, pattern recognition, hypotheses generation and evaluation, and situation assessment and understanding. Our extreme data interaction approach is exploring a variety of techniques including semantic-level fusion, multi-sensory interfaces, deliberate synesthesia, adversarial game methods, and cognitive aids for bias remediation and collaboration.