Analytics Applications Provide Actionable Results for Homeland Security

Download the PDF

The scope and alignment of intelligence and law enforcement resources has been in rapid flux since 9/11. Traditional data analysis has been superseded by a new emphasis on data interconnectivity, translating information into quick action, and more rapid deployment of software-based analysis tools and supporting systems. The first wave of software tools, though providing unprecedented data handling capacities, fall short at the critical juncture of being able to integrate those outcomes and put them into action.

An Integrated Platform for Complete Applications

Building on the success of its two flagship products, IR Discover™ and IR Analytics™, Intelligent Results anticipated the need for the dynamic integration of applications that offer analysis to those that enable action to fulfill the evolving Homeland Security mission. The new Predigy™ platform offers application development modules for data mining, analysis, outcome prediction and simulation, programmable action strategies, and reporting. Predigy provides the means for defining and publishing complete run-time applications that are capable of addressing the most challenging intelligence objectives.

Each published Predigy application becomes a self-contained component that can be reused and included in the definition of even more sophisticated applications and action strategies. The Predigy Platform has three fundamental advantages:

  • It supports the full suite of analytical, planning and reporting functions required for comprehensive data-driven solutions.
  • It provides an integrated development environment for customizing, testing, and publishing analytical applications for operational use.
  • It supports sharing and reuse of analytical applications as components of new applications.

Analytical Applications

An analytic application is a self-contained run-time software executable that has been designed to perform a data analysis and/or action function. It may be composed of single or multiple components, making it both powerful and easy to customize. Some examples of components that an analytical application may include are mixed data extractors and processors, prediction models, classifiers, event detectors, action strategies, report definitions, and connectors to other systems for data routing and alerting.

To gain value from both text and structured information, Intelligent Results has developed adaptive methods that consider the contributions of individual words, phrases, concepts, temporal ordering (sequential patterns), and combinations of features.

Data Extractors and Processors
Predigy's data extraction and processing components automatically extract structured, unstructured/text, temporal, and sequential information from a broad range of data types. Intelligent Results has developed technology to identify and organize key attributes from documents using techniques such as:

  • Unsupervised Classification - automatically extract the attributes necessary for classifying every document.
  • Creation of User-defined Attributes - use a few good examples of a concept or lexical query.
  • Use of Semi-Structured Expressions - such as XML/HTML tags, text labels or other regular expressions in the data.
  • Identification of Structured Attributes - from database fields, such as date, time, country/region and organization.

Prediction Models, Classifiers and Event Detectors

Predigy provides the development environment for easily creating and testing models, classifiers and event detectors that are capable of predicting the likelihood that any given document fits a targeted pattern. Predigy's IR Modeler™ module automates many of the analytical methods used by Ph.D. model builders and statisticians. All of the attributes extracted from the data sources are evaluated for their predictive contributions to potentially thousands of model variations, until the most powerful model is created and validated.

The IR Discover™ module provides the exploration and analysis tools to reveal hidden patterns and build custom classifiers. Other classifiers automatically structure data into conceptual clusters or categories, based on rules and mixed attributes. Event detectors with alerting actions can be created using a combination of modeling and classification techniques.

Action Strategies and Simulations

Predigy's IR Strategy™ module allows the strategist to interactively construct decision trees for defining actions. Tight integration among the Predigy Platform components means that IR Strategy decision trees are able to use any combination of predictive models, classifiers, event detectors and other attribute values to segment data and execute any desired action, such as alerting to events or trends, routing high probability documents to distribution lists, and aggregating statistics on the data stream for reports. All action strategies can be simulated in IR Strategy prior to export to the operational run-time component, IR Production Engine™, which scores each document in the data stream against all models, selected attributes, and action strategies, and executes the programmed actions in real-time.

IR Strategy offers fully interactive strategy trees where the value of any data attribute or model variable can be a decision point and the entire strategy can be simulated against any dataset prior to operational deployment.

Reports

Reports are defined and published at both the component and applications level. For example, IR Discover provides fully interactive online reports of analyses, and using IR Report™, any application can produce fully customizable reports on system performance, or aggregate statistics for data attributes and operational actions taken.

Connectors

Applications built with Predigy can exchange data with, and control the processes of, 3rd party applications using custom designed or included connectors. For example, a strategy may be defined that when all monitored documents exceed the minimum threshold for references to financing nuclear terrorism they are routed to a designated network directory for further analysis, and an email alert is sent to a distribution list of responsible personnel.

Application Development Environment

The Predigy platform makes it possible to build and validate powerful applications that address specific intelligence objectives. The IR Discover, IR Modeler and IR Strategy modules can be used independently or together in various combinations to reveal key attributes, establish functional relationships and identify the most effective action strategies to automate. Complete applications can be simulated in the development environment and then published to the operational environment as self-contained, runtime applications.

Analytic Applications as Components

All analytic applications, from simple component processes, such as a query, to complex systems of components, can be treated as self-contained run-time objects that can serve as additional building blocks for new applications. For example, an application designed to detect and report on specific types of financial transactions can serve as the input to another application designed to automate actions when certain patterns of transactions are detected. Libraries of related applications can be created to facilitate constructing new and more powerful applications.

Analytic applications are collections of analytic components that together define the processes to be executed in real-time on the IR Production Engine against unlimited volumes of data.

Example Applications for Homeland Security

Applications published from the Predigy platform are perfectly suited for deployment to environments where data volumes and performance requirements are high and continuous autonomous operation without fault is essential. Such applications will support both the strategic mission of intelligence collection and analysis, and also perform tactical analyses for the purpose of supporting urgent tasks, such as classifying and staging documents for immediate access and specialized analysis. Applications can be as simple as data routing systems using only select dimensions from IR Discover analyses as filters, or may include several embedded applications to leverage IR Discover dimensions and/or IR Modeler predictive models as input to an action strategy. The following examples describe two possibilities.

Example 1: Tactical Staging of Data
Many sources of data are available to the intelligence analyst who must respond to a Request for Information (RFI). Boolean queries and various search strategies are usually employed to find information potentially relevant to an analysis task. Queries against the largest databases can take considerable time to execute, with any errors in specifying the query wasting time and impacting database performance for other analysts. Such constraints make it difficult for analysts to work with several databases at once or to fine tune their search strategies.

A Data Staging Application created with Predigy can continuously monitor multiple sources of data and route copies of documents that satisfy search conditions to staging locations where analysts can readily access and interact with them. The advantage to this approach is that analysts will have the ability to stage various classifications of data as a starting point for RFI research and exploratory analysis using IR Discover. For example, document collections addressing certain regions of the world or organizations, or specific types of threats, can be updated in real-time so that they are always current and always ready for inclusion in analyses. These collections then become part of an analysis strategy with IR Discover selecting the documents of potential relevance to the RFI and creating a merged collection in a fully exploratory software environment. This wouldn’t preclude adding non-staged sources to the IR Discover collection, though any need to do so would be minimized and the productivity of analysts could be increased.

The following illustration shows a basic approach to using Predigy applications for monitoring and routing, and IR Discover for accessing and analyzing collections.

To create the Data Staging Application, analysts use IR Discover to define useful subsets of documents. The definitions of these subsets (dimensions) are then exported to the IR Production Engine for real-time monitoring of all interesting sources. Each document that satisfies one or more of the search definitions is added to one of the collections of staged data for later use. Each collection can be harvested by IR Discover and automatically updated on a regular basis so that a current and interactive analysis environment is always available. This enables analysts to quickly explore any collection of staged data. This approach significantly reduces the time required to respond to an RFI and will improve the product.

Example 2: Strategic Detection & Monitoring: The Financiers of Terrorism
Data is continuously being gathered about individuals, organizations and governments who may be associated with financing terrorists. In some cases the data is in the form of unstructured or semi-structured field reports, intercepted messages and interviews. In other cases, the data is in the form of bank and phone transaction records. The problem is cross-referencing the various data sources and identifying patterns that may indicate the sources and methods of terrorist financing. Ultimately, the goal of a strategic detection and monitoring application is to automate the review of data and to alert analysts to documents that are likely to be relevant.

With the Predigy platform, agencies can leverage the work of analysts who have already defined queries to find relevant information with the power of predictive modeling and action strategy trees. The synergies generated yield highly discriminating results and can be deployed as an automated detection application.

Detailed Review and Routing

The Venn diagram below illustrates how various definitions of the concept "financing" can be combined with definitions of terrorism and with geographical delineations. This type of analysis identifies documents that have a high likelihood to be related to information about terrorist financing in specific areas of the world—documents that would most likely have been missed using traditional tools. Even more discrimination can be built into a detector by adding additional definitions, such as target organizations, specific threats, personal names, etc. Predictive models can also be included to improve discrimination by looking for complex patterns in the data that are difficult to define using other methods.

As shown by the numbers in the Venn diagram and the table below, each document in all specified intelligence sources is tested against definitions for financing, terrorism and geography, and then assigned a risk classification of 1-4 and an action depending on the strategy design. Documents that satisfy all three conditions—financing, terrorist and targeted geography (marked with "X" in the table), are assigned the highest risk and priority, and automatically routed to analysts A. Other documents are assigned a lower priority and routed to other review queues. Each document can be further tested and assigned to more discriminating categories—each with a specific action assigned to it.

Using Applications within Applications for Appropriate Action

The illustration below shows how applications can be incorporated into other applications. In this case, the terrorist, financing and geographical detectors were defined and saved as separate applications for use with other analysis and monitoring scenarios. They are linked to the action strategy component in the master application, ensuring that each document that passes one or more of the three detector applications is acted on according to the logic of the action strategy.

In summary, the Terrorism Financing Detection Application demonstrates how complex analysis tasks can be captured and used to define an operational, stand-alone application. Once deployed, this application can autonomously and reliably monitor all sources for important patterns and relationships in the data and then take appropriate action, including alerting and routing high priority data to specific analysts and providing fully customizable data summaries and reports on all activity of the system. Multiple instances of the application can be deployed to cover all sources, with identical or customized alerting and routing lists.

One Platform for Actionable Intelligence

Intelligent Results’ Predigy Platform offers intuitive application development and deployment modules for data mining, analysis, outcome prediction and simulation, programmable action strategies and reporting. By capturing and automating the sophisticated routine analysis and modeling tasks, Predigy eliminates the need for computer language coding or special technical skills to produce run-time applications that address the most challenging intelligence objectives, including the monitoring, and automated detection and classification of communication and transaction patterns in high volume data streams. Predigy provides a truly integrated option to plan, predict and perform the kinds of critical analyses homeland security requires, at unheard of speed, without any compromise to quality.

David Lantrip, Ph.D., P.E., is co-founder and director of Insight Products at Intelligent Results. He brings 32 years of professional experience including human factors engineering, human/computer interface research, data-mining and visualization technologies development and marketing. David can be reached at david.lantrip@intelligentresults.com