April 2017 Digital Edition
March 2017 Digital Edition
Feb. 2017 Digital Edition
January 2017 Digital Edition
Nov/Dec 2016 Digital Edition
Oct 2016 Digital Edition
OPINION / Using good technology to find bad people
By Jonathan McDonald
Many government organizations are charged with combating terrorism, identifying criminals and effectively applying immigration policies. All these activities require that persons of interests (and organizations and networks) be accurately identified. Identification is an extremely complex process that is only compounded by increasing data volumes, complex relationships, and tightly coupled application and data architectures.
“Entity resolution technology” enables security personnel in intelligence, integrated law enforcement and immigration and border control agencies to rapidly identify persons of interest and detect and defeat threats before they occur. Gartner, a highly regarded research firm and consultancy organization, has been tracking the entity-resolution market for several years and stated in a recent report:
“Entity resolution and analysis was previously an obscure technology that has come to the forefront as a result of world events and market forces where it is used to identify the use of false identities and networks of individuals who are attempting to hide their relationships to each other.”
There are a number of government organizations responsible for counter-terrorism and counter-intelligence initiatives, including the Departments of Homeland Security, Justice and Defense, and other law enforcement, intelligence, immigration and border control agencies. These groups share many common goals in their efforts to combat criminal activities that impact national security. These goals include:
Stopping terrorism by finding potential threats before they happen;
Reducing crime by identifying potentially harmful persons of interest when in custody, and during field investigations and analysis;
Protecting the nation from potential threats trying to enter the country and detecting other unlawful border crossings;
Meeting or exceeding citizen service expectations (credentialing, verification, renewals, etc.) in an increasingly critical and high-risk environment;
Reducing costs, increasing efficiency and improving the outcome of national security activities and investigations.
Despite their common goals, intelligence, integrated law enforcement and immigration and border patrol organizations have their own unique set of demands and requirements.
Intelligence community -- Connecting the dots
The intelligence community (IC) is comprised of many agencies responsible for conducting a myriad of intelligence activities that protect national security. The IC is made up of, among others, the National Security Agency, the Central Intelligence Agency, the National Geospational-Intelligence Agency, the Defense Intelligence Agency, and the intelligence components of each of the military services. One of the missions that many of these organizations have in common is the need to perform counter-terrorism and counter-intelligence activities that connect the dots between terrorists, foreign intelligence operatives, persons of interest, and events or locations that might be of interest.
One of the jobs of an intelligence analyst is to identify, assess and monitor potential threats that are actively trying to mask their nature, status and intent. There are two ways that IC analysts gather and use data that could help with their investigation.
The first is to analyze records in databases that already exist. The second involves analyzing raw reports of real-time data coming in from the field: snippets of information, pieces of recorded phone conversations and other data that does not yet reside in a database. Recently collected field data is often put into a temporary data store, so that agents can quickly perform link analysis to determine whether people or events are related. This process attempts to resolve multiple conflicting reports and identify a single entity, which can be an extremely difficult and complex process.
Even when searching existing databases for potential threats, most intelligence analysts are accessing hundreds of disparate structured and unstructured data sets from public, internal, cross-organizational and clandestine sources. Each data set has its own schema, varies in terms of completeness and quality, and is typically voluminous and highly dynamic. Clandestine data from covert sources are especially problematic, as they often contain dirty and incomplete data. In addition, agents are often required to conduct cross-language script matching.
Despite the challenges, there is high value in introducing technology that can accurately assist with identification and resolution of entities (individual and complex) and associated relationships (both declared and inferred) across multiple data sets.
Integrated law enforcement -- Catching criminals
The integrated law enforcement (ILE) community includes federal, state and local agencies, each with their own stovepipes of data. The activities of ILE are comparable to the IC in that officials are looking for entities of interest in an investigative capacity. The threats are also similar, if the target is an individual offender, an organization (such as gangs), individual terrorists or terrorist cells and the members that comprise it. The difference is that ILE investigators deal exclusively with crimes.
ILE usually tracks different information than the IC. For example, crimes are tracked as their own entity to identify activities that are similar in nature. And, data about criminals such as known associates, vehicles, and tattoos or other marks are usually collected.
The ILE community’s challenge is how to find a known entity when data about them resides across multiple data silos. These silos are the result of the program- and application-centric nature of technology implementations within law enforcement, and the fact that each law enforcement agency has data about individuals, within its jurisdiction, who have been convicted of or charged with criminal activity. Ideally, data would be integrated to allow ILE organizations and agents to obtain a holistic, entity-centric view of the item of interest, while upholding privacy laws and controlling access based on a user’s need to know. Entity resolution technologies can tie these stovepipes together to create a complete view of each person of interest, while allowing the data to stay under the control of its data owner.
Immigration and border control -- Keeping out the bad guys
The Immigration and Border Control (IABC) community needs to quickly identify persons of interest from the more than one million passengers a day that enter the U.S. through ports, airports and border crossings. Each person entering through a port of entry is reviewed and checked against relevant data sets to determine whether they are, or are associates of, known terrorists, or affiliated with terrorist organizations or any person of interest on a watch list. Most individuals are legal entrants, not persons of interest, so the challenge for IABC agents is to be able to review documents rapidly to enable legal entrants to pass through quickly, while ensuring that undesirables are denied entry and are processed in the manner appropriate to their level of potential threat.
Although much of the information used by IABC is similar to IC and ILE, some is different. For example, just as the ILE tracks tattoos, IABC tracks the location where an individual is crossing the border as an additional entity to provide insight into the potential criminal groups with which that person may be affiliated.
Entity resolution technology used by IABC needs to be able to rapidly and accurately identify entities of interest, prevent legal entrants from being mistakenly identified as persons of interest, and ensure that the majority of people are crossing borders quickly with minimal impediment. The technology must be capable of processing a large number of transactions and deliver a response in 20 seconds or less. Access to specific details within the data may not be allowed or warranted, so there is often a need to support tactical situations where stoplight indicators (red / yellow / green) are required in response to queries by officers in the field.
Government organizations are using entity resolution technology to rapidly create high confidence, accurate and complete real-time views of persons of interest from enterprise applications and data sources distributed within and across organizations. It allows organizations with limited security resources to more effectively identify, assess and monitor potential threats. The technology is able to resolve multiple entity types accurately and simultaneously, and manage their associated relationships.
Entity resolution technology is able to handle hundreds of millions of records in sub-second response times and provides unsurpassed matching and linking technology that identifies and resolves information routinely, even when there is duplicate, fragmented, incomplete or dirty data. By disambiguating distributed data sets into resolved entities of high confidence, government agencies charged with keeping us safe can help detect and defeat threats before they occur.
Jonathan McDonald is vice president, intelligence, defense and national security markets for Initiate Systems, a provider of data management solutions for information sharing. He can be reached at: