Concerns over facial recognition software use

Your face is unique; it's your calling card. Now imagine if you were recognized in a crowd via a surveillance camera – linked perhaps to your driver's license from the state database, or another database – and identified as a possible criminal and hauled into a police line up of suspects.

While that may sound farfetched, it could be a reality through the use of facial recognition by law enforcement to catch suspects for all kinds of crimes, from shoplifting to car theft to fraud or terrorism. Facial recognition can provide a certain amount of deftness and simplicity to our lives through some commercial applications, such as airports that are now using facial recognition as a security tool, allowing us to get through TSA pre-check lines with ease and quick express. At Detroit Metropolitan Airport, Delta Airlines is using facial recognition at some gates to expedite boarding. Casinos, including Detroit's MGM and Motor City Casino, use facial recognition to keep track of cheaters entering their establishments – as well as to notify staff when a “whale” – or a big spender – is coming in who they should lavish drinks and attention upon. “Eyes” are on us in crowds such as sporting events and concerts. Shopping mall security staff can scan through surveillance camera feeds to see if noted shoplifters or other criminals are floating around in the throngs.

But facial recognition also captures images of you and I – for good and bad. And a question that has authorities and ethicists pondering is how accurate and reliable facial recognition is as a tool of law enforcement, and what invasion of privacy concerns there may be.

Facial recognition is technology that is capable of identifying or verifying a person from a digital image or from a frame of video that comes from a video source. Also described as biometric artificial intelligence, it is an application that is supposed to uniquely identify a person by analyzing patterns based on their facial textures and shapes. While there are multiple methods for facial recognition, they all come down to comparing a scan of your face to an image of faces within a database.

Facial recognition was initially a form of computer application that has now been expanded to wider applications on mobile platforms and other forms of technology, such as robotics. It is often used as an access control in security systems, and is compared to other biometrics such as fingerprints and eye iris recognition systems. It is considered more reliable than a fingerprint and nearly as accurate as iris recognition, and is less-invasive and contactless than fingerprinting. As the technology evolves, it is finding commercial uses, such as in identification and marketing, video surveillance, automatic indexing of images, human-computer interaction, video databases and tracking.

An example of a well-recognized commercial application that many of us utilize on a daily basis is our iPhones. The latest incarnations of Apple iPhones switched to facial recognition over a fingerprint to open in the last couple of years. Apple said the version of the technology, which they call Face ID, uses a suite of sensors to map our faces in 3-D. Infrared light illuminates the face, and a projector in the phone projects an array of infrared dots at it, while an infrared camera snaps an image of the dots, which the phone uses to authenticate against an already stored image of your face. Apple claims the feature is so secure, there is less than a one in a million chance that someone else could spoof you, which is much more reliable than a partial fingerprint on the former “home” button to unlock the phone.

Anil Jain, University Distinguished Professor in the Department of Computer Science and Engineering at Michigan State University, said he has been working on facial recognition for 25 years, and it has changed a great deal from its infancy.

“It's a fairly mature technology now, but it will drive us for many more decades,” Jain said.

Moji Solgi, director of AI and Machine Learning at Axon, a technology company, said that face recognition is a broad term that lumps together a collection of technologies. It includes face detection, face tracking, face re-identification, face matching and face attributes. Face detection finds faces and their locations, he noted, and “most commodity digital cameras, including mobile phones, run face detection to enhance image quality.”

Face tracking corresponds a face from one frame to the next, and allows for the face of one person to be matched between two frames. “This is useful for a police agency when they need to blur out an individual's face in a body-worn video (such as a body camera) that they want to release to the public,” Solgi explained.

Face re-identification is similar to face tracking, but frames are not necessarily consecutive in a video. Your face may be at the beginning of a video, and then not seen again until the end. But, Solgi said, face re-identification can recognize it's the same face without identifying your face by comparing it to a database of faces.

With face matching, a target face can be given along with a set of candidate faces and candidate matching will find which one of the candidate faces belongs to the target face. “This is where algorithms meet databases for face search and retention,” he explained. Some photo storage applications use face matching, such as Facebook, to tag a face that appears in various photos, and it's the application technology that allows smartphones to unlock the phone.

Face attributes can take it further. More data has been fed into the algorithms to allow face attributes to extract information such as gender, ethnicity, emotions, age, facial landmarks and other identifying recognition.

Jain said the primary difficulties, however, with facial recognition is in its acquisition of faces. The problem is often they're too perfect in their setting and exposition.

“It is usually in a constrained imaging environment,” he noted, “where the subject is cooperative. You have to pose in front of the camera, take off your sunglasses, your hat. You are indoors, in controlled neutral lighting, posed, six-inches in front of the camera. The poses eliminates expressions. That makes identification from facial recognition much easier.

“In these applications, for access and government identification, the subject follows directions, and the algorithms are much easier to create,” Jain explained.

He said the guidelines on how to take a mugshot are very similar. We've all seen it on TV crime shows – look this way, straight at the camera. It gives law enforcement a clear, head on photo of the suspect.

The bigger question is how, and if, law enforcement uses facial recognition technology to help them apprehend bad guys or identify suspects in line ups. Birmingham Police Chief Mark Clemence said, “We don't have any equipment to use it.”

Oakland County Sheriff Michael Bouchard said they do not use facial recognition technology – yet. “That's one of the technologies we're constantly monitoring,” he said, noting it currently has too many false positives, but that commercial applications are there, so he anticipates its use is on the horizon.

“The ability to know you've got a bad guy on your hands sooner rather than later would be great, because lots of times when they come in the back door, they lie to us. If there's highly accurate facial recognition just as they enter the system, it's invaluable.”

Wayne County Sheriff's Office Chief Robert Dunlap said they also are not using facial recognition technology “at this time.”

“I use it for my personal use on my (Microsoft) Surface Pro (tablet), and I think it's excellent,” Dunlap said. But for the department, “We have just not acquired the technology yet. It's fairly new.”

Ming Dong, professor and co-director of the Department of Computer Science in the College of Engineering at Wayne State University, is also working with Ford Motor Co. on facial recognition to help determine the gender, age and other characteristics of drivers and passengers in cars for autonomous vehicles, as well as the use of camera-mounts in cars, “capturing the facial image of a person, because it can provide greater safety precautions.”

He said his department at Wayne State has someone collaborating with Detroit Police Department, “and how the cameras mounted on a police officer can recognize a person and certain events. That potentially can detect valuable information very quickly in the database. The camera can recognize a person from a body-mounted or vehicle-mounted camera within seconds, in real time.”

Dong continued, “The technique is substantially similar to detecting facial recognition for any application. The next step, I see face recognition as a part of AI (artificial recognition) with potentially a lot of applications. It will transform our lives.”

Despite the proliferation of surveillance cameras throughout the central business district of downtown Birmingham, Clemence said, “We've never once used the surveillance cameras,” for facial recognition, “and we don't anticipate using them.” Part of the reason is the inability of getting clear, usable images from the cameras.

He noted that Delta Airlines is beginning to use photos for facial recognition to speed up the boarding process. “The Delta photos will be head-on, perfect photos,” Clemence noted. “But most surveillance photos are angled, or people are looking down, and they're not perfect.”

Jain noted that having machines verify through facial recognition images caught from surveillance cameras “is more challenging because people are passing by, but they're not posing,” even though there are more and more surveillance cameras all over.

In downtown Detroit, a network of 2,000 surveillance cameras was installed in late 2017, paid for by Detroit Conservancy 300, a non-profit organization which manages Campus Martius under a contract with the city of Detroit, providing a live feed to Detroit police and the security operations of several downtown businesses, including Dan Gilbert's Rock Ventures, DTE Energy, Blue Cross Blue Shield of Michigan and Ilitch Holdings. The cameras, like in Birmingham, are not capable of facial recognition.

That doesn't mean that Birmingham or other police agencies have never thought of accessing a facial recognition database. They just turn to outside help for their expertise.“Sometimes if you have a serious crime or good picture off of surveillance (at a business), we'll request assistance from Michigan State Police – they're the ones who have the expertise. Michigan State Police is on the cusp of all of that – although it's not like CSI on TV,” Clemence said.

Dunlap said of the plethora of surveillance cameras in Detroit, “Any type of camera that can capture an image can be useful – the question is privacy. Every citizen is not someone we need to acquire intelligence on. Airports are very useful places to do that. After 9/11, I thought that was advisable. It's an imply/comply consent – that way airlines have a record of who is on that plane. You're giving implied consent when you get on that plane. But it's a different story when you walk down the street, going from point A to point B. You might not have a choice of where you're going, and you don't have a choice of having your image captured.”

At last count, there were 24 surveillance cameras around downtown Birmingham, primarily to provide assistance to police for safety issues, to provide a feed to dispatch at the police station in order to send officers if a crime was seen on camera, or a citizen was seen in trouble. In contrast, Clemence said, in London, England, “every license plate coming in and out (of central London) is identified and tracked. As technology is improved, it's the wave of the future.”

London uses automatic number-plate recognition, a technology that uses optical character recognition on images to read vehicle registration plates to create vehicle location data. According to the Electronic Frontier Foundation, 200 U.S. cities use the technology, with a current database of 2.5 billion license plates scanned – 95 percent of which were not under suspicion of any wrongdoing. Similarly, facial recognition databases do not determine if someone is suspected of doing anything wrong – they just hold images. Some databases include mug shots of criminals or suspects – and others are of Michigan drivers' licenses.

“I do hope it gets to that, with the amount of surveillance cameras and the number of pictures out there from cell phones, home and store cameras, it could really help the number of retail frauds done by retail fraud groups,” Clemence said. “We have tons of photos now. But the problem is, we have absolutely no idea who these people are. There are tons of retail, fraud, identity theft, tons of credit card fraud at gas stations – but we don't have the technology to identify them – and the bad guys know it. They are organized and they move around from town to town. If we get to the point where we can use facial recognition successfully, it'll be a different game. When technology gets to that point, it will be great because we'll be able to catch way more bad guys.”

Angie Yankowski, section manager, digital analysis and identification section (DAIS), Biometrics and Identification Division, Michigan State Police, said that facial recognition is available for the state police to support all criminal investigations for which there is a law enforcement purpose.

“Facial recognition is used successfully to identify subjects without identification on a traffic stop and can assist detectives in developing a suspect in a criminal investigation when surveillance video or other suspect images are available,” Yankowski said. “Additionally, we have a project in which we are using facial recognition technology to detect potential fraud within our copy of the Michigan Department of State's driver's license database.”

The Michigan Secretary of State's office stores all of the information each of us has on our driver's license or state identification card, as well as the information that is given when a person applies for a license or ID card, said Fred Woodhams, former communications director for the Michigan Department of State. “As required by state law, the department allows law enforcement agencies access to the images.” Beyond that, Woodhams, who left his position with the incoming administration changeover, “I can't speak to what happens with the photos.”

In other words, they provide the database, but offer no interpretive expertise.

The Pew Research Center countered that, noting that “fraudsters or drivers with serious violations try to beat the system by getting multiple licenses using different names. The implications of the deceit are far-ranging: People use driver's licenses and state IDs to do everything from cashing checks to opening bank accounts to boarding domestic flights. States increasingly are foiling the crooks and scam artists by employing a high-tech tool – facial recognition software. The software uses algorithms of facial characteristics to compare driver's license or other ID photos with other DMV images on file.”

The Pew Center said that at least 39 states currently use facial recognition technology in some format or another, with excellent results.

“You have an opportunity, using this technology, to find people who are trying to skirt the system. It has really helped to identify fraudsters,” said Geoff Slagle, director of identity management for the American Association of Motor Vehicle Administrators.

The technology in the database works by taking a photo of anyone who has a photo for a driver's license or state ID – and the image is converted into a template created out of the individual's unique physical features, such as their cheekbones or the distance between their pupils. An algorithm compares the image with others in the database in a search for a match.

Sometimes, someone has gotten married or divorced and changed their name. But often, Slagle's agency said, it's “a deliberate attempt to violate the law,” whether because someone lost their license due to DUIs, and wants to obtain another one, or they want to get credit, buy a car or get a mortgage using a false identity.

Kevin Bowyer, Schubmehl-Prein Family Professor of Computer Science and Engineering, University of Notre Dame, whose specialty is computer recognition, as well as facial recognition and iris recognition, said a project in the news “was comparing driver's license photos from different states, where subjects would obtain multiple licenses to avoid tickets. It may be a truck driver who gets a license from several states, for example. There are a lot of reasons you would want to get licenses, from underage students to people who have lost their legal license. It would not necessarily be a security issue, but could be a safety or legal issue.”

“Still others are wanted felons or criminals, such as sex offenders, who are trying to hide their identity by using an alias,” said Jenni Bergal of the Pew Center. “Facial recognition has led to numerous arrests and administrative actions against drivers.”

“Facial recognition technology is helping to protect people not just from identity theft and fraud, but as drivers and neighbors,” said Betty Johnson, an administrator in Nebraska's Department of Motor Vehicles, which has used facial recognition technology since 2009. Since then, about 3,000 people have been prevented from getting a license and over 300 have been arrested for crimes they were sought for, and were identified via the technology.

Yankowski is careful to point out that, “per Michigan State Police policy, facial recognition is not considered to be a form of positive identification. It is considered to be an investigative lead only, requiring the investigator to continue the criminal investigation before making any final conclusion.”

She said that facial recognition search algorithms have advanced greatly over the last five years, “improving our ability to provide investigative leads in support of criminal investigations.” Since 2013, the Michigan State Police's Statewide Network of Agency Photos (SNAP) Unit's trained digital image examiners have processed over 9,500 facial recognition requests and provided over 2,800 investigative leads in support of criminal investigations. Their program adheres to an acceptable use policy addressing both auditing and penalties for misuse, Yankowski said. “My team of trained digital image examiners conduct random and targeted audits to ensure compliance with the policy.”

However, there are other factors that can impact the outcome of a facial recognition search, which can include image collection, such as compression of the image and camera position; image capture, in terms of perspective, aspect ratio and lighting; the subject's pose and facial expression, and obstructions that can include eyewear, hairstyle changes or color changes, clothing, and other intangibles. Yankowski said the state police department does not have real time video/facial recognition capabilities, and does not use it for mass surveillance situations.

A nationwide concern for facial recognition has been a bias against African American and Hispanic faces, where there have been instances of facial recognition misidentifying them as animals, an issue MSU's Jain said has been caused by those inputting the information into the computers.

It became so bad, in July 2018, the ACLU did a test of Amazon's facial surveillance technology software, called “Rekognition,” the software incorrectly matched 28 members of Congress, identifying them as other people who had been arrested for crimes.

“The members of Congress who were falsely matched with the mugshot database we used in the test include Republicans and Democrats, men and women, and legislators of all ages, from all across the country,” said Jacob Snow, technology and civil liberties attorney, ACLU of Northern California.

However, he said, the false matches were disproportionately people of color – including six members of the Congressional Black Caucus, among them noted civil rights legend Rep. John Lewis (D-Georgia).

“Our results validate this concern: nearly 40 percent of Rekognition's false matches in our test were people of color – even though they make up only 20 percent of Congress,” Snow said. “If law enforcement is using Amazon Rekognition, it's not hard to imagine a police officer getting a 'match' indicating a person has a previous concealed weapons arrest, biasing the officer before an encounter even begins. Or an individual getting a knock on the door from law enforcement, and being questioned or having their house searched, based on false identification. An identification – whether accurate or not – could cost people their freedom or their lives.”

Oakland County Deputy Prosecutor Paul Walton said they have not yet used facial recognition in a prosecution due to “its inherent reliability. We still need someone to come in and say it's reliable,” to be used at trial.”

In order to reach that point, he said, they would need to have a Daubert standard to establish if the science is reliable. In U.S. federal law, the Daubert standard is a rule of evidence regarding the admissibility of expert witnesses' testimony, based on a Supreme Court hearing, Walton explained. It was done to initially determine if DNA tests were accurate and reliable.

Walton said, to date there are no federal or state cases in Michigan that have used facial recognition for identification purposes. “A police officer cannot testify that person is the same person on the videotape unless he has an added element of familiarity – for example, he knows him from someplace else as well.

“We encounter a lot of camera surveillance, but not a lot of uniformity. The quality is not good enough to give facial recognition, especially from smaller businesses, mom and pops – there's a myriad of factors,” Walton said. “When there is a reliance on Homeland (Security), they're very clear. But otherwise, we deal with the quality of storage, the quality of digital retrieval ability, obstructions in the photo, sides of faces, distance. It's even not uncommon where propriety of the software is an issue – where we get a video and have to download software from a camera, and then to play it, we lose quality and the image that allows us to recognize a face.

“My gut feeling is that when you have a very controlled environment, it's reliable, but probably not in an uncontrolled environment. I think we're at the ground level, and someone has to come in at a commercial level to improve quality levels,” he said, acknowledging that tech companies are working on it. “I think we're at Beta versus VHS. Which one are you going to go with?”

Snow pointed out that Amazon – like many other technology companies – is actively marketing its facial recognition technology to law enforcement. The ACLU is urging Congress to enact a moratorium on law enforcement's use of facial recognition – but there has been no action on that. Nor is there any legislation in Michigan, or in any other state. State Senator Peter Lucido (R-Shelby Township) twice introduced legislation as a state representative to prohibit providing digital photographs in a federal database that uses facial recognition technology, but both in 2017 and 2018, the bills died in committee.

Jane Bambauer, a law professor at the University of Arizona, said the Constitution doesn't provide much protection against facial recognition. Surveillance technology like wiretaps, she said, are covered by the Fourth Amendment protections against search and seizure. Most police interest in facial recognition is in applying it to imagery gathered lawfully in public or to mugshots, she pointed out.

“When you build the systems, you have to train a facial recognition algorithm that these are faces, and these are not faces,” Jain said. “If, in training, there are only white faces and not black faces, it will only recognize white faces. Also, sometimes the algorithms have difficulty distinguishing men from women. Women can be difficult sometimes to identify because they change their makeup, hairstyles, color. It is all what trainers train, and that is increasingly improving on the part of trainers – becoming more balanced.”

Another factor, Jain noted, is “changes in the face as it ages, including medical conditions, lifestyles, medicines you take – it can all change a face over time. If I took a photo 20 years ago, and someone takes a photo of me today, the accuracy may not be as good of facial recognition. Not so much for people who see us all the time, but for the job of a machine to pick out one from all the millions in a database.”

“Facial recognition is an effective way of authentication, although it's not bulletproof, as for instance, the user's biometrics can change over time. This is why it's important to use it with other identifiers like behavior biometrics so that law enforcement can create a more accurate profile of the user,” said Don Duncan, security engineer, NuData Security, which works for MasterCard. “With multiple layers of authentication, many data points can be evaluated in real time to reduce the chances of misidentification. Additionally, if one identifier is spoofed or inaccurate, then other identifiers can be used to provide a true authentication with fewer chances of false positives.”

“As far as the limitations of the technology, automated facial recognition depends on the mathematically determined similarity between facial images, and is most effective in controlled situations (frontal-facing pose, even lighting, neutral expression), which is rarely the situation encountered with forensic material,” said Yankowski. “The software might not recognize details that humans can perceive (changes in expression, weight or age), leading to images of the same person scoring weakly or having highly similar facial images of different people scoring strongly. For this reason, Michigan State Police stresses the importance of having highly trained – human – examiners reviewing the comparisons of the images and determining the conclusion regarding the comparison.

“Even once an examiner identifies a potential match,” she continued, “that match is not considered a positive identification – it is only an investigative lead, or possible identification, that requires further investigation.”

The FBI's Criminal Justice Information Services (CJIS) Division developed and have incrementally integrated facial recognition biometrics to improve upon and replace the application of tenprint (all ten fingers) and latent fingerprint searches in an effort to harness new technologies, they said.

“This new system, the Next Generation Identification (NGI), provides the criminal justice community with the world's largest and most efficient electronic repository of biometric and criminal history information,” the FBI reported. “Biometrics has been incredibly useful to the FBI and its partners in the law enforcement and intelligence communities, and the bureau continues to look to new scientific advancements to increase the range and quality of its identification and investigative capabilities.”

The FBI reported that going back all the way to July 1999, their CJIS division has operated and maintained the IAFIS, which is the “world's largest person-centric database.”

The Integrated Automated Fingerprint Identification System (IAFIS) is a national fingerprint and criminal history maintained in the United States by the FBI's Criminal Justice Information Services Division. It provides automated fingerprint search capabilities, latent searching capabilities, electronic image storage, and electronic exchange of fingerprints and responses. IAFIS houses the fingerprints and criminal histories of 70 million subjects in its criminal master file; 31 million civil prints; and fingerprints from 73,000 known and suspected terrorists processed by the U.S. or international law enforcement agencies.

“Because of growing threats, new identification capabilities were necessary. Advancements in technology allowed further development of biometric identification services,” the FBI reported. Facial recognition fits the bill.

“The NGI system improved the efficacy and accuracy of biometric services to address evolving local, state, tribal, federal, national, and international criminal justice requirements,” they stated.

Included among their biometric services is the facial recognition search, designed primarily to aid state and local law enforcement. “Authorized law enforcement may submit a probe photo for a search against over 30 million criminal mug shot photos and receive a list of ranked candidates as potential investigative leads.”

The ability for local law enforcement to access the FBI NGI system explodes their search results, Jain noted.

“The Michigan State Police have about four million in their database; the (Michigan) Secretary of State has about 30 million in their database. With those hundreds of millions in the FBI database,” the access could be a game changer, Jain said.

But there are issues. Accuracy and reliability are one; invasion of privacy is another.

“The issue is if data is misused in some way,” said MSU's Jain. “But we have facial recognition capabilities with our smartphones.”

“History has shown us that if it's more convenient, or cost-effective, people will go along with it. People are willing to give up some of their privacy for convenience,” said Kevin Bowyer of University of Notre Dame. “People do for social media – every photo that is uploaded you can assume that facial recognition is being utilized and then marketed somehow more effectively to you. And it doesn't bother us, and companies keep doing it. Every large corporation and technology company is doing it. Government agencies are struggling to keep up.

“Look at our cell phones. You want your phone to be unlocked by you. Every vendor has played with facial recognition, and Apple got the most play from it, and dropped the partial fingerprint. They understood that facial recognition was more secure than a partial one centimeter fingerprint,” Bowyer continued. “Every smartphone vendor is looking at low security, high ease of use biometric to operate your smartphone. You have to have a high security biometric for banking and for financial transactions, and that's where high end facial recognition or iris may work. Still, facial recognition is the best right now, even though it needs improvements. It's still the most secure and accurate. And the ease of use is good for everyone.”

“Yes, there's an invasion of privacy. The thing is, if you're walking around, they have a right to take your photo. If you're minding your business and not violating any laws, there's no reason (for law enforcement) to look at those photos,” said Jain. “But, if you're doing something suspect, that's what the police are for – and that's not an invasion of privacy.”