UCCS research for facial recognition proved technology not ready

5 November 2019

Annika Schmidt

[email protected]

Terry Boult, El Pomar endowed chair of innovation and security and professor of computer science at UCCS, lead two research projects between 2007 and 2013 as part of a multi-university effort to analyze facial recognition algorithms.

The project was funded by the US Navy in an attempt to identify a suitable method for detecting human threats to their ports, bases and ships. At the conclusion of the research, the US Navy purchased the analysis of Boult’s results in 2013.

This specific research project involved secretly taking thousands of photographs of over 1,700 UCCS students in a public location on campus between 2012 and 2013. This dataset was then used to test both commercial and researcher facial recognition algorithms.

“One of the important questions we asked was how well the algorithms could handle non-cooperating subjects,” Boult said.

According to Boult, people’s faces change slightly if they know they are being photographed. Making the subjects aware of the study by getting permission would have defeated the purpose.

As part of the same research, the University of Maryland conducted research using a similar method but got permission from participating subjects. Boult also partially used this process, known as a cooperative biometric dataset, but the majority of the data from UCCS was collected from ‘non-cooperative subjects.

Since becoming public knowledge, the study has sparked privacy concerns. Despite controversy among the public, the research was in compliance and approved by UCCS’ Institutional Review Board (IRB). The IRB is an administrative body established to protect the rights and welfare of human research subjects.

Before Boult’s research was conducted, the IRB analyzed the proposed study and balanced the risks versus the potential benefits to society of the research.

“The answer was that there was very little risk,” Boult said. “Campus approved the research because it was in public and we were not identifying people.”

The biggest privacy risk that was identified was that an individual could recognize someone based on a facial photo in the dataset and know their location.

“If we took and released photos today, it would still be legal, but we wouldn’t want them knowing where you are,” Boult said. “I care about privacy.”

As a precautionary measure, the dataset could not be accessed for four years following the completion of the study in 2013, by which time photographed students would have graduated.

“It gives people virtually no information as to their current whereabouts,” Boult said.

Despite this, faculty, staff and students attending UCCS for more than four years could still be located by individual recognition, introducing a potential privacy risk.

The dataset is, however, not publicly available. The data is distributed under a research license. Individuals go through a process to get approved to receive the dataset and agree not to publish the photographs. Regardless, anyone could go through this process to access this data. No one can access the data anymore, however, because of a computer error. The mistake that caused the error occurred after the data had been cleaned up for use.

“We have taken the dataset down to clean it up again,” Boult said. Keeping the geotagging violates privacy and permits cheating in research-based competitions that use the dataset.

“We may rerelease the data, but we currently don’t have the funding to do the work in order to rerelease it,” Boult said. “It would involve a lot of work.”

Rereleasing the images would involve a process to protect the privacy of the participants.

Boult invited photographed individuals to come forward and delete their pictures from the dataset if they wished to be excluded from the data after the negative public response.

“People asked why I wouldn’t just remove them myself, but it’s because we can’t,” Boult said. “The quality of the images is so bad that I wouldn’t be able to recognize them in every single photo.”

The image quality from 50 feet away was poor and therefore the facial recognition was inadequate. The US Navy was looking for something that would be effective over much larger distances up to 200 meters, or just over 650 feet.

“The results of our research for the government was that this technology was not ready to be used,” Boult said. “The accuracy was horrendously bad.”

The best algorithms for long distance facial recognition were getting approximately 30 percent accuracy in 2013. The performance has improved over time, mostly because researchers are getting larger data sets.

By the end of 2016, UCCS was working on another project for the government, specifically Intelligence Advanced Research Projects Activity (IARPA). This project was more broad, but as part of that project, Boult was asked to evaluate how things had advanced in the last four years.

“At this point we did a competition,” Boult said. Researchers got access to the data and, using chosen algorithms, identified individuals in the photos, corresponding them to a specific number.

Two competitions were held, one in 2017 and one in 2018. Only about seven to 10 groups took part in the competitions since anonymous submissions were not allowed.

The best algorithms have about 55 percent accuracy as opposed to the 30 percent in 2013.

“That is significant progress,” Boult said, “But my goal is to caution the government that this is still not ready for some of the places that the government wants to use it for.”