How Urban Surveillance and Face Recognition Algorithms Track COVID-19

And in some cities, it's helping authorities intercept and detain those in most danger of infection, creating experiences not so different from John Anderton's fictional hallway.

RELATED: TECH LEADERS HAVE STRONG OPINIONS ON ARTIFICIAL INTELLIGENCE

Urban surveillance and privacy rights

If we're going by the numbers, roughly 89% of adults are in support of personal privacy rights — with 65% holding strong support — according to an April 2020 survey of 1,255 Americans taken by SurveyMonkey, a business with Better Business Bureau accreditation.

In the time of COVID-19, 52% of U.S. adults find personal privacy more appealing than safety measures which might throw it into question — even amid a global pandemic, when big tech powerhouses like Google and Apple are offering free apps to monitor contact tracing — a social distancing system that tracks points of potential person-to-person contraction of the virus.

Soon after the companies' joint announcement in April, they added that governments won't have a say in citizens' decision to use these apps, or not.

While the opt-in feature of apps is ostensibly a way to preserve privacy rights — it also creates layers of visibility within the coronavirus crisis: As apps rate the health of iPhone and Android users who opt-in, the ones who don't won't be visible in this system, either to users or to Apple and Google's contact tracing system.

"I think of the idea of visibility layers — technology supplying an invisibility layer over the real world," said App Creator Mo Saha to Interesting Engineering. Saha is one of the minds behind Antidate, an app from the twenty-teens that worked to give women more control over the online dating experience.

Similar to contact tracing apps' opt-in feature, Saha's conceptual dating app provided users with "an asymmetrical experience — [like] one-way glass between men and women, where women could see the men, but men couldn't see the women until the latter made a move."

People who don't opt-in to contact tracing apps won't necessarily know who did, which removes Saha's dating app's "imbalance in terms of exposure," unless joiners tell. Within the contact tracing system, no one user will see another without also being seen. But with face recognition surveillance of CCTV video streams, the idea of visibility layers comes back into play.

Internal versus external surveillance

If contact tracing apps are an internal, user-centric function of surveillance, the other half of the urban surveillance equation is face recognition algorithms. Connected to video streams from CCTVs and other devices, they work to identify and track people through varying environments.

There are "two ways to process [...] streaming video coming from cameras — at [the] edge, or sending it back to a central server and process[ing] it there — and there are different strengths and weaknesses to the two approaches," said Dr. Patrick Grother, a scientist at the U.S. National Institute of Standards and Technology (NIST), to Interesting Engineering. To identify faces in an image, "you have to run a face recognition algorithm, which can look at single frames or all frames of video."

Face recognition and computational power

As surveillance operations expand to encompass a larger population, so does the need for more cameras, and more powerful hardware. "Hardware requirements must exceed [the] number of cameras times the number of people times the frame rate of video as well — any system in busy [urban settings] would need to throw sufficient hardware at this issue."

A simple bank robbery requires only one recorded video to solve, which doesn't need to happen right away, said Grother. But monitoring at metropolitan scales, the video feed never stops — and it grows and multiplies as fast as the frame rate of every camera in the city. The "[d]ifference here is the real-time aspect — you've got to keep ingesting video and keep up with it," added Grother.

Notably, not all face recognition algorithms will process and identify faces in a video image at the same rate. "[S]ome will go in a tenth of a second, some 10 times slower — at which point you need to make some engineering trade-off," said Grother.

Real-time surveillance during the coronavirus crisis

While NIST doesn't develop or deploy algorithms in real-world scenarios, right now they're inviting industry developers to submit for testing new algorithms designed to recognize faces obscured behind medical masks.

And, according to a March study commissioned by the U.S. Department of Commerce, some of these algorithms are from a company called VisionLabs — a Russian computer vision and machine learning company. "When a face is detected in the frame a biometric template feature is extracted," said VisionLabs Senior Researcher Daniil Kireev, in an email exchange with Interesting Engineering.

Face recognition looks for unique and identifiable facial features in surveillance cameras, based on a "biometric template" provided from an earlier image of a person. Using CCTVs distributed throughout the city of Moscow, VisionLabs implements face recognition surveillance that can sift through "a quick comparison with multi-million-item databases," said Kireev.

Easing social distancing, face recognition, medical masks

Three months ago, people in major cities like New York or Chicago walked in urban spaces without worrying about the COVID-19 illness. Now, as the U.S. prepares to ease social distancing measures, many people will return to bustling areas of business and commerce, only with medical masks, which obscure faces, and leaves more room for error in face recognition processing.

When our water-soaked thumbs don't unlock our smartphones, this is a false negative — we have the correct thumb, but the fingerprint can't register through the layer of water.

The same can happen with face recognition and people in medical masks: if a CCTV camera catches a face that's mostly covered, there's a greater chance it won't identify the person behind the mask.

"Traditionally, recognition of faces covered by masks or clothes is a technically challenging task," said Andrey Khrulev, a business development manager at Speech Technology Center, in an email exchange with Interesting Engineering.

However, face recognition systems around the world have seen use on transports and city centers. In addition to these, Speech Technology Center's systems are even deployed at the Petrovsky Stadium in St. Petersburg — which is outfitted to process biometric data.

Khrulev added: it "often happens that part of the face [is] hidden by a fan hoodie or scarf (it is cold in St. Petersburg)." According to Khrulev and his colleagues, the need for urban surveillance to identify people obscured behind medical masks was there from the start, and the algorithms are adapting.

Real-time interception of people at risk from COVID-19

As urban surveillance and face recognition processing gets better at identifying potentially infected people in real-time en masse, new possibilities arise not just for social monitoring and contact tracing, but also for the ability to intercept people at risk of potential coronavirus infection. But it's important to note that not every country interprets what kind of action to take in the same way.

Russian citizens added to a quarantine list by their government are also put into a database of biometric systems. "If the people from [this] list are discovered in video recordings from the street cameras, in the entrance of a house, in a shopping center, the system automatically send[s] an alert or notification to the police," said Khrulev.

Barring a second, very serious wave of COVID-19, it's most unlikely this kind of police interception will happen in the United States. Apple and Google's apps won't share the health status of those who opt-in to their contact tracing apps, and the U.S. government departments that handle face recognition algorithms like NIST only test them, according to Grother.

However, it's important to note that as the architecture of urban surveillance transforms around us to match the task of beating COVID-19, the familiar layers of visual (in)visibility — whether from scarves, masks, or hoodies — may still work on other people, but not always on the cameras.