Forbes posted an article London Police Facial Recognition ‘Fails 80% Of The Time And Must Stop Now’. The question is, are facial recognition systems so bad in reality? To estimate it quantitively let’s rephrase the question to the following one: what is the probability that an individual identified as a criminal is a real offender?

First, let’s write a simple function in python to estimate it:

Now we are ready to answer the question. We can assume that the sensitivity of a given facial recognition system is 0.99
with a specificity of 0.99. Therefore, we have true positive results in 99 trials out of 100, and we correctly identify
false-positive 99 times out of 100. According to
the Annual Abstract of Statistics (National Office of Statistics. 2009.
Retrieved 23 January 2009), the prison population was about 149 people per 100,000 in 2007. From that, the
probability of criminal might be estimated as `149/100_000`

. With these numbers we have:

That is, a 12.8% probability to correctly identify a criminal. And to adjust probability to the claimed one in the Forbes article, let’s set sensitivity equal to 99.5% and specificity to 99.5%:

That is, a 22.9% probability to correctly identify a real criminal, almost equal to what Forbes reported in the article. Given this, are there any ways to improve performance? One of the options might be a sequential identification with different conditions:

Identifications in sequence | Probability of error |
---|---|

1 | (1-0.229)**1 = 0.771 = 77.1% |

3 | (1-0.229)**3 = 0.458 = 45.8% |

5 | (1-0.229)**5 = 0.272 = 27.2% |

7 | (1-0.229)**5 = 0.161 = 16.1% |

To summarize this short note, remember to check business requirements to estimate the performance of systems and understand their limitations in real applications. Properties of a system might be outstanding, but it can produce miserable results in wrong conditions.