Not Perfect, but Not Bad

But face_recognition can do more than find faces in pictures; it can identify the person to whom the face belongs. To compare two faces found in different images, the algorithm again cannot simply match raw images pixel by pixel. Rather, it has to normalize, equalize, and then extract a series of features.

A person might pay attention to the size of the nose, the color of the eyes, the forehead height, or the thickness of the eyebrows. The facial recognition algorithm, on the other hand, learns which features produce the most hits and the least false positives with millions of test images in the learning phase, based on matching and mismatching images. Afterward, however, the algorithm only consists of meaningless columns of numbers. As is usual in machine learning, no one knows which particular feature the algorithm uses to arrive at a particular decision.

Figure 5 shows the key data of the reference face extracted from Figure 1 provided by the face_encodings() function. The facial comparison algorithm in turn takes the key data from each recognized face and compares them to the reference. If two records approximately match, it is probably the same person.

Figure 5: The key data of the author's face from Figure 1.

With this tool, a script can extract a face from the reference image and compare the result with faces on other images. As a practical application, I have whipped up the script in Listing 3, which searches my own photo collection (containing an impressive 36,525 images) for images showing the person on the reference image – me.

The file hierarchy is based on what I call the shoe box principle, meaning new photos just get dumped into there without any extra archiving or indexing. Whittling down the collection by hand would be very labor intensive. But I can certainly show an artificial intelligence (AI) system the photo from Figure 1 and rattle through the image collection to see if the face in Figure 1 can also be detected on other photos.

Unpacking the Whole Collection

To do this, Listing 2 first defines an iterator for all JPEG photos below the /photos directory on my hard disk. It skips other formats and all entries in .cache directories where one of my image processing programs stores the thumbnails that I want to leave out of the face analysis action. The photos() iterator as of line 5 accepts the start directory and then runs through all the files it finds; the yield() operator returns them in line 12 bit by bit, when the main program asks for more.

01 #!/usr/bin/python3
02 import os
03 import re
04
05 def photos(dir):
06   for root, dirs, files in os.walk(dir):
07     if re.search(r'\.cache', root):
08         continue
09     for file in files:
10       if re.search(r'jpg$', file,
11                    re.IGNORECASE):
12         yield(os.path.join(root, file))
13
14   # testing
15 if __name__ == "__main__":
16     for photo in photos("/photos"):
17         print(photo)

01 #!/usr/bin/python3
02 import face_recognition as fr
03 import dbm
04 import re
05 from photos import photos
06 import sys
07
08 try:
09   _, ref_img_name, search_path = sys.argv
10 except ValueError as e:
11   raise SystemExit("usage: " +
12     sys.argv[0] + " ref_img search_path")
13
14 cache = dbm.open('cache', 'c')
15
16 ref_img  = fr.load_image_file(ref_img_name)
17 ref_face = fr.face_encodings(ref_img)[0]
18
19 for photo in photos(search_path):
20   if photo in cache:
21     print(photo + " already seen")
22     continue
23   cache[photo] = "1"
24
25   try:
26     img = fr.load_image_file(photo)
27   except:
28     continue
29
30   for face in fr.face_encodings(img):
31     hits = \
32       fr.compare_faces([ref_face], face)
33     if any(hit for hit in hits):
34       print(photo)

In the main image finder in Listing 3, lines 8 to 12 check whether the user has specified both a reference image and the top search path for the photos at the command line. The first element of sys.argv contains the script name, which gets discarded in the underscore variable (_); then, line 16 loads the reference image. The next line extracts the reference face, shown under index   of the returned coordinate list because the reference image only has one face in it.

Later, line 19 runs through all JPEG files found by photos.py; for each one, the script calls the compare_faces() function from the face_recognition project with the face values previously obtained from the reference image. The

any(hit for hit in hits)

construct checks whether one of the faces detected on the current image matches the one on the reference image. If this happens, one of the elements in the hits list has a value of True, and line 34 prints the image file's path to standard output, where the surprised user can pick it up and inspect it with a photo viewer.

Listing 4 shows how the script calls into the Docker container and displays its output. I was amazed by the photos it dug up, some from ancient history, unveiling a more youthful edition of yours truly. Oh, the good old days!

1 $ docker run -v /photos:/photos -v `pwd`:/build -it face bash -c "cd /build; python3 face-search.py me.jpg /photos"
2 /photos/2001/12/29/13:55:38.jpg
3 /photos/2001/07/22/11:47:27.jpg
4 /photos/2001/07/22/10:35:33.jpg
5 /photos/2001/07/22/15:43:23.jpg
6 [...]

So Many Nerds

However, the process is not perfect and occasionally makes downright laughable mistakes. For example, my collection had some pictures I took at open source conferences showing hundreds of young nerds, and the algorithm totally thought that one of them was me, which is impossible because I actually took the photos.

Since AI wastes lots of computing time, rummaging through the image tree and recognizing familiar faces takes a long time. If the script bombs out somewhere due to an error, it would be unfortunate to have to start over. Therefore, Listing 3 remembers results from all processed images in a persistent file named cache. Python closes it conveniently when the program terminates, so the script only has to open it at the beginning with the c flag (to create it for the first time if necessary). The script can subsequently access the cache dictionary to see whether it already contains the name of the file under investigation and skip it if so.