How I Organize Photos by Face with Local AI

Have you ever tried to find every photo of one person? Your dad at every birthday, or your kid at every age? A search by name only works if something has put names on the faces first.

I wanted that for my own photos. But my photos already sit in iCloud, and I didn’t want to hand my family’s faces to an AI company too. So I organize my photos by face offline, on my Mac, as part of the local AI photo library I built in October. Everything had to run locally, which limited me to models that were free, ran offline, and worked on my machine.

It now knows about 120 people across roughly 1,300 faces I named myself. Getting there had one big surprise: the first face model I picked scored 98.7% on a well-known public test, then couldn’t recognize people I had named many times.

Short answer: Yes, one Mac can name your family’s faces with nothing leaving it. What made it work was checking every model against faces I had named myself: that check took me from 79% to 91% of them recognized, with about 1 wrong name in 100. Here are the specifics: how it works, the models I tested, why I switched, how strict the matching is, and how I name faces in batches.

How offline face recognition works

Face recognition uses two models, programs that learned a skill from a huge pile of examples, plus a person who knows the names:

Compare two fingerprints and you get a similarity score between 0 and 1. Close to 1 means probably the same person. Close to 0 means strangers. The match threshold is the score where “same person” begins. Set it too low and strangers get names. Set it too high and people you know go unrecognized.

Choosing models on public photos

I narrowed the choice on public photo sets first. One was LFW (Labeled Faces in the Wild), a free collection of photos of named people that researchers use to test face models. I added cats and dogs too, because our cat, Chandi is in a lot of my photos.

To compare models fairly, I set each one strict enough that only 1 pair of strangers in 100 slipped through as “the same.”

“Matched” counts real same-person or same-pet pairs that passed the 1-in-100 rule. Pet figures come from the Oxford pets photo set. The pet finder found no pets in 300 photos of people.

YuNet¹ is a tiny, fast face finder. SFace² came from the same free OpenCV collection. The pet fingerprint came from a SigLIP2 animal identification model,³ trained to tell individual cats and dogs apart.

On a Mac, you might ask why I didn’t just use what Apple already includes.

Why I replaced SFace with AdaFace

LFW’s pictures show one face, centered in the frame. Family photos don’t. They’re full of group shots, small faces, and turned heads. When I measured SFace on my roughly 1,300 named faces, it couldn’t recognize the same person from the side, in a blurry shot, or years apart.

Compared one pair at a time, two pictures of the same person usually scored about 0.40. Two different people scored as high as 0.36. That’s too close for any threshold to separate. People I had named many times were never recognized again.

So I tested AdaFace,⁶ a face model built for poor-quality faces. I shrank and blurred LFW’s faces and ran both models under the same 1-in-100 rule:

On clear faces, the two were nearly tied. The gap opened on exactly the kind of faces family photos are full of.

Then I compared them on my own named faces, both allowed the same rate of wrong matches. The test reported only counts, never photos, the same way I kept my AI coding assistant away from the photos.

These are the same faces I later used to set the threshold. So they compare the two models; they don’t predict how either will do on next year’s photos.

How strict the matching is

I got the threshold wrong the first time. With SFace, the naming page suggested a name whenever a face scored 0.40 against someone, the typical one-pair score above. About half the faces nobody had named, plus some blurry things that weren’t faces, showed up as possibly someone.

The mistake: a new face isn’t compared with one picture. It’s compared with hundreds of named faces, and in a crowd that big, someone usually looks a bit closer than a typical pair does. The closest wrong person usually scored 0.41 and at most 0.55, so I raised SFace’s threshold to 0.55.

With AdaFace, I changed the rule itself. A new face is scored against the 3 named pictures of each person it looks most like, averaged, so one look-alike picture can’t pull a stranger into someone’s name. The scores shift with the model, so the thresholds moved too:

Automatic naming also requires the best person to beat the runner-up clearly, so two similar people never get a coin flip.

I tested this on 1,214 faces I had already named, hiding each photo’s own faces so they couldn’t match themselves. At 0.39, 1,107 got the right name, 12 got the wrong name, and 95 stayed unnamed. Down to 0.24, the guesses were right 1,157 times and wrong 24 times, and a wrong guess is easy to fix on the naming page. I picked both thresholds using these same faces, so the results describe them, not new photos.

Naming faces in batches

The first versions of the naming page asked about everything. At one point it showed over 3,000 groups to click through. I changed it nine times in one evening.

The version that stuck shows 48 faces a screen, with look-alikes side by side. Likely matches come first, labeled with the guess. Then come unnamed faces that keep turning up: at least 5 pictures from 3 different photos or videos. I select a batch and answer once:

1. A name. The faces join that person’s examples.
2. Stranger. A real face I don’t need to name.
3. Not a face. A blur or a pattern the finder got wrong.
4. Skip. Decide later.

Each picture is decided once and never asked about again. A face seen once or twice that looks like nobody (a guest, someone in the background) gets no question and stays unnamed. Before that rule, about 500 of the faces on the page were those one-offs.

Pets got the same treatment at first.

Each night, new faces above the threshold get a name. The rest wait until I open the page.

What I’d do from the start

If you try this yourself, use public photos to narrow the choice. Then measure again on a few hundred faces you’ve named, because that’s where my top-scoring model fell apart.

For technical readers, here are more details

The public tests ran on October 12, 2026. “1 in 100” means the threshold where 1% of stranger pairs were accepted. LFW faces are centered, so group photos and small or turned faces weren’t covered by that test.

The pet finder was RT-DETRv2⁷ (the small r18 version, trained on the COCO photo set). Pet results used 300 cat and 300 dog photos from Oxford-IIIT Pets;⁸ a find counted at a confidence of 0.5 or more.

AdaFace is the IR-101 network trained on WebFace12M, using the CVLface weights⁹ and the network code from CVLface.¹⁰ Each fingerprint is 512 numbers, and two are compared by multiplying them number by number and adding up (cosine similarity). Fingerprinting 7,885 library faces again took 656 seconds. Faces are aligned to five landmark points before fingerprinting. The blurry LFW test used LFW’s 6,000 standard pairs.

The real-name test used 1,304 faces of 120 people, split into five groups by file; the 1,214 with a match in another group were scored. Each face was matched against the other four groups, never against its own file or anything taken within 2 minutes of it, so bursts couldn’t match themselves. Automatic naming needs the best person at least 0.05 ahead of the next one. Of 3,525 faces of other people in a photo with someone named, 5 were taken for that person at 0.39 and 19 at 0.24. With SFace, the automatic rule used the single closest picture at 0.55, and a closest-3 guess line at 0.45 got 1,030 right and 22 wrong of 1,203.

Pets are linked automatically only at 0.94, using the single closest named picture. Within one video, a face joins an earlier sighting at 0.35 or more, above any two LFW strangers (0.28), so a person counts once per video, not once per frame. The smallest face kept is 40 pixels on a side, a guess that hasn’t been measured.

This is custom code for my own library, not a package you can install. The method carries over; the exact thresholds depend on the model and your photos, so measure your own.