One photo, three face shape tools, three answers: how an on-device classifier works

I run The Face Report, a site with a face shape detector that works in the browser. This month I put one portrait through it and through two other free face shape tools. As far as I can tell, all three use the same 478-point MediaPipe face mesh. They gave three answers: Heart at 88%, Round at 72%, and Oval, borderline with Diamond. The landmark model does not decide a face shape. It gives you points. Someone has to write the rule that turns those points into a label, and that rule is where the tools differ. This post shows how ours works, with code you can run, and what I got wrong. Get the landmarks import { FaceLandmarker, FilesetResolver } from "@mediapipe/tasks-vision"; const files = await FilesetResolver.forVisionTasks( "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.32/wasm" ); const landmarker = await FaceLandmarker.createFromOptions(files, { baseOptions: { modelAssetPath: "https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task", delegate: "GPU", }, runningMode: "IMAGE", numFaces: 1, }); const mesh = landmarker.detect(imageElement).faceLandmarks[0]; mesh holds 478 points. Their coordinates are normalized to the image width and height separately, so convert them to pixels before you measure anything. On a portrait photo, distances taken from the raw values are wrong. Turn eight points into four numbers Measure Landmarks Face height 10 (top of the mesh) to 152 (chin) Forehead width 54 to 284 Cheekbone width 234 to 454 Jaw width 172 to 397 const px = (i) => ({ x: mesh[i].x * width, y: mesh[i].y * height }); const dist = (a, b) => Math.hypot(a.x - b.x, a.y - b.y); // Angle at vertex v between points a and b, in degrees. function angleAt(v, a, b) { const [ax, ay, bx, by] = [a.x - v.x, a.y - v.y, b.x - v.x, b.y - v.y]; const cos = (ax * bx + ay * by) / (Math.hypot(ax, ay) * Math.hypot(bx, by)); return Math.acos(Math.max(-1, Math.min(1, cos))) * (180 / Math.PI); } const [top, chin] = [px(10), px(152)]; const [foreheadL, foreheadR] = [px(54), px(284)]; const [cheekL, cheekR] = [px(234), px(454)]; const [jawL, jawR] = [px(172), px(397)]; const cheekWidth = dist(cheekL, cheekR); const ratios = { heightToWidth: dist(top, chin) / cheekWidth, foreheadToFace: dist(foreheadL, foreheadR) / cheekWidth, cheekToJaw: cheekWidth / dist(jawL, jawR), jawAngle: (angleAt(jawL, chin, cheekL) + angleAt(jawR, chin, cheekR)) / 2, }; There is no forehead-to-jaw ratio. It equals foreheadToFace * cheekToJaw, so adding it would count the same geometry twice. Find the nearest profile Each shape is a set of ideal values for those four numbers. The classifier measures how far a face is from each set, in units of tolerance, and picks the nearest. const WEIGHT = { heightToWidth: 5 / 17, foreheadToFace: 4 / 17, cheekToJaw: 5 / 17, jawAngle: 3 / 17 }; const TOLERANCE = { heightToWidth: 0.1, foreheadToFace: 0.07, cheekToJaw: 0.1, jawAngle: 10 }; const PROFILES = { oval: { heightToWidth: 1.24, foreheadToFace: 0.88, cheekToJaw: 1.26, jawAngle: 137 }, round: { heightToWidth: 1.13, foreheadToFace: 0.88, cheekToJaw: 1.2, jawAngle: 141 }, square: { heightToWidth: 1.16, foreheadToFace: 0.9, cheekToJaw: 1.2, jawAngle: 130 }, heart: { heightToWidth: 1.24, foreheadToFace: 0.92, cheekToJaw: 1.37, jawAngle: 134 }, oblong: { heightToWidth: 1.34, foreheadToFace: 0.88, cheekToJaw: 1.24, jawAngle: 137 }, diamond: { heightToWidth: 1.24, foreheadToFace: 0.84, cheekToJaw: 1.31, jawAngle: 134 }, }; function classify(ratios) { const ranked = Object.entries(PROFILES) .map(([shape, ideal]) => { let sum = 0; for (const key of Object.keys(WEIGHT)) { sum += WEIGHT[key] * ((ratios[key] - ideal[key]) / TOLERANCE[key]) ** 2; } return { shape, distance: Math.sqrt(sum) }; }) .sort((a, b) => a.distance - b.distance); const [best, next] = ranked; return { shape: best.shape, runnerUp: next.shape, matchStrength: Math.round(100 / (1 + best.distance)), borderline: next.distance - best.distance < 0.25, }; } classify({ heightToWidth: 1.22, foreheadToFace: 0.88, cheekToJaw: 100 / 78, jawAngle: 135 }); // { shape: 'oval', runnerUp: 'diamond', matchStrength: 85, borderline: true } That call uses the portrait from the start of this post: cheekbones 100, forehead 88, jaw 78, height 1.22 times the width. matchStrength is not a probability. It is 100 for an exact match with a profile and falls as the distance grows. I would read a percentage on any tool like this the same way unless the tool says how it was measured. What I got wrong I started from textbook ratios. Textbooks measure from the hairline, and the top of the mesh sits below it, so height reads short. The textbook values sat outside the 5th to 95th percentile of real scans on most measures. Square had an ideal jaw angle of 100 degrees when the 5th percentile of real scans was 128. Most faces came out oval by default. I re-centred the profiles on 2,217 production scans and kept each shape's position relative to the others. That is calibration to a distribution, not an accuracy benchmark. I have no labelled dataset, so I cannot tell you how often the label is right. Oval is still the largest group: 49.2% of 15,429 scored scans on one day in August. The first score was clipped. Each ratio scored inside a band and stopped changing outside it, so a very long face could lose the information that it was long and land nearer round than oblong. Unclipped distance keeps growing, so the direction survives. Tolerances have to match across shapes. If one profile has a wider tolerance on a measure than the others, it wins at extreme values by default. Each measure here has one tolerance shared by all six shapes. A single label hides close calls. When the runner-up is within 0.25 distance units, the result says so and names both shapes. The portrait above is one of those. The failure that did not throw On Android, in about 1% of scans, the GPU delegate's texture upload failed silently. The model ran on a blank frame and returned one off-scale mesh, usually with coordinates in the hundreds, and the same pose on every photo. No exception was raised, so the app measured it and reported a result. One person scored 9 out of 100 from a mesh that was not on their face. The fix is a range check on every point and one retry on the CPU delegate: const plausible = (points) => points.every((p) => [p.x, p.y].every((v) => Number.isFinite(v) && v >= -2 && v