Skip to content

0007 — aesthetic & expression cull signals

Status: IMPLEMENTED (2026-07-03) — Layer 1 (pure seam + ranking/verdict integration) and Layer 2a (geometric expression estimators) ship in Phase 2. Layer 2b — the aesthetic model adapter — is deferred to Phase 4 (see the 2026-07-03 decision below); the aesthetic Scorer seam and its ranking/verdict/cache integration ship now but inert (no backend, scorer off). The design for 0006 §8 (Phase 2's ONNX track, its final slice). Companion to 0001-krites.md (§4.2 signal table, §5 providers), 0004-face-eye.md (the machinery this reuses — provider seam, purego/ORT adapter, calibrated anchors, layered build), 0005-decision-capture.md (the learning substrate these signals feed), and 0006-develop.md (R-AES-1..4, the requirements this spec implements).

Decision (2026-07-03, Matt) — Layer 2b deferred to Phase 4. The Q2 search turned up no reliably-hosted, checksummable NIMA-MobileNet .onnx meeting the R-MLR-7 bar: the reference (idealo/image-quality-assessment, Apache-2.0) ships Keras HDF5 weights, not ONNX, and every community ONNX conversion is unhosted/unprovenanced. Weighing the throwaway cost, we asked whether NIMA has value beyond Phase 4 — and it does not: at Phase 4 the CLIP+aesthetic-head swaps in behind the same Scorer seam because the learner needs a per-frame embedding (score and embedding in one pass), which NIMA cannot provide, so NIMA gets deleted, not kept. Its only value was a Phase-⅔ stopgap ranking signal, bought with a throwaway convert→self-host→calibrate. So we defer the aesthetic model to Phase 4 and build CLIP directly as the first real backend (R-ASC-1 makes the seam ready today). Aesthetic capture on Phase-⅔ shoots can be backfilled later — originals are immutable (R-ND-1). This closes slice 7 at Layer 1 + Layer 2a; the aesthetic ranking tier is present and inert until a backend lands. See 0007-DEC-1 below.

Decisions (2026-07-02, Matt). §8 resolved: Q1 — geometric-from-landmarks expression (zero new models; the classifier stays a config-selectable later strategy, R-EXPR-4). Q2 — NIMA-MobileNet as the first aesthetic scorer; the CLIP+aesthetic-head is the recorded Phase-4 upgrade behind the same seam (score + reusable embedding in one pass). Q3 — score every frame at cull (one-time, cached; the learner needs the full distribution, rejects included). Q4 — expression ships verdict-neutral (SmileSoft = FacingSoft = 0): ranking + capture only; demote-to-maybe is opt-in (R-EXPR-5 confirmed).

Scope: the two 0001 §4.2 "Phase ≥2" signals — expression (smile, looking-at-camera) and an aesthetic score — as cull signals in the exact mould of the eye/blink signal: pure projections + verdict/ranking integration, fed by provider-backed analysis, config-gated, off by default, calibrated on real frames, captured for the Phase-4 learner. §8's questions are resolved; nothing gates the build.

1. The shape of the work — and the free lunch

0004 built more than an eye signal: it built a face analysis pass — the UltraFace detector finds boxes, the InsightFace 2d106det model returns 106 facial landmarks in pixel coordinates per face, and a pure, calibrated projection (EAR → EyeOpen) turns geometry into a signal. The landmarks cover the full mouth contour, eye contours, brows, nose and face outline.

The free lunch: expression needs no new model. Smile is mouth geometry; looking-at-camera is head-pose geometry — both are pure projections of the landmarks the eye pass already computes per frame, behind the same calibrated-anchor approach as EAR. One detector+landmark pass feeds eyes and expression (0006 R-AES-1), at zero additional inference cost.

The aesthetic score is the one genuinely new model (0006 R-AES-2): a whole-frame "is this a strong photograph" scorer with no deterministic equivalent. It gets its own provider seam and its own ONNX adapter, reusing the 0004 runtime plumbing (purego/ORT, config-selected EP, model registry + checksum fetch, calibration method).

expression aesthetic
new model? no — rides the 2d106det pass yes — one new ONNX model
pure part mouth/pose geometry → probabilities (like EAR) score → ranking projection
provider existing face.Analyzer (extended Face) new aesthetic.Scorer seam
default off (rides face.enabled) off (aesthetic.enabled)

2. Package layout & the WASM boundary

The 0004 split, extended — pure seams in the engine, native adapters outside the wasm-check set:

pkg/analyze/face/            # PURE (existing) — gains expression projection
  signal.go                  #   + ExpressionSignal: Result → cull signal fields
pkg/face/onnx/               # NATIVE (existing) — gains geometric estimators
  smile.go                   #   mouth geometry → Smile (per face, from landmarks)
  pose.go                    #   landmark symmetry → FacingCamera (per face)
pkg/analyze/aesthetic/       # PURE (new) — Scorer interface + ranking projection
  aesthetic.go               #   Scorer, Score; fake in tests
pkg/aesthetic/onnx/          # NATIVE (new) — the aesthetic model adapter
  onnx.go                    #   reuses the pkg/face/onnx runtime plumbing
  • R-EXPR-1 (MUST) The expression projection lives in pkg/analyze/face (pure, WASM-safe, R-FACE-1); the geometric estimators live in the existing native adapter behind the same face.Analyzer seam (R-FACE-2/3).
  • R-ASC-1 (MUST) pkg/analyze/aesthetic holds only the Scorer interface, its types, and the pure ranking projection; every model-backed scorer lives in its own package behind it, injected, faked in tests — never a package-level hook (the keryx rule).

3. Expression — extending the face pass

3.1 face.Face gains attributes

// Smile is the probability (0..1) that the face is smiling; ExprUnknown (-1)
// when the estimator could not judge (mirrors EyeOpen's sentinel, R-FACE-5).
Smile float64
// FacingCamera is the probability (0..1) that the face is oriented toward the
// camera (a head-pose proxy, not eye-gaze); ExprUnknown when unjudged.
FacingCamera float64
  • R-EXPR-2 (MUST) Both are per-face probabilities in [0,1] with the -1 unknown sentinel, never silently coerced; a backend that only judges eyes (today's) reports unknown and nothing downstream changes (R-FACE-5 mould).

3.2 Geometric estimators (no new model — §9-Q1)

Behind the adapter's existing per-face estimator seam, from the 2d106det landmarks already computed:

  • Smile — mouth geometry: mouth aspect/width ratios and mouth-corner lift relative to the mouth centre (the "MAR" family — the same shape as EAR). Raw geometry maps to Smile through calibrated anchors (closed ↔ broad smile), tuned on real frames exactly as the EAR anchors were (0004 §6, calibration harness).
  • FacingCamera — head-pose proxy from landmark symmetry: the left/right asymmetry of eye-outer/nose/face-outline distances estimates yaw (and brow/chin ratios estimate pitch); large deviation ⇒ low FacingCamera. Honest naming: this is head faces camera, not iris gaze — 2d106det carries no iris points. Good enough for "subject turned away" ranking.
  • R-EXPR-3 (MUST) Both estimators are deterministic geometry over landmarks; the maths is pure and unit-tested with synthetic landmark fixtures (smiling/neutral mouths, frontal/turned poses) independent of any model, and calibrated against real frames via the 0004 integration harness before the knobs ship enabled.
  • R-EXPR-4 (SHOULD) A classifier-based expression strategy (e.g. a FER+-class emotion model on the face crop) remains a config-selectable alternative behind the same attributes (face.onnx.expression: geometric | classifier), mirroring R-MLR-6 — wired later only if geometry proves insufficient on real frames. Not built in this slice.

3.3 Signal projection & profile knobs

cull.Signals gains (worst-face semantics, like eyes — one grimace spoils a group shot; same MinFaceBox floor):

// MinSmile is the worst per-face smile probability among counted faces, or -1.
MinSmile float64
// MinFacing is the worst per-face facing-camera probability, or -1.
MinFacing float64

cull.Profile gains SmileSoft and FacingSoft (0..1 soft floors; 0 = disabled), plus nothing hard:

  • R-EXPR-5 (MUST) Expression never hard-rejects and — unlike eyes — the wedding-default seed ships it verdict-neutral (SmileSoft = 0, FacingSoft = 0): a solemn ceremony face or a turned candid is normal at a wedding, not a defect. Expression's default value is **best-of-burst ranking
  • capture for the Phase-4 learner**; demote-to-maybe is opt-in by raising the soft floors. (The R-EYE-3 philosophy, applied harder.)
  • R-EXPR-6 (MUST) Unknown (-1) never affects verdict or ranking (R-EYE-4 mould); zero-value Signals stays safe via FaceCount.

4. Aesthetic — the one new provider

Status: seam-only in slice 7. Per 0007-DEC-1 the Scorer seam, cull fields, ranking tier and cache slot below ship now (Layer 1) but inert; the model backend is deferred to Phase 4 and built as CLIP (§4.2), not NIMA. The requirements R-ASC-1..4 are the contract that Phase-4 adapter satisfies.

4.1 The Scorer seam (pure)

// Package aesthetic is the provider seam for whole-frame aesthetic scoring.
type Scorer interface {
    // Score rates img's photographic strength in [0,1] (calibrated; higher is
    // stronger). Deterministic per (img, model); honours ctx (R-GLOBAL-8).
    Score(ctx context.Context, img image.Image) (float64, error)
}

cull.Signals gains Aesthetic float64 (-1 = unscored). cull.Profile gains AestheticFloor (0..1; 0 = disabled — below it a keep demotes to maybe, the "maybe-floor" of 0006 R-AES-2).

  • R-ASC-2 (MUST) Aesthetic is a soft ranking signal: it feeds best-of-burst and the optional maybe-floor, and never hard-rejects (0006 R-AES-2). Seed: AestheticFloor = 0 (ranking-only).
  • R-ASC-3 (MUST) Scoring runs on the cull-time image (the embedded preview, R-FACE-9 mould), is config-gated off by default (aesthetic.enabled, R-PRIV-1 posture), degrades gracefully per frame on error/timeout (R-FACE-8 mould), and is cached in the analysis cache keyed by model id (R-FACE-10, R-MLR-5; bump shoot.AnalysisVersion).

4.2 Model candidates (§9-Q2)

All CPU-EP-runnable via the existing purego/ORT runtime; registry + checksum fetch per R-MLR-4:

candidate size CPU cost/frame notes
NIMA (MobileNet) — AVA-trained mean-opinion scorer ~13 MB ~50 ms the reference "aesthetic" model; small, fast, well-understood; output = 1–10 distribution → mean → calibrated to [0,1]
LAION aesthetic head on CLIP ViT-B/32 ~350 MB ~200–400 ms modern + robust; the CLIP embedding is a reusable feature vector for the Phase-4 learner (score + embedding from one pass) — the strategic option
MUSIQ / transformer IQA large slow best quality; ONNX availability/opset spotty — not a first pick

Recommendation: NIMA-MobileNet first — smallest surface, fastest across 4,000 frames (~3–4 min), proven; behind Scorer it is swappable. Record the CLIP+aesthetic-head as the planned Phase-4 upgrade: when the learner needs per-frame embeddings (0005), one CLIP pass yields both, and the scorer swap is a config flip + adapter (R-ASC-1).

  • R-ASC-4 (MUST) The raw model output is mapped to [0,1] through calibrated anchors against real frames (the 0004 method): verify score separation between Hailey's keeps and rejects before the signal ships enabled; anchors live in config, not Go constants.

5. Ranking & verdict integration (pure, Layer 1)

Best-of-burst extends its deterministic preference chain (R-EYE-6):

  • R-RANK-1 (MUST) eyes-open ▸ expression ▸ aesthetic ▸ sharpness — an open-eyed frame beats a blink; among open-eyed frames the better worst-face smile wins (when known); then the higher aesthetic (when known); then sharpness. Unknown values skip their tier (today's behaviour unchanged when the providers are off, R-FACE-7 mould). Golden unit tests pin every tier.
  • R-RANK-2 (MUST) Demotion reasons stay explainable: a burst loser names the tier that decided it ("kept the smiling frame", "kept the stronger frame") — and both signals land in verdicts.yaml reasons only when they acted (0004 §8.1 polish mould).
  • R-CAP-1 (MUST) All four new signal fields persist in the analysis cache and therefore in the decision-capture substrate (0005 R-CAP-3) — the learner sees smile/facing/aesthetic alongside sharpness/eyes from day one, even while verdict-neutral. Never written to XMP (0006 R-AES-4).

6. Implementation plan (the 0004 layering)

Layer 1 — pure, no models, ships on Linux now (ungated): 1. face.Face attributes + ExpressionSignal projection + fakes (R-EXPR-1/2/6). 2. pkg/analyze/aesthetic seam + fake (R-ASC-1). 3. cull.Signals/Profile fields, verdict semantics, ranking chain + golden tests (R-EXPR-5, R-ASC-2, R-RANK-1/2); AnalysisVersion bump; pipeline wiring (nil-safe, additive, R-FACE-7/8 mould).

Layer 2a — geometric expression (no new model; gated by §9-Q1): 4. smile.go/pose.go estimators in pkg/face/onnx from the existing landmark pass; synthetic-fixture unit tests; INT_TEST calibration harness on real frames → seed anchors (R-EXPR-3).

Layer 2b — the aesthetic adapter — DEFERRED to Phase 4 (0007-DEC-1): The aesthetic Scorer seam, cull fields, ranking tier, verdict semantics and analysis-cache slot ship now (Layer 1) but inert — no backend, scorer off. The model adapter itself is deferred and, when built, is CLIP not NIMA: 5. NIMA registry entry + pkg/aesthetic/onnx → Phase 4: pkg/aesthetic/onnx (or …/clip) on the shared runtime, aesthetic.* config, INT_TEST accuracy + calibration (R-ASC-3/4) — built as the CLIP+aesthetic-head so the Phase-4 learner gets score and embedding from one pass. 6. Studio aesthetic toggles/knobs → Phase 4 with the backend (the expression toggles ride the existing providers panel now). 7. Aesthetic-tiebreak BDD scenario → Phase 4 with the backend. The expression burst scenarios (smile beats neutral; providers-off unchanged) ship with Layer 2a.

0007-DEC-1 — aesthetic model deferred to Phase 4 (2026-07-03, Matt)

NIMA has no value beyond Phase 4 (CLIP supersedes it wholesale — score and the embedding NIMA lacks), and no clean checksummable NIMA ONNX exists to pin (R-MLR-7). Building it now is throwaway. Decision: defer the aesthetic model adapter to Phase 4 and build it as CLIP directly. Everything with lasting value — the pure Scorer seam, the nil-safe ranking/verdict/cache integration, and the free-lunch expression estimators — ships in slice 7. The aesthetic ranking tier is wired and inert (aesthetic.enabled has no backend to enable yet); R-ASC-1/2/3/4 remain the contract the Phase-4 CLIP adapter satisfies.

Layer 3 — deferred: classifier expression strategy (R-EXPR-4); CLIP scorer swap (Phase 4); CoreML EP on the M-series (0004 Layer 3, shared).

7. Non-goals

  • Emotion taxonomy (happy/sad/angry…) — krites needs "strong frame of people at their best", not sentiment analysis.
  • Iris-gaze tracking — FacingCamera is a head-pose proxy; true gaze needs an iris model and isn't worth the surface yet.
  • Aesthetic as a hard gate — forbidden by design (0006 R-AES-2); a technically-imperfect but emotionally-strong frame is the photographer's call.
  • The Phase-4 learner itself — these signals only feed it (0005).

8. Open questions — all resolved (2026-07-02, Matt)

  • Q1 — expression approach. ✅ Geometric-from-landmarks (§3.2) — zero new models, pure, calibratable; the FER+-class classifier stays a config-selectable later strategy (R-EXPR-4). Gates Layer 2a.
  • Q2 — aesthetic model. ⚠️ Superseded by 0007-DEC-1 (2026-07-03). The "NIMA-MobileNet now" answer was reversed once the R-MLR-7 licence/hosting check found no clean NIMA ONNX and NIMA proved valueless beyond Phase 4: the model adapter is deferred to Phase 4 and built as CLIP directly. The seam ships now, inert. (Original reasoning retained in §4.2 for the CLIP rationale.)
  • Q3 — scoring scope. ✅ Every frame at cull — one-time (~3–4 min over 4,000 with NIMA), cached; Phase 4 needs the full distribution, rejects included.
  • Q4 — expression default posture. ✅ Verdict-neutral (R-EXPR-5 confirmed) — ranking + capture only; solemn ≠ defect at a wedding; any stronger posture is earned from Hailey's own overrides (Phase 4) or opted into via the soft floors.