detecting-tips-zones

Detecting TIPS Zones

Zero-shot zone detection: text prompts → patch-grid cosine heatmaps → bboxes. Companion to svg-portrait-mode — replaces manual focus_targets / focus_edges annotation with a TIPSv2 B/14 forward pass.

Quick Start

from tips_zones import detect_zones
from portrait_mode import portrait_mode

focus_targets, focus_edges = detect_zones(
    "photo.jpg",
    targets=["dog face"],
    edges=["dog paws", "dog ears", "dog body"],
    distractors=["wooden floor", "carpet rug", "shoes", "wall"],
    ckpt_dir="/path/to/tips/checkpoints",
    tips_root="/path/to/tips",
)

svg, stats = portrait_mode(
    "photo.jpg",
    focus_targets=focus_targets,
    focus_edges=focus_edges,
    style_transforms={"background": "desaturate:0.7"},
)

Amortise model load across multiple images:

from tips_zones import load_models, detect_zones

models = load_models(ckpt_dir, tips_root, device="cpu")
for img in images:
    ft, fe = detect_zones(img, targets=[...], edges=[...], distractors=[...],
                          ckpt_dir=ckpt_dir, tips_root=tips_root, models=models)
    ...

How It Works

image → B/14 vision encoder (MaskCLIP values trick on last block)
     → (32×32 patch grid at 448, or 64×64 at 896) × 768-d patch features
text labels → prompt ensemble (9 TCL templates) → B/14 text encoder
     → per-label mean feature → L2-normalise
per-label heatmap = cos(patch feature, label feature)  # raw, no softmax
bbox = top-k% patches → largest connected component → scaled + padded to image coords

Why no softmax over labels

Naïve softmax assumes labels are mutually exclusive. dog face, dog ears, and dog body are all true of the same pixels, so softmax collapses to near-uniform and every heatmap covers the whole subject. Raw cosines + per-label top-k threshold works much better — at the cost of requiring distractor labels to anchor the relative scale. Always pass some distractors (floor, wall, props — whatever is in the scene but not the subject).