Who: On-device Audio-Visual Speaker Diarization
On-device audio-visual speaker diarization for talking-head video: per-turn speaker IDs, face-track anchors, and millisecond timestamps.
Who is on-device audio-visual speaker diarization for talking-head video. One call returns Whisper-compatible turns with per-segment speaker IDs, face-track anchors, normalized face bounding boxes, and millisecond timestamps, suitable for cutting, captioning, and follow-the-speaker camera switching. Who is video-first, combining audio and face tracking rather than diarizing pure audio.
Use cases
Follow-the-speaker editing
Cut or switch camera on speaker change with face context, for talking-head and multi-cam video.
Captioning
Attach speaker labels to Whisper-compatible transcript turns for captions and subtitles.
What it does
- Per-turn: speaker id, face track id, normalized face bbox, millisecond timestamps.
- Whisper-compatible turns for captioning pipelines.
- Video-first audio-visual, not pure audio.
Specs
- Accuracy
- 9.87% DER on VoxConverse-dev (collar 0.25), vs 13.97% for the Silero+CAM++ baseline
- Size
- ~17 MB weights
- Platform
- Apple (Swift) first
FAQ
What is Who?
On-device audio-visual speaker diarization for talking-head video: per-turn speaker IDs, face-track anchors, and millisecond timestamps.
Does Who run on device?
Yes. Who runs entirely on device: inference happens locally with no server call, so data never leaves the device.
Is Who available yet?
Who is in closed beta. You can request early access from its page.
How much does Who cost?
Every model is free up to 100k monthly active devices per SDK. Unlimited inference per user. Contact us for custom licenses.
Early access
Tell us what you are building and we will get you set up.