Higgs Realtime Duplex: Face to Face, Working in the Background

The Boson AI Team
November 5, 2026

Today we’re introducing Higgs Realtime Duplex, the next generation of Higgs Realtime and Boson’s first full-duplex interactive omni model. Like a person, it listens, watches, and talks at the same time, and it has a face of its own. While it does all of that, it also thinks, calls tools, and works on things in the background. It decides for itself when to speak and recognizes when a remark is not addressed to it.

• Full duplex. It listens, watches, talks, and renders its face at the same time, in 160 ms chunks.
• Works in the background. It thinks, calls tools, and delegates to a bigger model without pausing the conversation.
• 45B MoE front model, trained on about 2 trillion tokens of text, audio, and video.

What a good interaction model needs

We think a model people talk to every day has to get three things right at once.

Feels human. In human conversation, a response typically lands within 200 ms of the other speaker finishing. Sometimes, people respond even before the other speaker finishes, when the timing is right. People also decide when to speak up: they say “mm-hm,” they cut in, they help out when the other person gets stuck, and they stay quiet when a remark was not for them.

Smart. Most conversations with an agent are about getting something done. That takes emotional intelligence, to read the person and the situation, and real reasoning, to break a vague request into tasks and carry them out.

Costs little. We believe talking will be the main way people work with agents, for hours every day. To make that possible, our design target is that an hour with the model costs at most one-tenth as much as an hour of a person’s time.

Capabilities

The model sees you and shows its own face while it talks. Full duplex gives it control over when to speak, when to listen, and how to keep thinking and working as the conversation continues.

Seeing and showing its face

Watches your camera or shared screen the whole time, and answers with a face of its own, rendered in real time in sync with its voice.

When to speak or stop

Says “mm-hm,” answers before you finish, and stops when you cut in, while ignoring background noise and remarks meant for someone else.

Thinking and working

Starts reasoning while you are still talking, and runs tools and delegated tasks asynchronously, keeping the conversation going until the results come back.

The original post includes a video of Higgs Realtime Duplex in a team meeting: it presents an architecture, takes questions mid-sentence, reads the whiteboard, and updates a diagram as the talk goes on. Other clips show more scenarios.

Spoken-English tutor. Times a picture-description exercise, catches “west” for “vest,” watches your mouth as you practice, and paces the drill like a real coach.

At the drive-thru. Asked to check the markets and a training run, it stays quiet while the driver orders with the cashier, then reports back once the order is done. At the pickup window it confirms the driver never ordered animal-style fries: it heard every word, and knew which ones were for it.

Hands-free engineering. The engineer reads a book the whole time. By voice alone, it diffs two engine revisions, asks Claude Code to build a missing mass tool in the background, runs the compliance check and applies the fixes while that tool is built, emails the designer when told to, and reports the new mass once the tool lands.

Our Approach

One model, many threads

Higgs Realtime Duplex runs many threads at once. Your voice and video stream in while the model’s voice and face stream out. At the same time it can think, call a tool, or hand a question to a bigger model, and none of that stops it from listening and watching.

You and the front model—the part of Higgs Realtime Duplex that talks with you directly—exchange audio and video continuously, in real time. The front model thinks on its own and makes async calls straight to the backend model and to tools; their results come back while it keeps talking and listening.

Slice time, not turns

We cut every stream into 160 ms chunks: your voice and camera feed, tool results and backend replies coming in; the model’s voice, face, and background work going out. Each step, the model reads the chunk that just arrived and writes the chunk that plays next. Silent intervals are represented as chunks too. With video enabled, frames keep streaming in even during silence, so the model never stops listening and watching.

In a sample conversation, each cell represents 160 ms. The model says “mm-hm,” thinks, and calls a tool while you talk. A tool result arrives as you finish; the model replies, then pauses its thinking. When you cut in, it stops, re-plans, and delegates, then resumes talking before the backend returns. Its face renders on every step.

Thinking and tool-call generation can span multiple chunks; concatenating those chunks recovers the full output stream. Each arriving tool result enters its input chunk as a payload, without a fixed per-chunk text-token quota. Some tools return intermediate results in stages as they work. The model doesn’t wait for a full result before it starts to speak.

Eight steps up close: at each step, the model reads an input chunk and generates the next output chunk. Thinking and tool-call generation span several chunks. The tool result arrives in parts over three input chunks, and the model starts replying while continuing to think.

Improving intelligence: thinking, tool calls, and delegation

We want the front model to be as smart as it can be on its own, and to reach for help only when it has to. It has three ways to raise the quality of an answer:

Think. Thinking usually takes seconds, and nobody wants to wait seconds for every answer. As each meaningful phrase or idea arrives, the model can build on an ongoing reasoning thread without waiting for the full question. That reasoning continues across chunks, alongside both listening and speaking, so it can keep thinking even as it talks.

Call tools. Search, knowledge bases, booking, CRM lookups, or any function a deployment defines. Calls go out as early as possible, asynchronously, and the model starts answering from partial results. The front model keeps listening, thinking, and speaking while tools work.

When a task is beyond the front model, it hands it to a much bigger backend model, along with the full shared context. While the backend works, the front model keeps the conversation going, including on other topics, and brings the result back when it is ready. This works best when the backend is a frontier model.

This is where we differ from Thinking Machines’ interaction models, whose front model leaves thinking and tool calls to a background model. Ours thinks and calls tools itself. A system that delegates all reasoning and tool use to a larger model is simpler to train, but a large reasoning model may take ten seconds or more to respond, and calling it every turn makes cost hard to control. So the front model handles most requests on its own and saves the backend for the turns that need it.

Reducing cost: a tight token budget

In full duplex the model processes every chunk, even when nobody is talking, so the context fills fast. We squeeze every stream: one audio token per chunk in, two out, a camera frame only every six chunks, and two latents for the face.

Audio in runs at 6.25 tokens per second and audio out at 12.5. A camera frame, about a dozen tokens, comes once every six chunks.

The audio and visual streams use about 2.6K tokens per minute. With optional thinking, tool calls, and their results, the total is roughly 2.6–5K tokens per minute, depending on the workload. That estimate already assumes progressive compaction of intermediate reasoning and tool traces, with the information needed for ongoing work carried forward. In production, the model usually retains the last five minutes of audio and visual streams alongside this compacted text—roughly 13K–25K tokens for this recent window, depending on activity and compaction. Older history is retained as text in addition to this budget.

Zero-shot voice and face

We provide default voices and faces, but they are not baked into the model. They are loaded zero-shot at deployment time: a short reference audio clip sets the voice, and a single reference image sets the face, with no fine-tuning. The references go into the two decoders that render the model’s output, not into the model itself.

One stream, two renderers. Each output chunk carries two audio frames and two image latents. A causal flow-matching decoder turns the audio frames into speech, conditioned on a reference clip; a causal DiT turns the latents into video, conditioned on a reference image and the previous two latents. Swapping the references changes the voice and the face, zero-shot.

Audio decoder. A causal flow-matching model. The reference clip is encoded into a conditioning vector for the decoder, and the speech it generates matches that voice.

Video decoder. Video needs more detail, so it uses a causal DiT, a diffusion decoder. Each step it generates from the reference image and the two most recent latents, so the face stays visually consistent and continuous for as long as the conversation runs.

Model and training

A 45B mixture-of-experts model. The front model is a 45B mixture-of-experts (MoE) model, neither small nor huge. We tried larger models, and they do score higher, but they cost more to run. Since the hardest questions can go to a bigger backend model anyway, a mid-sized front model with solid intelligence gives the best balance.

Trained on 2 trillion tokens. We start from a model pre-trained on text and continue training it on about 2 trillion tokens of text, audio, and visual data. Text training helps preserve the general intelligence the model learned from language. Training on audio and visual data teaches the model to process both modalities directly and connect what it hears, sees, and says. We train it on our own NVIDIA H100 cluster, in a data center Boson built itself, using a training stack based in part on Megatron.

Multi-teacher on-policy distillation. For post-training, we use multi-teacher on-policy distillation (MOPD). Starting from the same base model, we train specialist teachers with reinforcement learning, including GRPO, on groups of related tasks: natural conversation, interactive behavior, dynamic delegation, and task completion. Through interaction, the teachers learn what to say and when to act—when to speak or yield, begin or extend a reasoning thread, call a tool, or delegate. We then distill them into a single student that combines these capabilities.

A recursive self-improvement loop. We also use a recursive self-improvement (RSI) loop: our agent system interacts with simulated environments, reviews its failures, synthesizes targeted data, and trains the model on it before repeating the cycle. The simulation must reproduce the timing and variety of real interaction—interruptions, overlapping speech, side conversations, background noise—realistic enough to teach natural behavior and hard enough to expose weaknesses. The engineering target is the whole harness of simulators, tools, and judges, not an isolated model. For a small team, the loop turns edge cases into training data and lets us iterate with far less manual data work.

Benchmarks

We compared Higgs Realtime Duplex with the Thinking Machines Lab interaction model and with the realtime and live models available from OpenAI and Google. The suite covers spoken QA, full-duplex interaction, joint video and audio understanding, proactive video (the model watches a clip and chooses when to speak), agentic task completion, and text instruction following. We don’t report voice-cloning or avatar-cloning results, since the other models don’t support them.

Other models’ scores are taken from published reports and leaderboards (TML, Artificial Analysis, Scale AudioMC, OpenAI). We reproduced more than half of them to confirm that our evaluation pipeline matches theirs.

Two things stand out. GPT and Gemini models usually reach their best scores with thinking set high. Model size is another distinction: our front model has 45B parameters, compared with 276B for TML Interaction Small—about six times as many. Even so, the results trade blows. We read this as a sign that the field is still early: data, architecture, and training methods matter more than raw scale.

Try it

Higgs Realtime Duplex is built to feel human, to be genuinely smart, and to cost little. It processes input and output in 160 ms chunks, thinks while you are still talking, works in the background without dead air, and keeps long sessions cheap. An API for Higgs Realtime Duplex is coming soon; join the API waitlist to hear when it launches.

In the meantime, our end-to-end model with avatar output is available today as Higgs Realtime. See the pricing page for rates.

Acknowledgments

Technical Contributors: Mu Li, Dongming Shen, Silin Meng, Ahmad Salimi, Junming Chen, Ke Bai, Alexander Smola, Lindsey Allen, Murdock Aubry, Alex Chen, Yu-Hua Chen, Sepehr Harfi, Ray Huang, Daniel Lee, Jaewon Lee, Erik Li, Bang Liu, Yizhi Liu, Jonah Mackey, Ruskin Manku, Saharat Saengsawang, Nicholas Stranges, Yuzhi Tang, Sergii Tiugaiev, Jixuan Wang, Helin Xiang, Chenxu Xiong, Weisu Yin, Hao Yu, Richard Zhang, Xiling Zhao, Zhewei Zhao, Huapeng Zhou