What This Project Does

microsoft/VibeVoice is a family of open source speech AI models from Microsoft that covers both text to speech (TTS) and automatic speech recognition (ASR). The core idea is to use a large language model to understand the text and the conversation context, then attach a diffusion generation head that fills in the high fidelity acoustic detail. The approach relies on a 7.5 Hz low frame rate continuous speech tokenizer that encodes both acoustic and semantic information, which keeps the audio quality while lowering the compute cost of long sequences. The family has three parts: VibeVoice-ASR turns long audio into text, VibeVoice-TTS synthesizes long text into multi speaker voice, and there is also a lightweight, real time VibeVoice-Realtime built for streaming where you speak and it answers on the fly.

What Problem It Solves

It targets the hard cases of speech work: very long, multi speaker, and high fidelity, the kind of job that used to mean cutting the audio into chunks and running models in sequence, which often broke the flow. VibeVoice-ASR handles a single audio clip up to 60 minutes long and produces structured transcripts inside a 64K token window, labeling who spoke, when they spoke, and what they said in one pass. Speaker separation, timestamps, and recognition run jointly instead of needing a separate diarization model, and you can add custom hotwords to steer the output. VibeVoice-TTS synthesizes up to 90 minutes of dialogue with as many as 4 speakers while keeping each voice consistent and the meaning coherent, which makes podcasts, audiobooks, and course recordings far less tedious to produce. VibeVoice-Realtime-0.5B is a lightweight streaming TTS with first audible token latency around 300 milliseconds, good for voice assistants and live captions. ASR also ships a streaming mode and an edge CPU build that runs without a GPU, so the same model can live on a server or a small device.

Language and Architecture

  • The main language is Python. Model weights are republished under the microsoft organization on Hugging Face, with a Colab demo and papers, and the TTS paper was accepted as an Oral at ICLR 2026.
  • The model lineup includes VibeVoice-ASR-7B, VibeVoice-ASR-Streaming, VibeVoice-ASR-BitNet (the CPU build, which uses I8_S plus I2_S heterogeneous quantization to shrink 4.62GB down to 1.58GB and stays under RTF 1 on 3 or more CPU threads, run through VibeASR.cpp), VibeVoice-TTS-1.5B, and VibeVoice-Realtime-0.5B.
  • On integration and inference, ASR is already wired into Azure AI Foundry Labs and shipped in Hugging Face Transformers, with vLLM inference and streaming recognition supported. ASR is natively multilingual across 50 plus languages, TTS covers Chinese, English, and more, and Realtime-0.5B supports 9 languages and 11 English styles.
  • The base model underneath is Qwen2.5 1.5B. One thing to watch: in 2025-09 the repo removed the inference code for VibeVoice-TTS over misuse risk, so it now positions itself as a research framework. The team advises responsible use within compliance and warns against deepfakes and misinformation.

License

MIT

Microsoft builds it and it is a frontier open source speech AI, which already draws attention. Its 60 to 90 minute long context and multi speaker ability, combined with real time inference on edge CPUs that have no GPU, are a rare mix, because most long form speech tools either eat VRAM or cut the length short. Add the academic backing of an ICLR 2026 Oral and the Azure integration, and the heat stays high. It has around 56k stars in total, with about 223 added that day. For teams shipping speech products, a Microsoft backed, permissively licensed open source option that also runs on CPU saves the effort of building a base from scratch, and it removes the dependency on a commercial API whose bill grows with every minute of audio.

Where to Find It

https://github.com/microsoft/VibeVoice (project site https://microsoft.github.io/VibeVoice/)

Getting Started

To try it locally, run the ASR Colab first and watch how the long audio transcription comes out structured, with speakers and timestamps where you expect them. For production, the CPU build runs through VibeASR.cpp: quantize the model and drop it onto edge devices for real time recognition, which avoids GPU cost and fits meeting notes and interview cleanup. The TTS side has no inference code right now, so reproducing it means implementing from the paper, and that is best kept to research rather than production. If you also want to edit and denoise the recordings, see What Is Audacity and How to Use It, which chains with VibeVoice into a workflow from recording to transcript to polish. Whichever part you use, do not clone someone's voice without consent, since the team has flagged deepfake risk explicitly, and the responsible use boundary is part of the project, not an afterthought.

Summary

microsoft/VibeVoice puts three hard problems, long form, multi speaker, and high fidelity, into one open source framework, and it also solves the edge case of running without a GPU. For teams that need long audio transcription or long dialogue synthesis, it is one of the options worth watching, especially when the budget is tight and you do not want to be locked into a commercial API that charges by the minute.

FAQ

Can VibeVoice ASR and TTS be used together? Yes, but they are two separate pipelines. ASR first turns long audio into text with speakers and timestamps, then TTS synthesizes that text into multi speaker voice. They share the same tokenizer idea, but there is no end to end join in the code, so you feed the output back yourself.

Can it run without a GPU? Yes. The BitNet ASR build runs on CPU through VibeASR.cpp, and after quantization the model is only 1.58GB and stays real time on 3 or more threads. TTS has no official CPU inference code, so reproducing it is harder and has to follow the paper.

Why can I not find the TTS inference code? The repo removed the VibeVoice-TTS inference part in 2025-09, and the team now treats the project as a research framework, mainly out of concern it could be used for deepfakes. To use TTS you implement from the ICLR 2026 Oral paper and stay inside responsible use boundaries.