Controllable TalkNet in 2026: TTS Architecture, Setup, and Evaluation

Controllable TalkNet in 2026: the right framing first

Controllable TalkNet is a text-to-speech model, not a text generation model. It takes a written sentence and produces audio. The “controllable” part is what made it popular in the singing-voice and character-cover community: the user can supply an explicit pitch contour and explicit per-phoneme durations, and the model will follow them. The base architecture comes from NVIDIA’s TalkNet 2 (arXiv:2104.08189); the fork most people actually use is by community author SortAnon, published as the AGPL-3.0 ControllableTalkNet repo with a Colab notebook and two local install paths.

One correction worth making up front, because this article previously repeated it and so does much of the secondary coverage: there is no Controllable TalkNet Space on Hugging Face. SortAnon has published none, and a search of the Hugging Face Spaces index returns no match. The real entry points are the Colab notebook, a Windows setup script, and a Docker image, all listed in the repo README.

This article is the 2026 walkthrough: what TalkNet actually does, how the controllable fork extends it, how to run it in Colab or locally, how to evaluate the output, and where TalkNet fits in the broader open-weight and publicly available TTS landscape that now includes Coqui XTTS-v2, Bark, MetaVoice, Parler-TTS, and OpenVoice.

TL;DR

What is TalkNet?

TalkNet is a non-autoregressive convolutional TTS model. It splits speech synthesis into four predictable steps:

1. A grapheme-to-phoneme (G2P) front end converts text into a phoneme sequence (CMUdict / ARPABET in the canonical NVIDIA implementation).
2. A duration predictor decides how long each phoneme should last.
3. A pitch predictor decides the fundamental frequency at each frame.
4. A mel-spectrogram generator produces the spectrogram, which a separate vocoder (HiFi-GAN is the common pairing) converts to a waveform.

Because the duration and pitch are predicted up front and the rest of the network is non-autoregressive, TalkNet is fast at inference and stays stable across long utterances. NVIDIA’s original TalkNet 2 is described in Beliaev and Ginsburg, 2021 (arXiv:2104.08189).

The image below shows the canonical TalkNet pipeline: text passes through G2P, duration prediction, pitch prediction, and a mel generator before the vocoder.

Figure 1: TalkNet TTS Architecture

What “Controllable” adds on top

The community fork that is most commonly called “Controllable TalkNet” lives at github.com/SortAnon/ControllableTalkNet. Its key additions:

• Explicit phoneme-level duration control. The user can lengthen or shorten any phoneme.
• Explicit pitch contour control. A reference audio can drive the pitch curve for the target voice.
• Reference audio matching. Provide an audio clip and the model will try to match its prosody when speaking new text in the target voice.
• A library of community-trained voices. Many of the voices used in the wider Pony Preservation Project and similar communities are TalkNet checkpoints, which is why the fork is associated with character covers and singing voice synthesis.

The Colab notebook by the same author exposes these controls in a browser UI, which is the lowest-friction way to try TalkNet.

Running Controllable TalkNet

Option 1: the Colab notebook

The fastest path is the community Colab notebook by SortAnon, linked from the repo README. It needs only a Google account. Pick a voice, paste text (or upload reference audio), tweak pitch and duration in the UI, generate. This is good for prototyping a voice match and for non-engineers who only need a clip.

Option 2: Local install

git clone https://github.com/SortAnon/ControllableTalkNet.git
cd ControllableTalkNet
pip install -r requirements.txt

You will need PyTorch with a CUDA-capable GPU for reasonable speed on inference. The repo’s README documents the model checkpoints and the expected directory layout. For new projects, prefer a current TTS stack and treat TalkNet as a specialist tool for when you need explicit pitch and duration control. See the landscape section below for what “current” means in 2026, and note that some of the models this article previously recommended here are either unmaintained or restricted to non-commercial use.

Option 3: Programmatic use in a voice agent

Most production voice agents do not call TalkNet directly. They call a managed TTS API for the TTS step and use TalkNet only for offline tooling (voice match, dataset creation, dubbing). If you do wire TalkNet into a serving path, the usual shape is: ASR (Whisper) → LLM → TalkNet → vocoder → audio out, with traceAI instrumenting each step so you can see WER, LLM latency, and TTS latency in one trace.

Evaluating TalkNet output

Voice quality has four core measurable dimensions. Track each one.

1. Intelligibility (WER)

Run an ASR model (Whisper, Distil-Whisper, or Faster-Whisper) over the synthesized audio and compute word error rate against the source text. WER above a few percent on clean text usually means the voice or the vocoder is failing — though WER alone is not enough for voice agents.

2. Speaker similarity

If you are trying to match a target voice, compute speaker embeddings with ECAPA-TDNN (SpeechBrain) for the synthesized audio and a reference clip, then take cosine similarity. Higher is better; below 0.6 usually means the voice does not match.

3. Prosody fidelity

Compare pitch contour and per-phoneme duration of the synthesized audio against the reference. Standard metrics are pitch RMSE (Hz) and average phoneme duration error (ms). These are the metrics that matter for singing-voice work and dubbing. The full rubric set is in how to evaluate TTS quality for voice AI.

4. End-to-end conversation quality (for voice agents)

For agents that use TalkNet (or any TTS) inside a real conversation, the right level of evaluation is the conversation turn, not the audio clip. Capture traces of the full agent, score each turn for task completion, helpfulness, and safety, and route findings back into the prompt or the TTS configuration.

This is where evaluation tooling earns its keep. Future AGI’s ai-evaluation library (Apache 2.0) ships evaluators that work on trace data:

from fi.evals import evaluate

score = evaluate(
    "answer_relevance",
    output="Your appointment is confirmed for Thursday at 10am.",
    context="User asked to book Thursday morning.",
)
print(score)

For traces, traceAI (Apache 2.0) is the OpenTelemetry-native instrumentation library. It exports spans from any LLM or agent framework via fi_instrumentation.register and FITracer and bundles auto-instrumentors such as traceai-langchain (LangChainInstrumentor), traceai-openai-agents, traceai-llama-index, and traceai-mcp. A voice agent emits one trace per turn with spans for ASR, LLM, and TTS, which is what makes per-step quality and per-step latency visible. Future AGI cloud judges available through fi.evals.evaluate include turing_flash (about 1 to 2 seconds), turing_small (about 2 to 3 seconds), and turing_large (about 3 to 5 seconds) per the cloud-evals reference.

Ethics, consent, and guardrails

Voice cloning systems amplify all the existing risks of synthetic media. The deeper treatment is in voice cloning safety and brand voice management. The 2026 minimum bar:

• Consent. Use only voices you have a rights agreement for. Public figures are not opt-in by default.
• Disclosure. Synthetic audio should be labeled, especially in customer-facing and journalistic contexts.
• Watermarking. Where feasible, embed inaudible watermarks so synthesized audio can be detected downstream.
• Policy guardrails on production agents. Restrict the prompts that can reach the TTS step (no impersonating real people, no harassment, no political fundraising in regulated contexts). Future AGI’s fi.evals.guardrails.Guardrails and the Agent Command Center BYOK gateway at /platform/monitor/command-center can evaluate every voice-agent turn against these rules and, when wired into a blocking workflow, gate the call before TTS runs.

Real-world applications

Where Controllable TalkNet specifically shines:

• Character voice covers and singing voice synthesis. The pitch control is what made the model popular in the fan-music community.
• Dataset prototyping. Quickly produce a sketch of how a target voice would sound saying new text before recording a real performance.
• Offline dubbing experiments. Provide a reference performance and ask TalkNet to speak new lines that match its prosody.

For general voice agents, customer support, and scaled content production, a managed API like Cartesia, ElevenLabs, or Deepgram is usually a better fit.

How Controllable TalkNet compares to other open-weight TTS options in 2026

This is a no-rank list of open-weight and publicly available options; the right choice depends on your use case and the model’s license.

• Controllable TalkNet (SortAnon fork): explicit pitch and duration control; strong for singing voice and character covers; smaller installed footprint of pretrained character voices.
• Coqui XTTS-v2: voice cloning from a few seconds of reference audio, multilingual, and still widely used. Two caveats that matter more than the capabilities. Coqui the company shut down in January 2024, so the original coqui-ai/TTS repository is no longer officially developed; the maintained community fork is idiap/coqui-ai-TTS from the Idiap Research Institute, installable as coqui-tts. And the licensing is split: the Python library is MPL-2.0 and fine for commercial use, but the XTTS-v2 model weights are under the Coqui Public Model License 1.0, which permits non-commercial use only, including the audio the model produces. If you are shipping a product, check that before you build on it.
• Bark (Suno): expressive non-verbal sounds and music; slower; harder to control prosody precisely.
• Parler-TTS: text-prompt controllable TTS; good for descriptive style control (“a slow, calm female voice”).
• MetaVoice-1B: 1B-parameter open-weight TTS with voice cloning.
• OpenVoice (MyShell): style and voice transfer from a short reference.

What the production TTS landscape actually looks like in 2026

Worth being blunt about where this list sits. The open-weight models above are the 2023-24 generation, and most of them have not kept pace. If you are choosing TTS for a production voice agent rather than for offline voice-match work, the decision is now made among hosted models, where the leaders are Cartesia Sonic-3.5 on latency, ElevenLabs Flash v2.5 for real-time conversation and Eleven v3 for expressiveness, Deepgram Aura-2 where you need a signed HIPAA BAA, and Hume Octave 2 for emotional control. Microsoft added MAI-Voice-2 Flash in July 2026.

The trade is the obvious one: hosted APIs give up open weights in exchange for lower latency, SLA-backed availability, and someone else’s problem when a model needs updating. For the current month-by-month picture including pricing and measured latency, see best voice AI models in July 2026, and for a straight vendor comparison see best text-to-speech providers in 2026.

TalkNet’s position in that world is unchanged and still legitimate: it is not competing with any of them. It is the tool you reach for when you need to dictate pitch and timing explicitly, which none of the hosted APIs expose.

Limitations and what to watch in 2026

• TalkNet is not a generative LLM. If you want to change what the voice says, change the upstream LLM. TalkNet only changes how it sounds.
• The rest of the field moved on, and mostly to hosted models. XTTS-v2, OpenVoice, and Parler-TTS cover use cases TalkNet does not, but the open-weight generation they belong to has largely stalled, and production voice work in 2026 runs on hosted APIs. Check maintenance status and weight licensing before adopting any of them.
• The community fork is effectively dormant. The repository is not archived, but its last push was in July 2023. Treat it as a fixed artefact that still runs, not as a maintained production dependency, and do not expect fixes.
• Voice cloning regulation is tightening. Expect more jurisdictions to require disclosure of synthetic audio; build the disclosure into the agent surface, not as an afterthought.

How Future AGI helps voice teams evaluate and monitor TTS pipelines

TTS itself is not Future AGI’s product. The platform’s role around a TalkNet (or XTTS-v2, or ElevenLabs) deployment is the evaluation, observability, and guardrail layer:

• traceAI (Apache 2.0) instruments the full ASR → LLM → TTS stack with OpenTelemetry spans through fi_instrumentation.register + FITracer, plus auto-instrumentors for LangChain, LlamaIndex, OpenAI Agents, and MCP.
• fi.evals.evaluate and fi.evals.Evaluator score the spoken conversation per turn for helpfulness, answer relevance, faithfulness, and tool correctness.
• fi.evals.metrics.CustomLLMJudge + fi.evals.llm.LiteLLMProvider let you bring your own LLM judge for voice-specific criteria (clarity, persona match).
• fi.simulate.TestRunner generates synthetic voice agent scenarios so you can A/B prosody and prompts before they hit production.
• fi.evals.guardrails.Guardrails plus the Agent Command Center BYOK gateway at /platform/monitor/command-center evaluate every call against safety, brand voice, and synthetic-audio disclosure policies, and can be wired into approval or blocking workflows where the team chooses to enforce them inline.

Authentication uses FI_API_KEY and FI_SECRET_KEY (two variables, not one).

Summary

Controllable TalkNet is a text-to-speech model with explicit pitch and duration control, derived from NVIDIA’s TalkNet 2 and packaged as the community SortAnon fork. It is not a text generator; it does not invent prose. In 2026, TalkNet remains the right tool for singing voice and character cover work, while general voice agents have moved to managed APIs like Cartesia, ElevenLabs, and Deepgram. Whatever TTS stack you ship, instrument it with traceAI, evaluate every turn with fi.evals.evaluate, and put guardrails around what the voice is allowed to say.

Sources

• TalkNet 2 paper, Beliaev and Ginsburg (arXiv:2104.08189): https://arxiv.org/abs/2104.08189
• ControllableTalkNet repository and install instructions (AGPL-3.0): https://github.com/SortAnon/ControllableTalkNet
• Coqui shutdown announcement (January 2024): https://github.com/coqui-ai/TTS/discussions/3489
• Maintained Coqui TTS fork by Idiap Research Institute: https://github.com/idiap/coqui-ai-TTS
• XTTS-v2 model card and Coqui Public Model License text: https://huggingface.co/coqui/XTTS-v2
• Future AGI traceAI (Apache 2.0): https://github.com/future-agi/traceAI