2. The three parallel encoder streams
The heart of TriStream is a set of three encoders that process the same underlying audio event from entirely separate vantage points, with no ability to exchange information until much later in the network.
| Stream | Depth | Receives | Models |
|---|---|---|---|
| Source | 3 layers | Log-scaled F0 contour, voiced/unvoiced flag | The glottal pulse train — perceived pitch |
| Filter | 5 layers | Mel spectrogram, singer embedding, emotion/technique embedding, lyric cross-attention | Vocal-tract resonance — formants, timbre, identity |
| Residual | 2 layers | Mel spectrogram | Breath noise, fricatives, articulation texture |
Stream 1 — Source. This stream receives only two numeric inputs: a log-scaled F0 pitch contour and a binary voiced/unvoiced flag. Critically, it never receives the mel spectrogram, the singer embedding, or any timbre-carrying signal whatsoever. It is architecturally incapable of representing anything about who is singing — it can only ever describe what pitch is being produced.
This is what makes the disentanglement structural rather than merely encouraged. There is no pathway through which timbre information could reach the source stream, even if training somehow pushed the model in that direction. Most systems separate pitch from identity by adding a loss term and hoping; here the separation is a property of the wiring.
Stream 2 — Filter. The deepest of the three, reflecting that formant structure and timbre are the hardest components of a voice to model well. It receives the mel spectrogram along with a singer identity embedding and an emotion/technique embedding, and cross-attends against the lyric representation so it always knows what phoneme is currently being produced. This is the only stream that carries identity information.
Stream 3 — Residual. Handles breath noise, fricative consonants and general articulation texture — the parts of a vocal performance that aren't well described by either clean pitch or clean timbre.
3. The lyric and speaker encoders
The lyric / phoneme encoder is a 4-layer transformer that processes input lyrics independently, converting them into a representation that conditions the Filter stream and, later, the Fusion Trunk, by cross-attention. This keeps "what should be sung" cleanly separated from "how it should sound" throughout most of the network.
The speaker encoder is a separate convolutional network that extracts a portable "voice fingerprint" from an arbitrary reference clip — including from voices the model has never encountered during training. It uses:
- Dilated convolutions, to build a wide receptive field cheaply
- Squeeze-excitation blocks, letting the network weight which channels matter most
- Attentive statistics pooling — a weighted mean and standard deviation rather than a simple average — to collapse a variable-length clip into a single fixed-size embedding
It is trained with a contrastive objective: two different clips from the same speaker are pulled together in embedding space, while clips from different speakers are pushed apart. That is what gives the embedding space genuine, learned structure — rather than an arbitrary set of numbers, it becomes a space where "closeness" reliably corresponds to "sounds like the same person," which is the property that makes zero-shot cloning of unseen voices possible at all.
4. Late fusion and the adversarial singer head
The late fusion trunk is an 8-layer transformer — again using cross-attention against the lyric encoder — where the three separately-processed streams are finally allowed to combine into one unified representation. This fusion point is deliberately placed as late as possible in the network. Everything before it has been kept artificially separated; this is the single location where pitch, timbre and texture information are permitted to interact.
The adversarial singer head is a small classifier attached to the trunk's output, trained to guess which singer produced a given piece of audio using only the shared representation. Its gradient passes through a gradient reversal layer before propagating back into the trunk — so as the classifier gets better at identifying singers, the trunk is pushed in the opposite direction, actively degrading its own identifiability.
The practical effect is a shared representation explicitly trained to be as uninformative about singer identity as possible. It is pushed toward encoding general vocal mechanics — how a formant transition behaves, how a phoneme boundary sounds — rather than person-specific fingerprints that could be exploited to memorise particular singers.
This is domain-adversarial training, applied here specifically as an anti-memorisation mechanism rather than for its more typical use in domain adaptation.
5. Duration, pitch and beat alignment
When generating an entirely new performance — as opposed to editing existing audio — the model has no ground-truth timing or melody to read from. It has to invent both.
The duration predictor estimates how many audio frames each phoneme should occupy, predicted in logarithmic space specifically because real note durations are heavily skewed: a sustained vowel might last dozens of times longer than a short consonant, and predicting in log-space prevents the long notes from dominating the training signal.
The pitch predictor deliberately avoids outputting one blended pitch curve. It separately predicts:
- Base pitch — the target note itself
- Scoop — the natural human tendency to slide into a pitch rather than hit it instantaneously, which is one of the clearest tells separating human singing from robotic-sounding synthesis
- Vibrato depth and rate — ramped in gradually over roughly a quarter-second rather than snapping on instantly, mirroring how real vibrato builds over a sustained note
Because these components are generated separately, they remain independently adjustable after the fact — vibrato intensity can be dialled up or down without disturbing underlying pitch accuracy.
Beat alignment. When a backing track accompanies a generation request, a beat-tracking step extracts tempo and downbeat positions from that track, and predicted note durations are pulled toward that rhythmic grid using soft quantisation rather than a hard mechanical snap. Perfectly quantised timing is one of the most reliable tells of synthetic vocals; TriStream's alignment allows a controlled degree of natural deviation from the grid, including an adjustable swing parameter, specifically to avoid that mechanical quality.
6. The flow-matching decoder
The final generative stage is a 5-layer decoder built around rectified flow matching, a more modern and computationally efficient alternative to the diffusion-based approaches used by earlier singing synthesis systems.
Rather than learning to slowly reverse a noise-adding process across 100 or more denoising steps, the model learns to predict a direct velocity field pointing from random noise toward the target mel spectrogram. This allows full-quality audio generation in roughly 32 sampling steps — a substantial efficiency improvement over the diffusion approach used in systems like the original DiffSinger, while maintaining comparable output fidelity.
7. Two modes, one underlying model
Editing is the model's native training objective — not a feature added on top of a generation-only system. During training, random contiguous spans of real audio are masked out, and the model is trained to reconstruct exactly that masked span using the surrounding, unmasked audio as context, along with the lyrics for that section.
This means "regenerate this one phrase with different lyrics" or "make this section sound more breathy" is not a special-cased inference trick bolted on afterward — it is the literal task the network was designed and trained around from its first training step.
Generation mode reuses every one of the same trained components, but begins from a fully masked target rather than a partially masked one: the speaker encoder clones the requested voice from reference audio, the duration and pitch predictors invent complete timing and melody from the lyrics (and any supplied melody guide), and the decoder generates the entire performance from scratch rather than filling a gap in existing material.
8. Training methodology
Singer-disjoint validation. A held-out set of singers is determined by a stable hash function applied to singer identity, and that set is completely excluded from training data at the point of data loading — not merely reserved after the fact. Every reported validation score is measured exclusively against voices the model has never once encountered.
This is meaningfully stricter than a random validation split, which can silently allow the same singer's recordings to appear in both training and validation — producing validation scores that look strong but don't measure generalisation at all. The singer-disjoint approach is the only way to be confident the model is learning transferable vocal mechanics rather than quietly memorising the individuals in its training data.
Crash-resilient, autonomous checkpointing. The training pipeline automatically pushes checkpoints to a remote Hugging Face repository whenever the singer-disjoint validation score improves on its previous best, and additionally on a rolling time-based interval regardless of improvement. If a session is interrupted for any reason — a platform session limit, an infrastructure failure, a manual restart — the next invocation of the training script automatically detects the most recent checkpoint, including full optimiser state, and resumes from precisely that point rather than restarting from initialisation. This infrastructure was built and hardened specifically in response to real interruptions encountered during development, including data-loader failures under multiprocessing and transient Hub connectivity issues, both now handled automatically with retry logic and graceful fallback.
Careful data curation. Training data is deliberately restricted to dry, isolated vocal recordings, with instrumental bleed, heavy reverb and mixed-track audio explicitly excluded — because mixed audio actively corrupts the mel-spectrogram reconstruction target the model trains against, regardless of how much of it is fed in. The corpus combines large-scale speech data, used to teach general vocal-tract mechanics and broad phonetic coverage, with smaller, carefully verified singing corpora used specifically for pitch behaviour, vocal technique and musical phrasing. Datasets are individually probed and verified before inclusion — the pipeline tests that each configured source actually loads and decodes correctly before committing to it in a training run.
Anti-overfitting measures, beyond the adversarial disentanglement described above:
- Waveform-level augmentation — pitch shifting, time stretching and gain variation, so the model never sees the exact same acoustic realisation of a clip twice
- SpecAugment-style frequency and time masking applied directly to the mel spectrogram
- Mixup between training samples within a batch, discouraging the model from latching onto exact spectral fingerprints of individual recordings
- Early stopping keyed directly to the singer-disjoint validation score, with patience-based tolerance for normal step-to-step noise
- The best-performing checkpoint by validation score — not simply the most recent — is what gets preserved and deployed, tracked and saved as an entirely separate file throughout training
9. Technical specifications
| Property | Value |
|---|---|
| Total parameters | 321.8M |
| Architecture family | Custom transformer, source-filter factored |
| Positional encoding | Rotary Position Embeddings (RoPE) |
| Normalisation | RMSNorm |
| Feed-forward network | SwiGLU |
| Attention | Scaled dot-product with cross-attention conditioning |
| Generative paradigm | Rectified flow matching |
| Typical inference sampling steps | ~32 |
| Sample rate | 24 kHz (architecture supports scaling to 44.1 kHz) |
| Mel spectrogram resolution | 100 mel bands (scales to 128 at higher sample rates) |
| Speaker conditioning | Zero-shot, via dedicated contrastive speaker encoder |
| Validation methodology | Singer-disjoint held-out evaluation set |
| Checkpointing | Automatic, crash-resilient, Hub-synchronised |
| Training precision | bf16 mixed precision on supported hardware |
10. Limits and development status
TriStream-SVS is actively in training and is not a finished product. The complete architecture and pipeline are functioning end to end — data ingestion and verification, the three-stream encoder system, adversarial disentanglement, the speaker encoder, duration and pitch prediction, flow-matching generation, and fully automated crash-safe checkpointing are all live and operating together. Validation loss measured against held-out, previously unseen singers has continued to improve across training, which is the primary signal being tracked.
What is working
- The full pipeline runs end to end, from data ingestion to generated mel spectrogram
- Singer-disjoint validation loss improving across training
- Automated crash-safe checkpointing and resume, hardened against real failures
- Editing and generation sharing one set of trained components
What it is not, yet
- Not a stable release. There is no frozen release checkpoint; behaviour changes between checkpoints as training continues
- Not benchmarked. No published objective scores and no formal listening tests. Validation loss is the only tracked signal, and loss is not the same thing as sounding good
- Not full-band. Output is 24 kHz with 100 mel bands. The architecture supports 44.1 kHz and 128 bands, but that is a future configuration, not what runs today
- Sensitive to reference quality. Training is restricted to dry, isolated vocals, so reference clips with instrumental bleed, heavy reverb or mixed-track audio are out of distribution and should be expected to degrade cloning quality
- Bounded by its corpus. Phonetic and stylistic coverage extends only as far as the training data; languages, vocal techniques and genres outside it are not supported claims
- Not for production or safety-critical use. This is an independent research project
The model's own output is a mel spectrogram, so producing a listenable waveform requires a vocoder stage on top of it — worth knowing before planning an integration. As with CorX1.5, the limits here are documented as carefully as the capabilities, and this page will be updated as training progresses rather than rewritten to look better than the results are.
11. Responsible use
TriStream-SVS can reproduce the vocal characteristics of a person from a short reference clip, including people who were never part of its training data. That is a capability worth stating plainly rather than burying.
Clone a voice only with the consent of the person it belongs to. Do not use this model to impersonate anyone, to produce material attributed to a real artist without their agreement, or to generate anything intended to deceive a listener about who is singing. The adversarial singer head reduces the model's tendency to memorise specific individuals from training data — it does not, and cannot, address misuse of the zero-shot cloning path at inference time. That part is on whoever runs it.
12. FAQ
What is TriStream-SVS?
A 321.8M-parameter singing voice synthesis model built from scratch by CorX Labs in Jamaica. It builds the source-filter model of human voice production directly into its architecture: a source stream handles pitch, a filter stream handles timbre and formants, and a residual stream handles breath and fricative texture. The three are processed separately and combined only in a late fusion trunk.
How does it separate pitch from voice identity?
Structurally rather than statistically. The source stream receives only a log-scaled F0 contour and a voiced/unvoiced flag — never the mel spectrogram or the singer embedding. There is no pathway through which identity information could reach it, so the separation holds regardless of training pressure.
Can it clone a voice it has never heard?
Yes — that is what the speaker encoder is for. It turns an arbitrary reference clip into a fixed-size voice embedding using dilated convolutions, squeeze-excitation blocks and attentive statistics pooling, trained contrastively so that closeness in the embedding space corresponds to sounding like the same person. Quality depends heavily on the reference clip being dry and clean — see limits.
Why flow matching instead of diffusion?
Rectified flow matching predicts a direct velocity field from noise toward the target mel spectrogram, instead of reversing a noise-adding process over 100+ steps. That gets full-quality generation in roughly 32 sampling steps — a substantial efficiency gain over the diffusion approach used in systems like the original DiffSinger.
Is it finished? Can I use it now?
It is actively in training. The architecture and pipeline run end to end, but there is no stable release checkpoint, no published benchmark scores and no listening tests. Treat it as a research project in progress. The Hugging Face repository and its commit history are the source of truth for what currently exists.