Post-Training a Sub-1B Voice Model for Hindi and Hinglish

TL;DR

• Reduced Word Error Rate (WER) to 21% on customer call-center validation sets (vs 27–31% for leading STT vendors)

• Halved streaming latency to ~93 ms (vs ~210 ms for the next-best model on comparable hardware)

• Ran 20× faster on CPU-only inference vs Whisper-v3-turbo

• Trained on 10,000+ hours of proprietary call-center audio inside a 5-day compute window, inside the customer's VPC

Frontier speech models continue to get better on benchmark evaluations, but hit a wall in genuine enterprise settings in India: a regional Indian dialect, a noisy showroom mic, or enterprise data residency requirements. These failure modes prevented Tata Motors, one of the world's largest automakers, from successfully implementing off-the-shelf AI models in their voice workflows.

In partnership with Tata Motors, Hyde built a specialist voice AI model using pre-existing customer conversation recordings. We took years of call-center recordings, shaped them into training data, generated synthetic examples grounded in Tata's own environment, and post-trained Nvidia's 0.6B-parameter Parakeet voice model into a Hindi and Hinglish automatic speech recognition model. Our fine-tuned model outperformed every leading Speech-to-Text model in performance testing on Tata Motors' own data.

This post walks through how we did it: preprocessing and synthetic data, self-supervised continued pretraining, post-training and the kernel work that made it feasible, and deployment across both GPU and CPU targets.

Why post-train at all?

The primary motivation for post-training was simple: frontier models failed to do the job required. We tested a variety of models - Whisper v3, Deepgram Nova-2, Sarvam Saarika, Azure Speech, ElevenLabs Scribe, Google STT - and found that each of them showed degraded performance on Indian regional Hindi and Hinglish code-mixing. Reductions in error rates were crucial to business outcomes: an incorrect transcription meant that voice agents making automated sales calls often heard a customer referencing the wrong car e.g. Tata 'Sierra' instead of Tata 'Harrier', leading to a wasted call as opposed to a targeted follow-up.

The second motivation for post-training was ownership: a post-trained model would be Tata Motors' own asset, and could be continually improved using their own training data.

Data pre-processing and synthetic data generation

We picked Parakeet-TDT 0.6B as the STT base model and Qwen3 for the sales agent workflows (the 0.6B variant for a sales next-action predictor and the 8B variant as a conversational LLM) and built the rest of the pipeline around getting these models to actually work on this customer's data.

Shaping the data

Tata Motors' call-center stack (inbound and outbound sales, service, and retention calls) already had raw audio files with metadata: dealer code, agent ID, customer phone number, call disposition (closed_lost, sale_closed, callback), CRM lead ID, and timestamps. This data lives inside the customer's Palantir Foundry tenant, enabling us to model it as ontology objects (CallRecording, linked to Customer, Lead, Dealer, Vehicle, Touchpoint).

Ontology objects, by design, gave us three significant advantages relative to using raw file dumps:

• Every transcript, embedding and training run can be traced back to a source recording and its consent flag

• Consent and data privacy policies are enforced automatically at the per-row level e.g. a data scientist without the relevant PII clearance cannot access unredacted audio recordings

• The same objects feed both the training pipeline and the live agent's lookups at inference, ensuring that there's no separate copy of data that falls out of sync with what the model learned on.

This feeds through to our training manifest creation. Typically, training manifests are built by exporting a snapshot of the data, writing one-off code scripts to filter it, and saving the result as a file. Our training manifests are queries that return training datasets on demand.

manifest = (
    call_recordings
        .where(disposition.in_(["sale_closed", "closed_lost"]))
        .where(language_label == "hi")
        .where(duration_sec.between(2.0, 30.0))
        .where(consent_flag == True)
        .where(pii_redacted == True)
        .select("audio_uri", "transcript_clean", "duration_sec", "dialect_cluster")
)

manifest = (
    call_recordings
        .where(disposition.in_(["sale_closed", "closed_lost"]))
        .where(language_label == "hi")
        .where(duration_sec.between(2.0, 30.0))
        .where(consent_flag == True)
        .where(pii_redacted == True)
        .select("audio_uri", "transcript_clean", "duration_sec", "dialect_cluster")
)

This query resolves into a JSONL manifest (audio_filepath, text, duration) that feeds directly into the training scripts below. The result: training datasets that are always current, with clean audit trails and compliance enforced by design.

Synthetic data generation

Even with 10,000+ hours of real recordings, we found significant gaps in the training dataset:

• Long-tail phrases: SKU names (Curvv EV Dark Edition, Punch CNG XT Plus), dealer financing schemes, and festival offers - each crucial to the sales pipeline - were under-represented in any corpus of this size.

• Numeral and code-mixing: Spoken Hindi numbers (e.g. "chaar lakh saath hazaar") need to map reliably to "4,60,000," and romanized loanwords ("EMI," "down payment," "booking") need consistent Devanagari transliteration.

• Long-tail objections: Specific but important customer concerns (e.g. "worried about waiting period" or "comparing against a competitor's offer") appear only a few hundred times across 10,000 hours, proving too sparse to train on.

Synthetic data generation - conditioned on Tata's actual sales environment - allowed us to systematically plug these gaps.

Cleaning and transliteration

Our cleaning and transliteration pipeline used a frontier LLM (Qwen3.5-35B-A3B, a 35B-parameter MoE with 3B active) to run a four-step pass over every transcript:

• strip annotation tags (<unintelligible>, <noise>)

• transliterate Roman characters to Devanagari

• convert numerals to their lexical spoken form (preserving whether a number was spoken as a whole number or digit-by-digit)

• filter out any pair whose characters-per-second ratio exceeds physiological speech limits, a reliable signal of misalignment.

Code-mixing aware language classification

As we referenced above, one of Tata's biggest challenges was dealing with a peculiarity of Indian language use: sentences are often constructed as a mix of Hindi and English (Hinglish). Standard language ID models classify a whole utterance as one language, leading them to fail on a phrase like "Main Sierra book karna chahta hoon" (i.e., a sentence using Hindi grammar but English nouns).

To solve this challenge, we decided to fine-tune facebook/mms-lid-256 (a 1-billion parameter model trained to recognize 256 languages). Our primary challenge was training data: labeled audio clips of mixed-language phrases don't exist off the shelf. We built our training corpus using two techniques:

• We already had actual separate Hindi and English call segments. We spliced these segments together at sentence boundaries with original language tags serving as training labels.

• We also had transcripts with genuine code-switching: sentences where customers wove Hindi and English together in the same sentence. We fed these transcripts to a text-to-speech system to generate synthetic audio clips (using a variety of voices that covered different Indian accents). These artificial audio clips could be paired with the per-word language tags from the transcripts, which served as training labels for this set.

These two methods cover each other's weaknesses: one contributed genuine audio clips with artificial splices, and one contributed artificial audio clips with genuine language-switching.

This classifier sits upstream of our main speech-recognition model; thus, each additional point of language detection accuracy in the upstream classifier results in a lower transcription error rate in the downstream automatic speech recognition workflow.

Self-supervised continued pretraining

Training speech-recognition models requires labeled training data. Labeled training data is expensive. We had an enormous corpus of un-labeled audio from Tata Motors. Self-supervised pre-training allowed us to exploit it.

The magic of self-supervised pre-training is that it turns the existing un-labeled audio recordings into the teacher. The "wav2vec 2.0-style" approach we used takes an audio clip, hides random chunks of it, and makes the model predict what belongs in the gaps using the surrounding context. This approach requires no labeling - the answer key is the original, unmasked audio.

We used the following setup:

• FastConformer encoder - the encoder's architecture matches that of the Parakeet-TDT 0.6B v2 so weights transfer directly

• Gumbel-Softmax vector quantizer with product codebooks

• Contrastive loss with 100 negatives, plus a diversity loss to keep codebook utilization healthy

• Time masking at mask_time_prob = 0.65

  
  

python train_pretrain.py \
    --train_manifest unlabeled_audio.json \
    --pretrained_encoder /path/to/parakeet-tdt-0.6b-v2.nemo \
    --exp_name

python train_pretrain.py \
    --train_manifest unlabeled_audio.json \
    --pretrained_encoder /path/to/parakeet-tdt-0.6b-v2.nemo \
    --exp_name

We ran this pre-training on 8,000 hours of unlabeled audio: we combined the Vaani corpus (an open dataset of Indian speech recordings) with Tata's own un-transcribed call backlog.

At this stage, the model isn't transcribing; thus, to figure out whether the pre-training was useful, we measured a proxy: is the model able to tell regional accents and dialects apart? If the pre-training was successful, we would expect accuracy on this metric to increase, and it did: regional accent/dialect classification accuracy jumped from 35% to 47% when the fine-tuning was initialized from the pre-trained encoder instead of the stock encoder.

We performed an additional, cheaper check to ensure the pre-training was actually learning something dialect-relevant and not just memorizing noise: we extract encoder embeddings (the internal representations produced by the model for different recordings) and run UMAP (a technique for flattening this high-dimensional data into a picture). The result is shown below: northern and southern dialect recordings cluster in separate regions - a strong indication that the pre-training procedure absorbed genuine signal structure from the audio recordings.

Encoder embeddings projected with UMAP. Northern and southern dialect recordings cluster in separate regions.

Post-training and kernel work

With the pretrained encoder in hand, we ran supervised fine-tuning on the full labeled corpus: 600 hours of IndicVoices plus the 10,000+ hours of customer call-center recordings.

The training stack is plain PyTorch Lightning, emitting NeMo-format checkpoints so it slots into any NeMo-aware serving stack:

python train_asr.py \
    --train_manifest train.json \
    --val_manifest val.json \
    --tokenizer_model tokenizer.model \
    --pretrained_model /path/to/parakeet-tdt-0.6b-v2.nemo \
    --exp_name

python train_asr.py \
    --train_manifest train.json \
    --val_manifest val.json \
    --tokenizer_model tokenizer.model \
    --pretrained_model /path/to/parakeet-tdt-0.6b-v2.nemo \
    --exp_name

The tokenizer itself is built from the customer's actual call distribution rather than a generic Hindi corpus i.e. the vocabulary contains words found in Tata's actual sales calls ("ex-showroom", "EMI") and does not waste vocabulary slots on words that Tata's customers never say. This decision, made prior to training, pays off throughout the entire workflow.

The kernel bottleneck

The bottleneck in our workflow wasn't the data-loader or self-attention; with 10,000 hours of training data, the bottleneck was the Token-and-Duration Transducer (TDT) loss itself.

The off-the-shelf PyTorch implementation of TDT (which is an audio streaming-friendly loss function that scores every plausible way of lining up the transcript with the audio, all at once) spends most of its time in inefficient non-fused and non-vectorized lattice operations. GPU utilization sat at 19% - the model was idle, waiting for the loss calculation, for most of every training cycle.

We adapted Warp-Transducer with custom CUDA kernels specifically for TDT using three techniques:

• diagonal wavefront traversal for the forward/backward passes

• precomputed log-softmax for both labels and durations

• an omega parameter for RNNT/TDT loss mixing

Our adaptations took GPU utilization from 19% to 66%, and end-to-end training efficiency improved 100–200× over the stock kernels. We managed to take a training cycle that would have usually taken multiple weeks and compress it to run in just five days.

You can't do post-training research at scale if every experiment costs three weeks. This style of optimization is a key feature of the way we build post-training workflows for customers at Hyde: we recognize that reducing experimentation time is crucial to companies building their own fine-tuned models.

Validation

For validation, we built a held-out dataset that is frozen at ingestion time - a recording is assigned 'validation' status once and that label is never changed, ensuring no leakage of training data into the evaluation set.

We track the following metrics: val_wer, val_cer, training/validation loss, gradient norms per encoder/decoder/joint head and per-dialect WER (to ensure the model isn't failing on specific regional accents).

To confirm the soundness of our methodology, we ran an early fine-tuning pass on just the 600 hours of IndicVoices: we found that WER was brought down to a 16% range, which was low enough for us to commit the same architecture and data pipeline to a full training run on the larger 10,000 hour corpus.

Validation WER on 600 hours of IndicVoices.

Deployment: one more model, two serving paths

Before deploying the voice stack, one more fine-tuned model rounds out the system.

Distilling sales next-best actions into a small model

Tata Motors' sales challenges extended beyond just speech recognition: they were looking for a model that could recommend a next action to sales agents based on a customer's touchpoint history. This workflow needed to scale cost-effectively to millions of customers, ruling out the possibility of using large models with prohibitively expensive inference.

We fine-tuned Qwen3-0.6B to create a lightweight and accurate model. We generated training data in a three-step process:

• we sampled clusters of touchpoints for each customer (WhatsApp messages, inbound calls, missed calls, and web visits)

• we used a larger model (Qwen3-30B-A3B) and asked it to play an experienced sales agent and produce a grounded next-best action (e.g., "send finance brochure for Sierra in Hindi over WhatsApp; do not call before 6pm")

• we validated every generated action against hard business rules: no calls during dealer off-hours, no offers below the minimum retail price floor, correct dealer routing

Once our STT model converges and the training workflow is complete, we shift to deployment. The model is converted back to NeMo format and the deployment footprint is split across two different workflows with separate latency requirements, both using the same model checkpoint: a real-time workflow for live phone calls and a batch transcription process that runs overnight.

Real-time voice-to-voice (GPU, AWS)

The live phone call workflow utilizes two models: i) the fine-tuned STT model converting customer speech to text, and ii) an off-the-shelf Qwen3-8B model that decides what the voice agent should say next. The infrastructure choices below enable us to serve up to 70 concurrent calls.

The full voice loop also brings in Cartesia for TTS and Plivo for the SIP trunk. A sluggish start is fatal for a live voice call workflow, so we configured a LiveKit AgentWorker that prewarms a voice-activity detector ensuring the encoder is already warm when a call connects.

Once connected, the loop is:

• the STT transcribes the call

• the conversational LLM, holding full conversation context, decides whether to answer directly or call a tool

• the tool calls (getProposalDetails, getOpenBookings, sendProposal, recordCustomerResponse) route into the customer's Palantir AIP agent workflows for grounding lookups (holding no PII in-session beyond ephemeral JSON)

• TTS streams the response

The user-facing client itself is hosted in AIP, which also issues the OAuth token that authenticates the call.

Batch and pipeline work (CPU, inside the customer's data warehouse)

The same Parakeet checkpoint runs on CPU for overnight transcription, training-data generation, and QA tagging. We use our fine-tuned facebook/mms-lid-256 code-mixing classifier to route recordings into per-language corpora. The fine-tuned Qwen3-0.6B sales-agent recommender runs as part of the pipeline - it is automatically triggered whenever a new touchpoint lands on a lead.

Why CPU here? CPU isn't winning on raw throughput-per-dollar: a GPU still beats it on that metric. It wins for two narrower reasons. First, call-center fleets and Foundry tenants have CPU capacity sitting idle overnight that's already paid for: using it is close to free. Second, for latency-insensitive work like training-data extraction and compliance review, a CPU-only instance is cheaper to provision than the smallest GPU instance that would do the same job. The right metric here is cost-per-day's-recordings, not throughput-per-dollar: that's why the batch path runs on CPU inside Foundry and the live path runs on GPU on AWS.

Benchmarks and comparisons

STT accuracy and latency

On the customer's call-center validation set (Hindi + Hinglish), against Whisper v3, Deepgram Nova-2, Sarvam Saarika, Azure Speech, ElevenLabs Scribe, and Google STT:

None of the vendors we tested came within 6 absolute WER points of the fine-tuned model on this set, and the next-fastest streaming latency was more than 2× slower on the same hardware tier.

CPU batch throughput

Transcribing a day's worth of call-center audio:

The 20× gap is what makes the batch path practical at all: at 40 hours per 13 hours of audio, a batch job would slip past the next day's intake; at 2 hours, the day's calls are fully indexed before the morning shift starts. Our fine-tuned model also requires only 4GB of RAM, enabling it to run on modest hardware.

Effect of self-supervised pretraining

On regional dialect classification accuracy:

That 12-point jump comes from stacking two stages of post-training: self-supervised pretraining on 8,000 hours of unlabeled audio, followed by supervised fine-tuning on 600 hours of IndicVoices plus 10,000+ hours of customer recordings.

Frontier problems we're working on next

These are the three threads we're actively pulling on:

• Omni-model fusion: Building on Qwen3-Omni to fuse STT, LLM, and TTS into a single end-to-end model, with the goal of reducing end-to-end pipeline latency below that of a chained system of separate models

• Agentic STT: Fine-tuning larger models like Canary to handle long-form, open-ended sales conversations, turning automatic speech recognition from a passive transcriber into a co-pilot that grounds every utterance in real-time lookups against the customer's systems.

• Silent voice co-pilots: A model that listens to a human agent's live call, surfaces relevant proposals, and writes back to the CRM, without ever speaking into the call itself - this enables the same stack to support a human agent workflow as well.

If you want to talk about any of this in more depth (model architecture, the CUDA work, the synthetic-data pipeline, or the deployment setup), please get in touch.

Team Hyde, info@hyde.ai