Benchmarking Streaming ASR for Vietnamese
Benchmarking Streaming ASR for Real-time Deployment: A Robustness Scorecard and Error Taxonomy for Vietnamese
Built a reproducible benchmark for Vietnamese ASR models across offline and simulated streaming settings. The work standardizes manifest generation, Vietnamese text normalization, WER/CER evaluation with bootstrap confidence intervals, latency proxy measurement, real-time factor reporting, and publication-ready result aggregation. The paper was presented at ICCCNet 2026 in Manchester, United Kingdom.
Timeline
2026
Type
Publication
Status
published
My work
- •Built the manifest-first data standardization and ASR evaluation workflow
- •Implemented result aggregation, plots, bootstrap confidence intervals, and report-ready artifact generation
Outcome / Impact
- •Implemented a standardized JSONL manifest and strict Vietnamese normalization flow for fair ASR evaluation
- •Benchmarked models across offline and pseudo-streaming modes with WER/CER, real-time factor, latency proxy, and stability metrics
- •Produced publication-ready tables and figures including Pareto frontier, delta WER heatmap, significance analysis, and robustness views
- •Packaged reproducible scripts for cloud matrix runs and final paper artifact regeneration
- •Presented the paper at the 6th International Conference on Computing and Communication Networks (ICCCNet 2026), 17-19 July 2026
Tech / Skills
Project Media
Demo video and visual walkthrough for this project.
Project Screenshots
Certificates (1)
ICCCNet 2026 - Paper Presentation & Co-author Certificate
Case Study
1) Context / Problem
Vietnamese ASR systems are often compared in offline settings, while many product deployments need partial, latency-sensitive output. The project focused on making offline and pseudo-streaming comparisons reproducible, auditable, and easier to interpret across datasets and models.
2) Your Role
I built the benchmark workflow around data standardization, evaluation scripts, result aggregation, plotting, and report-ready artifact generation. The work connected engineering reproducibility with research-facing analysis.
3) Approach
Designed a manifest-first pipeline with deterministic dataset preparation, configurable model runs, strict Vietnamese text normalization, WER/CER computation, bootstrap confidence intervals, streaming chunk/overlap profiles, and matrix-run aggregation for multi-model comparisons.
4) Result / Impact
The benchmark produced comparable offline and streaming metrics across Vietnamese ASR datasets and model families, with visualizations for WER-latency trade-offs, robustness at 4000 ms latency, and significance of streaming degradation. The resulting paper was presented at ICCCNet 2026 in Manchester, United Kingdom.
5) Learnings
ASR quality cannot be summarized by WER alone when deployment requires streaming behavior. Latency, stability, normalization policy, dataset composition, and confidence intervals all change how a model should be selected.
6) Links
See links above.