Building an Interruptible Voice Runtime on NVIDIA Jetson

Introduction

In this article, I’m going to share my experience building an interruptible voice runtime on an NVIDIA Jetson Orin Nano. The goal of this project was to create a system that can handle voice commands in real time, with the ability to interrupt ongoing processes based on user input. The user presses a button and says a voice command like “Turn right”, voice activity detection (VAD) detects the speech, Whisper transcribes the utterance on the GPU, and a JSON command with a latency breakdown is emitted. I use a closed grammar with a small set of commands, so the transcribed text is matched against the grammar to determine the intent.

In one line the whole process flow is: AudioSource → VAD → UtteranceQueue → Transcriber → CommandDispatcher

The questions I’m going to investigate: where does latency accumulate? How does model inference compare with the rest of the pipeline? What does “cancel” mean when inference is already running? To answer these questions, I will document the architecture decisions and measure endpointing, queuing, speech-to-text and dispatching time.

You can find the whole project, including the measurement results, on GitHub.

Project setup

Hardware & Software

In this project I use my NVIDIA Jetson Orin Nano 8 GB Developer Kit, introduced in my previous article.

For recording, I chose my Anker PowerConf S3, a USB audio device with a built-in microphone and speaker. For better latency measurements, I connected it to the Jetson via USB instead of using Bluetooth. In the environment report, you can check the technical details of the hardware and the installed software on the board.

The Anker PowerConf S3 has a built-in noise suppression feature, which is very useful for conference calls, where echoes and background noise can be problematic. For this project, however, it caused some problems when I was prerecording my test set (see Dataset). After a stretch of silence, its noise gate takes a moment to “open” again, so the first syllable is attenuated or cut off before it reaches the USB stream. For example, I said “Turn right” and only the “right” part was detected. This behavior was sporadic, but it shows well that built-in noise suppression can introduce unexpected behavior that need to be taken into account when designing a voice interface for a real robot.

Models

For speech-to-text, I used faster-whisper with the CTranslate2 backend.[1] I built CTranslate2 from source for CUDA 13.2 (sm_87) to run it on the Jetson’s GPU.[2] The three reasonable model sizes for Orin Nano with 8 GB of memory are the ’tiny’, ‘base’ or ‘small’ variants, with 39M/74M/244M parameters.[3] Choosing between them is a trade-off between accuracy and speed. I chose the ‘base.en’ Whisper model with float16 precision. Since I use predefined commands and a closed vocabulary only in English, I explicitly chose the English-only model (.en suffix) to skip the language detection pass, which adds latency that can be avoided.

For the voice activity detection (VAD) I chose the Silero VAD model, which is a lightweight and efficient model for edge devices.[4] I chose to run it on the CPU with ONNX Runtime,[5] so it can detect speech segments in real-time. The model is available in ONNX format, which allows for easy integration into this pipeline. VAD is a per-frame classifier: its output is the probability of speech in each 32 ms frame. Silero is a small neural network with recurrent state. Endpointing is implemented as a small state machine that turns noisy per-frame probabilities into utterance boundaries. It enters SPEECH mode when p >= 0.5 and counts silence when p <= 0.35. It has a hangover of 300 ms, which means it will stay in SPEECH mode for 300 ms after the last frame classified as speech. The minimum speech duration is 150 ms, while the maximum utterance length is 4 seconds. The preroll is 320 ms, which means it will include 320 ms of audio before the detected start of speech in the output. Endpoint latency is the time between the last frame classified as speech and the moment the endpointing state machine decides to end the utterance.

Architecture Decision Records (ADRs)

Some of the key decisions I made during the design and implementation:

• 0001: Cancellation is achieved through a generation counter, not by stopping the GPU during inference. See Cancellation section below.
• 0002: One thread owns the state, other threads send events to it. This simplifies state management and avoids complex locking. All decisions are made as pure functions on the runtime thread.
• 0003: Every place where one thread hands data to another is a queue, and every queue fills up when its consumer is slower than its producer. Every queue has a written capacity and a counted policy for when it is full, except the event queue: it is deliberately unbounded, because dropping an event would break the state machine, and its depth is recorded instead. See Queues and metrics section below. *
• 0004: Keep the GPU exclusive to STT. Silero VAD runs on CPU via ONNX Runtime and never competes with STT for the GPU, so a long or stuck transcription can’t delay a VAD decision or a speech start.
• 0005: To add latency measurement points, the backend runs Whisper’s steps itself: features, encode, decode, with a time for each stage and a cancel check before each one, so a cancel during the encoder skips the decode.
• 0006: For simplicity, I accepted the following limitations:
  • tested only with one speaker and one microphone,
  • used exact-match grammar that will fail safe, but not smart (“turn right” transcribed as “Done right” is not accepted) to emphasize safety criciality,
  • push-to-talk only, wake up world can be a future improvement.

Event-driven runtime and the concurrency model

I used the single-writer pattern, which means that a single runtime thread owns all mutable state (state machine, generations, current job) and everything else talks to it by posting events to one queue.

Thread #1: PortAudio callback thread, which receives audio blocks from the Anker S3 device (captured with python-sounddevice, the Python binding for PortAudio[6][7]) and posts them to the VAD thread. It is a real-time thread, so it should not block or do any heavy work.[8]

Thread #2: VAD thread, which runs the Silero VAD model on the CPU. It receives audio blocks from the PortAudio callback thread and runs inference on them. Whisper and Silero expect 16 kHz input, while the Anker S3 samples at 48 kHz, so the VAD thread first converts the audio: a stateful FIR low-pass filter, then it keeps every third sample. It posts SpeechStart and SpeechEnd(utt) events to the runtime thread.

Thread #3: STT worker, the Whisper model running on the GPU. It is a single worker thread, so it processes one utterance at a time. It receives complete utterances from the runtime thread through the job slot and runs inference on them. When it finishes, it posts the result back to the runtime thread.

Thread #4: Keyboard thread, which owns the terminal in cbreak mode. It sends Ptt (push-to-talk), Cancel and Quit events to the runtime thread when the user presses the corresponding key. By default: Space for Ptt, c for Cancel, q for Quit.

Thread #5: Runtime thread, which owns the state, the generation, the job and the dispatcher. It consumes the events from the event queue received from the other threads and updates the state accordingly.

One rule shaped this split: no slow work on a thread that has a real-time obligation. The VAD thread has to process a 32 ms audio block every 32 ms. If the Whisper STT worker ran on the same thread, a transcription would block it, so the work is handed to another thread through a queue.

Queues and metrics

As introduced in ADR-0003, every queue gets a capacity and an overflow policy, with one deliberate exception (the event bus):

• Audio blocks queue (callback -> VAD): capacity ~2 seconds (62 blocks); when full, it drops the oldest block, but it never blocks the producer (PortAudio callback thread). Metric: audio_blocks_dropped and input_overflow.
• Job slot (runtime -> STT): capacity 1; when full, the newest wins and replaces the pending job. Metric: superseded.
• Event bus queue (all threads -> runtime): unbounded, nothing is ever dropped, because losing a cancel or a transcript would break the state machine. Events are rare and small (maximum depth 2 in the campaign). Metric: event_queue_max_depth, recorded but not enforced.

The command output is a synchronous write, so no queue is implemented.

State machine

The following diagram shows the state machine of the runtime thread. It has four states: IDLE, LISTENING, IN_SPEECH and TRANSCRIBING. The transitions between these states are triggered by events from the other threads.

What happens on each transition:

• PTT: generation++ and reset the VAD. In TRANSCRIBING it also stops the running job, because the newest press wins.
• SPEECH_END: submit the utterance as a job to the STT worker.
• TRANSCRIPT: the trascription of the current generation: emit the command, or no_command if the grammar does not match.
• CANCEL: generation++, and from TRANSCRIBING the job won’t emit the command.

Two rules apply in every state: an event from an older generation is discarded without changing anything, and a CANCEL in IDLE only increments the generation. A PTT during IN_SPEECH is ignored, so a press can’t cut off a command that is being spoken.

Cancellation

When the user presses the cancel key (c), ‘Cancel request’ is timestamped at the keyboard thread. ‘Cancel acknowledgment’ is timestamped when the runtime has transitioned to a listening-ready state, and from this instant on no result from before the cancel can have any effect. Since the in-flight CUDA kernel cannot be aborted and faster-whisper/CTranslate2 expose no abort for the decoding, ‘Drain’ measures the remaining inference time. When the orphaned result finally arrives, it’s discarded before it reaches the dispatcher, logged as ‘Stale-result rejection’.

The runtime holds an integer called the generation, which is incremented on every cancel and on every new press. Every utterance carries the generation it was created in. When the result arrives, the runtime compares the generation of the result with the current generation. If they are different, it means that the result is stale and should be discarded. These commands are called stale commands later. This pattern is also known as a fencing token: a number that only increases, checked at the point of the side effect, so that work started under an older number can never take effect.[9] For illustration, please see the following diagram:

To save GPU time, the runtime creates a cancel token, which stops processing the utterance between pipeline stages (features -> encode -> decode). It is useful for jobs that are cancelled while queued. It is only an optimization; the guarantee is the generation check.

Measurement setup

• Run A (offline STT): the STT on its own. The Whisper model runs without the VAD and the runtime, so the measured latency is only the STT inference time.
• Run B (replay): the whole pipeline, but the audio is replayed from a file instead of a real microphone. This allows for repeatable and controlled testing conditions, ensuring that the results are consistent across different runs. Latency is also compared between DVFS (default clocks) and pinned clocks: Run B runs once with default clocks and once when the clocks are pinned. The dataset is played as one continuous stream paced in real time, every clip followed by 2 s of silence, in a shuffled order (fixed seed), with the talk key pressed when a clip starts: 5 repetitions with pinned clocks, 2 with default clocks.
• Run C (cancel-fuzz): the whole pipeline with replayed audio, and cancels or second presses fired at seeded random moments, to check that no stale command is ever emitted. 150 trials: 60% cancel 0–300 ms after the job is submitted, 20% cancel between the press and the end of speech, 20% a second press on b2b clips (supersede).
• Run D (live): the real microphone, with me speaking live: the terminal prompts what to say, and some prompts ask to cancel mid-command: 20 prompts, 4 of them “start saying it and press c”.
• Run E (sync baseline): the same cancel-fuzz as Run C, but with the transcription on the runtime thread instead of the STT worker, to show what a cancel costs when the runtime thread is blocked. 30 trials.
• STT length sweep: like Run A, the transcriber on its own. Five quiet commands (one per intent, cut at the label ± 50 ms), each followed by digital zeros up to 1.5, 2, 4, 8, 15 or 30 s; 10 repetitions in a new random order each time, 300 timed runs.

Common to all runs:

• In the offline and replay paths, the speech_end_s labels come from the CSV, and for live runs they come from the VAD’s estimate.
• All tests include a warm-up. Load the model and run 5 dummy inferences before accepting PTT. Report the very first (cold) inference separately; for a robot, “first command after boot” is an important detail.
• The clocks are pinned to their maximum with jetson_clocks (CPU, GPU and memory), except in Run B’s default-clocks comparison.
• Power mode is recorded.

Latency measurements are done with the following calculation:

Dataset

For testing, I defined five intents with closed grammar: TURN_LEFT “turn left”, TURN_RIGHT “turn right”, FORWARD “go forward”, BACK “go back”, STOP “stop” spoken approximately 0.5 meters from the microphone.

The total test dataset contains 30 clips with 35 labeled utterances in 4 conditions:

Ambient noise was ~32 dB in the quiet condition and 37–42 dB in the noise condition; speech was 60–62 dB in both, measured with the Apple Watch Noise app.

For the latency measurement, it is important to define the start and end of each command precisely. First, I inspected the dataset by using loudness envelope analysis to see the dBFS (Decibels Relative to Full Scale) profile. I asked Claude to measure the clips using the envelope analysis script to label the starting point when the waveform first rises above the noise floor, and the end when it returns to the noise floor after the last sound. After that, I verified the results in Audacity 4 and adjusted them where needed; the noisy clips in particular were not always accurate.

The recordings with the manifest are available in the repository. The manifest contains the following information for each recording:

• descriptive utterance id (utt_id)
• path to the audio file (path)
• condition (quiet/noise/negative/b2b)
• reference text, audio transcription (ref_text)
• intent, command that the robot should execute (intent)
• exact start of the speech in seconds (speech_start_s)
• exact end of the speech in seconds (speech_end_s)

For the noise-only negatives (door, keys, cough), the start and end mark the sound instead of speech, and the reference text and intent are empty.

After label validation in Audacity 4, I froze the dataset by recording a SHA-256 of each audio file and of the manifest, so the labels are frozen too, and every code change is tested against the same input data.

Results

All numbers are medians with pinned clocks unless noted; the full tables are in the repository.

Where the latency accumulates

A quiet command is emitted 320 ms after the end of speech (p95 439 ms). Two components make up almost all of it: the endpointing, 184 ms, and the STT, 137 ms (43%). The runtime’s own work, queueing and dispatch, stays below 1 ms. Endpointing is mostly the hangover, a chosen parameter: the endpointer waits 320 ms after the last block the VAD classified as speech. That block typically comes ~120 ms before the labeled end, because the labels include the word’s fading tail, so the measured endpointing is ~190 ms rather than 320 ms.

In noise, endpointing dominates: 1096 ms. The VAD keeps classifying the radio’s speech as speech, so the end of the command is found late, and in 6 of 10 noise clips silence never ended the utterance at all; only the 4 s maximum or the end of the file did. The STT is then only 11% of the latency. The results show that in a noisy room, silence is not a reliable end signal.

With default clocks (DVFS), the STT takes 187 ms instead of 137 ms, and a quiet command takes 370 ms end to end. Pinning the clocks saves about 50 ms per command.

Inside the STT job

The encoder is 111 ms of the 137 ms (81%), and it costs the same at any input length: Whisper always encodes a 30 s window and pads shorter audio. From 1.5 s to 30 s of input, the encoder changes by less than 1 ms (111.4 → 111.9 ms). Only the feature extraction on the CPU grows with the length (2 → 25 ms), and the decode grows with the number of tokens (12.5 ms + 3.85 ms per token). Within the runtime’s range (at most 4.3 s of audio), the input length changes the STT time by 2 ms. So for short commands, the real-time factor is misleading: a 1 s command costs as much as a 2 s one.

Cancel while Whisper is running

With the STT worker, a cancel that arrives during a transcription is acknowledged (“ack” in the figure) in 0.34 ms (p95 4.2 ms): from then on the runtime listens again, and nothing from before the cancel can be emitted. The GPU is busy for another 32 ms (p95 100 ms) until the running stage finishes (“drain”), because a CUDA kernel can’t be interrupted; the next stage is skipped. In the sync baseline (Run E), the cancel waits for the whole transcription: acknowledged after 55 ms (max 135 ms).

Stale commands emitted: 0 in 150 trials with the worker and 0 in 30 with the sync baseline. An independent check over all five campaign traces (357 jobs) found that every job ended exactly once and every discarded result was justified.

Recognition

• Offline (Run A): 27 of 30 commands recognized, word error rate 4.9%, 0 of 5 non-commands accepted.
• Through the pipeline (Run B): 17 of 20 commands per clip, 0 of 5 false accepts. Replay is deterministic per clip, so the three failures (“All forward.”, “Pull back.”, “The one right”) fail in every repetition. Two of them are noise clips that Whisper also mishears with the labeled cut in Run A, so the radio is the cause. Only quiet_forward_01 passes offline (“Go forward.”) and fails in replay (“All forward.”): the preroll keeps the whole command in the audio, but the VAD’s cut has a little less silence around it than the labeled cut, and that small difference changes Whisper’s output.
• Never a wrong command: every miss ended as “no command” or “nothing transcribed”. The exact-match grammar fails safe.
• Live (Run D): 16 of 16 command prompts recognized and 4 of 4 cancels acknowledged (0.21 ms), with an STT time of 135 ms.

Queues

No queue came close to its limit: 0 dropped audio blocks and 0 microphone overflows in every run, an event queue depth of at most 2 (4 in the sync baseline), and the VAD took 0.5 ms per 32 ms block.

Conclusion

In this project I built a push-to-talk voice command runtime on the Jetson Orin Nano. Since this was my first project in speech recognition, it was a very good learning exercise to think through the design of the speech processing pipeline, make architectural decisions and learn about concurrency handling for a system where real-time performance is critical.

My goal was not only to measure latency, but also to lay the foundation for a voice interface that can be used later on a real robot. The measurements show that the system is fast and responsive, emitting a command 320 ms after the end of speech, and the cancellation mechanism works: 0 stale commands in 180 trials.

There is still room for improvement, especially in noisy environments, and there is more to learn from experimenting with different model sizes and settings. In my next project I want to focus on building a computer-vision-based behavior recognition system. Nevertheless, as a long-term goal I plan to implement the voice runtime as a ROS 2 action[10] combined with other systems to achieve multimodal human-robot interaction.

References

[1] SYSTRAN, faster-whisper: Faster Whisper transcription with CTranslate2. https://github.com/SYSTRAN/faster-whisper

[2] CTranslate2 installation: https://opennmt.net/CTranslate2/installation.html

[3] Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), arXiv:2212.04356. https://arxiv.org/abs/2212.04356

[4] Silero VAD: https://github.com/snakers4/silero-vad

[5] ONNX Runtime: https://onnxruntime.ai/

[6] python-sounddevice: Play and Record Sound with Python. https://python-sounddevice.readthedocs.io/

[7] PortAudio: an Open-Source Cross-Platform Audio API. https://www.portaudio.com/

[8] Ross Bencina, Real-time audio programming 101: time waits for nothing. http://www.rossbencina.com/code/real-time-audio-programming-101-time-waits-for-nothing

[9] Martin Kleppmann, How to do distributed locking (2016), section “Making the lock safe with fencing”. https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

[10] ROS 2 Documentation (Jazzy), Understanding actions. https://docs.ros.org/en/jazzy/Tutorials/Beginner-CLI-Tools/Understanding-ROS2-Actions/Understanding-ROS2-Actions.html